IP spoofing can only work for a DDoS attack if the return path is valid. For example, if I'm using that same ip address, and traffic is sent to that IP, I'm going to be the one that receives it. Spoofing via UDP is a possibility, but it's really not possible with TCP.
A true DDoS attack on your IP will be from multiple sources and not just one.