I didn't notice any real delay when we fought. It was worse with Daddy lol.
It could be simply a problem with your router denying the ping request (fairly common situation). When the other Xbox tries to ask your Xbox what it's ping is, instead of replying it simply says "Shut up! Go away!" so the game just returns a 0.
Some networks are uber paranoid about security. I had this problem until I unchecked "Block Anonymous Internet Requests" on my router.