that can happen if his upload slots are busy or connections are being blocked by firewalls, and so on.

see if he can connect to you (send him your ip:port number that is shown just above the direct connect entry box).

cheers