Are you running on a Linksys router? If so I have the same problem.
But my issue is the newer versions of the WRT54G models (Above v5) can't handle multiple source downloading from what I understand. The router chokes when you try and download from too many people. + My bittorrent download speed has drastically dropped from like 800kb/s to 200kb/s for single torrents. I really hope they fix this with a new firmware upgrade.