Make sure you don't upload at full speed. Always leave at least 2-3k/sec (12-15k/sec if you are serving as an Ultrapeer) of your upstream unused by uploads.

If your upstream is completely used up, your computer cannot sent the acknoledgement-packets fast enough to the hosts you are downloading from, so your download-speed will be reduced.