If you're using Norton does it have it's firewall active? Remember MS comes with its own firewall so you're best only using one of them. Having 2 will cause them to clash & result in less security not more. Most firewall companies will tell you the same thing. MS did also have a page that said the same thing.
Other than that, download speeds are generally dependent upon sources. So searching new sources helps using the search function.

I don't know much if anything about Vista. But I do know with Xp their SP update actually resulted in many less simultaneous connections. There is a 3rd party patch to fix that. I don't know about Vista or whether it has same problem.

Try downloading a magnet link from the following link site & tell us about the speeds & connection that happens: http://www.magnetmix.com/video.shtml