The connect timeout was set at 5 seconds. Everything timed out until I set it to 15. For good measure I also tripled the other 2 connect settings.
Now it works allright, but that was annoying as hell. <GROWL>