Salut mec, it is highly probable that your router can't handle more than 3 persons at any time on LW.... This is sad but probable. Did you do an upgrade of the router firmware? That might help a little. Otherwise, I highly doubt the modem is the bottleneck and you might want to check the "Disable ultrapair capacity" in the Options -> Speed of all your 4 LW, because being an UP makes more connections and higher the possibilities of hardware failure (router).
This is obviously not a firewall related problem. Must be the router.
Ciao et bonne chance