1. Decide between windows & AVG firewalls. If it's only things like checking viruses & spyware, etc, fine, but for firewalls should only use one.

2. Make sure none of the computers are using the same connection or listening ports. Routers will reserve a single port for each program on each computer making it inaccessible to the other computers. So try changing their ports under Tools > Options > Advanced > Firewalls. Might need to set it to manual but if you do that, the UPnP for the router won't work so NAT might block incoming. O.o Ports to try: 49152 - 65535
Of course port forwarding would work too.