Unless you want to be a free loader (leech), then it would be best to configure the problem so the firewall icon goes away. Otherwise people will struggle to downld from you. What firewalls do you have & what broadband modem/router brand & model do you have?
BTW yes it can have 'some' effect on your download speed.