I have seen two versions of this behavior that crop up occasionally:

1) No matter what I set the max downloads to, Gnotella always thinks I've set a larger number.
2) No matter what I set the max downloads to, Gnotella always thinks I've set it to 0.

Quitting and relaunching Gnotella will fix the problem. I haven't yet figured out how to reproduce it reliably.