This can happen when one person is uploading at a moderate speed, not even the whole bandwidth. I can always tell when someone connects: my download speed drops, though not every time.