Browse host is a bit like chat... has a mind of it's own
You might not be able to browse someone then 5 minutes later, you'll be able to.
Remember, browse host has a lot more chance of success if you have chat enabled (tools > options > chat. Tick 'enable chat' & apply) and if the other person also has chat enabled - you'll see a smiley face.

For a lot more info, use the search (top, next to quick links) to look for 'browse host'. There are so many threads about this topic

You can't really prevent anyone from browsing your files unless you choose not to share... as you said, that's not in the spirit of file sharing!