I presently only have very limited internet access and cannot answer you fully. I'm not sure which tool you are referring to. But most Photoshop tools have a sensitivity option (or Tolerance) and reducing or increasing the sensitivity might help prevent your issue, else feather the selected portion. The sensitivity/Tolerance option is usually on the left side in the horizontal bar. Edit: the Background Selection tool also has a Limit's option: Contiguous, Discontiguous, or Find Edges.
There is also a Protect Foreground Layer option you will see.

Another option might be to use that background selection and turn it into a mask so you can adjust the edges manually wherever you were finding an issue with it. I previously mentioned (last year or prior) about adjusting a mask's edges by selecting the masked layer and selecting the mask in the Channels tab, and using a white or black pen/paintbrush to add or subtract from the red or non-red masked selection. Then if you uncheck the eye for the mask & simply Windows-key click on the layer name it will then show the selection.