I think you might have to add a new comparision feature to incoming connections. It will have to decide, if it wants to drop a working connection when it gets a new incoming one, and that code can get quite complicated (including a setting to reserve slots for other vendors to make sure that Phex doesn't isolate itself - or a setting to only grant at most 2/3rd of the slots to a specific vendor or similar to avoid isolation in a single-vendor-island).
I'm not perfectly sure about the right place, but the OIOServer looks fitting:
https://phex.svn.sourceforge.net/svn...OIOServer.java
Gregor should know that better...