Actually I don't know the details how TreeTable works myself. I just added the class from the SwingX project
https://swingx.dev.java.net/ if I remember correct and used the functionality.
As a roadmap I would just suggest you to think of something easy to improve on the GUI. There are a few things on the Wishlist in the Wiki
http://www.phex.org/wiki/index.php/Wishlist
or in the Phex Bug Tracker, like this one:
SourceForge.net: Detail: 1810708 - clicking enter in result monitor should start passive search
I can also assign you one which should be good to start with.
As the next step I would suggest to start from the corresponding tab or dialog class to check where to integrate it. Most GUI part is not so difficult like the tables.
Most code in Phex uses JGoodies Forms to layout the components. So I would suggest you get familiar with it. But you can also use standard JDK LayoutManagers when you design a whole new panel and you feel more familiar with it.
So instead of trying to understand the whole GUI, which is just not possible in short time I would suggest you to start just with the parts where you want to change things. If you feel unsure where to look you can just ask or send me code to review I can help you get back on track.
Gregor