This could always happen with a Java runtime, you can use startup options to set the maximum Java heap size (64M is default), start phex as follows:
java(w) -Xmx256M -jar phex.jar
in your unix shell or Windows command line. On Windows you can make a shortcut of your phex.jar and change its command line, so the doubleclick action still works.

Helge