Big thanks to LOTR. I went back to LW4.8.1 and it runs fine. Getting internal error messages ocasionally when it has been open for a long time, but I remember getting those before. Ran it for two hours and everything is responding normally.
However I'm really confused now because the activity monitor is still showing all the physical memory used up and an ever-increasing VM allocation. So maybe the acitvity monitor is lying and it was just a red herring. When I get time I will work forwards through the versions and see at what point it goes wrong.

I just tried to really test it by opening a photoshop document. It was a bit slow but it opened ok and seemed to steal the memory back from LW. When I closed PS the free memory went back up to 200M and LW started to slowly use it all up again.