In reply to Devin.

I did the swap thing after the first time it ran out of memory. Helped a little bit, but was still frustrating whenever the client when to try to do something the site was down (and this was a basic install of WP, nothing special at all). Might try again one day with nginx :)