I found that one of the best things I did for my WordPress install on my DreamHost VPS was to harden its security. Sucuri has a good guide here: http://tools.sucuri.net/?page=docs&title=wordpress-hardening

Memory usage was getting away from me until I followed the hardening. I guess WordPress installs are a favorite amongst folks who probe for vulnerabilities.