I actually disabled Gravatar after reading this post and my page load decreased from over 20 seconds to under 5 seconds for some of my most popular posts. So yeah, don't think I'll be turning it anytime soon.