We already know that Gravatar is a third party service and that support for it is built into WordPress. But, is that dependence on Gravatar slowing your site down? SEOMofo takes a hard look at the performance hit that Gravatar can cause on a WordPress powered website with various configurations in place such as whether they are enabled or disabled, custom default Gravatar, etc. It’s not surprising that a page load increases as the number of comments with unique Gravatar hash urls also increases. The numbers presented in the post may seem low but when taking a look at the big picture, there are performance gains to be had. For example, hosting the default Gravatar image on your own web server instead of using the one on Gravatars server is a big help.
Read through their performance tests to see the hard numbers and then tell me whether or not it’s worth the effort.