Anyone that's run the P3 (Plugin Performance Profile) plugin knows that Jetpack is always the weightiest plugin they have, by far. If it isn't then God help whichever one it is because it'd have to be a monster. But it also makes sense and may even be appropriate, considering the number of modules most people are going to be using at one time. If you only needed a couple of features that Jetpack offers, then it may be smarter to use dedicated plugins and CDN or just code it up instead if you're really concerned about incremental gains in speed.