I personally use Jetpack plugin on my blog and it doesn't impact on my site performance. the important think is that by using jetpack i have to use less amount of plugins as most of my needed plugins are within jetpack.