Unless one is running a highly visited blog such as wpTavern, I don't worry about plugins slowing down the performance.

The benefits of plugins is far too great to be ignored.

The only concern for plugins is that of potential conflicts. Having run a blog with 30+ activated plugins, I guess I've been fortunate to avoid conflicts!

Well, JavaScript conflicts have occurred. The plugin authors have to learn how to wpenque those JS libraries.