In reply to Jonas Tietgen.

I can think of two real issues with a bloated media library:
1. When browsing the library, the more items there are, the harder it is to find what you're looking for.
2. When backing up a site, either routinely or as part of a migration, media files take up the most space and therefore the most time.