As Nacin said, the Customizer is very well-architected. The ease with which the improvements in 4.0 were made is an excellent example of this.

I think we are overhauling the Customizer, from a developer's and end-user's perspective. But in core, we're doing it itertatively; maintaining backwards compatibility because we can and should, and making changes bit-by-bit so that improvements get to users more quickly. In 4.0 we have improvements to the Customizer controls UI/UX, and several significant additions to the PHP API. For 4.1, 4.2 and beyond we're already working on several things including a much-expanded JS API that would enable significant performance gains, alternative approaches to accessing the Customizer so that its role shifts to a squarely front-end context, the addition of custom menus to the Customizer (fixing numerous existing issues with menus in the process), a better experience on mobile (the Customizer is functional on mobile currently in Core and that fact should be made more obvious, although WordPress.com's extended version isn't), and, of course, improving speed/performance. I predict that we'll have the entire appearance-customization aspect of WordPress integrated into the Customizer within the next year or so, including integrating themes, provided the community continues contributing work to the Customizer in core at a strong rate.

In WordPress core, significantly backwards-incompatible changes simply aren't an option because the developers do go to extreme measures to put users' needs first. We take our philosophy seriously, and the majority of users' problems are the result of plugins and themes ignoring the core philosophy or being otherwise poorly coded.

There are many options for developers who aren't satisfied with a WordPress core feature; big ones are to leverage WordPress' extensibility to either expand the feature (like WordPress.com currently does with the Customizer) or replace it with a custom implementation of something better (like Automattic is experimenting with for their Customizer), or to help improve the feature in core. Core contributors are just regular developers; I barely consider myself a developer now and I've been contributing to core for over a year. You don't need to do anything special to jump in; all help is appreciated, patches are welcome, and the more people we have contributing to the project, the more we can accomplish and the more we can improve WordPress for everyone. Even if you're not a developer, anyone can help test features and work on improving the user interface and user experience.

So rather than dwelling on issues with the Customizer, I'd love to see more people jump in and work on constantly evolving it through core's iterative approach, and taking advantage of newer technologies like Backbone.js and APIs like WP-API as they become integrated into core over time.