I definitely run into issues with managing widgets, particularly when it's pertaining to wanting specific widgets on certain pages -- not all widgets on all pages.

I've been using Widget Context to achieve it, and it's been working really well.

But I totally agree that the widget area of WordPress needs some love.