It doesn't have to be complicated, but yes, it is developer-centric overall.

Essentially, the API provides a standard means of accessing and/or manipulating the data sitting in a WordPress instance. This seems smaller than it actually is.

Consider a theme that wants to, say, display posts without making a refresh to go to the next page, which would be the traditional way to do it. Obviously, for such a case, you can roll-your-own endpoint to access the data you need, and in the past, this is what most solutions have done. You make a PHP function to return the data you need, and some Javascript in the theme to access that endpoint. A standardized API provides a standard set of endpoints for doing this sort of thing, all you have to write is the Javascript, which is actually quite easy to do.

Consider a plugin that wants to, say, take input from a form and save it into the database, maybe using a custom post type. The plugin has to have code to get the data from the form, insert the post, sanitize, validate, and so on. With an API that will let you write posts, you can use that API to do all the grunt work. All you do is to write the form and some JS and maybe also do some authorization checks and such.

A standard means for getting and manipulating the data can indeed let you redo the whole admin side if you want. It would make it very simple to do something like front-end-editing. Or maybe to drag and drop and rearrange the widgets live. But the point is that it's rather generic and standard, allowing for a wide variety of uses.

It also eliminates a lot of the security problems inherent to roll-your-own code to do essentially the same things, because plugin and theme authors have proven, time and again, that they're actually not very good at PHP, and that security is hard to get right. One standard place to manipulate the data, with security built into it, is probably a good thing.