I don't see the problem. In fact, I disagree that this is plugin territory. A theme can add service workers tailored to the function of the theme (and that is better practice esp for the fetch section of a service worker). The end user doesn't need to know anything that way - that's a red herring issue in any event. End users shouldn't need to know anything about caching, CSS or serviceworkers in order to have their CSS cached for offline use (to give one small example).