None of them. The current solution assumes everyone will want to use block patterns all the time. As I wrote in https://github.com/WordPress/gutenberg/issues/23301

It would appear to me that the current API for registering, deregistering and rendering block patterns is rather inefficient.
It could benefit benefit from lazy loading, separating the rendering logic from the registration.

If the API is not improved, sooner rather than later, then I envisage significant bloat when plugins and themes start adding their own Block patterns and categories.