I was doing a bit of research earlier and discovered the filter "should_load_remote_block_patterns". This allows you to disable all block patterns that are fetched from the WordPress patterns directory. You can then register your own client-specific patterns as you wish.

That said, your overall point is well taken. While there is value to the new pattern explorer to many segments of the WordPress community, it also can be a hindrance as well. With any new feature, there also needs to be a way to turn it off.