I might be misunderstanding the issue, but can’t you already split the built core block styles and then dequeue them individually?

Using something like: wp_deregister_style( 'wp-block-gallery' );

Then enqueueing your own styles?