In reply to David Maillard.

Since WP 5.8, theme authors can opt into only loading the block styles used on the page. More details in this announcement. Maybe that'll help with what you need.