You should remove all unused plugins and themes for two reasons.

1. Even if they are deactivated in wp-admin, the code is still directly accessible on your server. Here's an example of an exploit that works even when the plugin is deactivated.

https://technicalagain.com/2019/04/14/conwell-quotes-wordpress-plugin-with-a-backdoor/

2. The themes and plugins may fall out of date if inactive and you stop updating them. The more they age, the more likely exploits will become known and published. WordPress has a very active community of security researchers and hackers that make exploits very easy to scan for and execute.

Bottom line is deactivate and delete anything that is unneeded. Update as soon as practicable.