In reply to alex.

Updating a plugin from the WordPress.org repository from within WP-Admin should delete the folder and then create a fresh folder with the updated version. I believe the same happens for themes as well.

For manual updates, that is the same approach that needs to be taken. Delete the folder and upload a fresh copy of the updated file. Otherwise, as you say, something old will come back to bite us.