In reply to alex.

WordPress actually does remove the offending files in this case: https://core.trac.wordpress.org/changeset/32386.