IMO if there is going to be a change in how the Smilies are implemented it should be that they are removed from core all together. They are a pretty clear case of "plugin territory" as they do not serve an essential widespread use case. Most WP users are not aware the Smilies are even in core, but those who stumble on them often do so by error and are frustrated when they appear.

Smilies have their place, in particular in chat streams as emoticons, but WordPress is not a chat application.

Alternatively, if Smilies are to be upgraded I would propose adopting a wider range of emoji options to go beyond "just" smilies. The emoji found in Google Hangouts are great but even that enormous collection is limiting in scope. No easy solutions here. Just more questions.