Building a business on Open Source is always tricky, since it's constantly evolving and changing based on the collective group working on it. So doing something above and beyond a simple feature edition is where you'll likely find more success.

Premium Plugins seem to be the hot button topic lately, but if we look back to a couple years ago when the theme market exploded, WordPress itself always had great themes and offerings, yet people still purchased the premium themes, and they will with plugins as well, if your plugin (and extras you're offering, like support, etc) are worth the ticket price.