I tend to agree with @Otto too. It is a good idea, but you definitely need to build in security for it to work properly.

Apart from that, I host all my plugins on both the WordPress Repository and Github. But what's on Github is "stable" beta as I like to call it and on the official Repository I only host the latest stable versions.

So for normal users I would not recommend to download my plugins via Github.