Using multi-site how do i upgrade all of my plugins on a network wide basis without logging into all my sub-domains.

When you upgrade one plugin, you;re upgrading the ONE file used by all the sites in the network.

When you say that you have to create a mu-plugins folder/directory in wp-content, does this then mean transferring all my existing plugins into this folder and deleting original plugins folder?

No, this is an *extra* folder for plugin that run continuously. If a plugin is meant to go here, the instructions for that plugin will say so.

Extra reading:
http://wpmututorials.com/basics/what-is-the-mu-plugins-folder/

I have a normal wp install which i have upgraded to multi site. Do plugins have to be MU specific to upgrade network wide?

No. like I said above, if you install a plugin, it is now accessible to all sites in your network. One copy.