Restart
should quit the current instance of the app and start a new instance, but sometimes the shutdown doesn't work as expected because of something your app is doing. I have no ideas what that is, but the workaround is really simple.
Call:
Application.Restart();
Application.Exit();