You can use the framwork provided StopWatch class, but if you are trying to learn and implement your own stopwatch, then your mistake is actually where you have pointed out yourself, you need to hold the :
var difference = DateTime.Now - stopTime;
return String.Format("{0} HR:{1} MN:{2} SEC", diff.Hours,diff.Minutes,diff.Seconds));