I've experienced something like this before. This is a memory leak of sorts, and revolves around the way .NET deals with "Unmanaged resources".
Your service is creating too many unmanaged Windows Handles and not deleting them in a timely fashion, causing it to hit a hard limit of 10,000 handles.
Look at the Process tab of the Windows Task Manager, click on the view menu, Select Columns. Add User Objects to the list. Most programs will show a few in use, but the offending process will (in time) show thousands.
You need to make sure that every object which you create derived from IDisposable is also Disposed when it is no longer needed. Also, any event handlers to those objects need to be disconnected prior to disposing of them.
If the process is fairly linear, I suggest looking at the using statement. In the following example, Font is derived from IDisposable. The using statement ensures that Dispose is called at the end of the block.
using (Font font1 = new Font("Arial", 10.0f))
{
byte charset = font1.GdiCharSet;
}
On top of this, you may need to force garbage collection as well to ensure that all of the objects are deleted.