It all started with an email sent to a daemon. A windows service hosting two modules, each of which monitors an inbox for automation, dutifully ignored warnings from IT that basic authentication for O365 would be switched off in several months. Months went by… Even though Microsoft’s announcement was 2 years ago by the time the right people were informed the deadline was just 2 months away. Sound familiar? Unfortunately, the current IMAP API doesn’t support OAuth2 authentication so would have to be replaced. Even worse, we wasted weeks sorting out access with our Azure admin even though we had step-by-step instructions from day one.
Investigating mainstream IMAP APIs supporting OAuth2 turned up MailKit whose author was reassuringly active on Github and StackOverflow. We quickly discovered devs everywhere were addressing this issue and there was much debate on how, or even if, it could even be done (some of this doubt from the author himself). Thankfully after a painful couple of weeks, we were authenticating a daemon without user interaction (aka OAuth2 client credential grant flow).
When it comes to writing an API there is a spectrum in abstracting and obfuscating the inner workings from the user. On one hand, an API written to be 1:1 with the server is less usable but may give minute control and transparency allowing for better debugging. This path requires more ramp-up time and leaves more complexity to the user. On the other end of the spectrum, the API does some of the heavy lifting and aims to provide a usable, easy-to-use interface. A typical trade-off being the inner workings are a black box which might bite you in the ass down the road.
MailKit is wholeheartedly in the former camp compared with our old API. The old one connected, there was a new email event, then disconnect when the service shuts down. It was just overall easier to use from deleting a message to searching for new emails. For example, the email UID was part of the email object. With MailKit this information must be queried separately because technically that’s how it’s stored on the server. And this sets the tone for the entire experience of interacting with MailKit.
As mentioned above, even if a bit more difficult to use, it was very reassuring to see how active the author and user community are. While porting code from the old API required a lot of rewriting there was plenty of documentation, discussion, and examples out there to answer our questions. What was unexpected was that the server events did not work without building a full IMAP client, which reminds me of implementing a Windows message pump, to idle and process events in its own thread. Thankfully documentation and examples, although complicated, were available to build upon.
With the preamble out of the way, we can finally talk about the code. What follows is a C# wrapper for the MailKit API. We can use it in two different ways. You can simply instantiate it and execute a command with two lines of code. This will automatically connect, run the IMAP command within the IMAP client thread context, and then disconnect. Alternatively, you can use it as a long-running connection which will launch the IMAP client as a robust task which will stay connected until stopped. This allows the use of an event the wrapper exposes to process new messages. There is also a command queue so that code can be queued to run within the IMAP client thread context.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Runtime.CompilerServices;
using System.Threading;
using System.Threading.Tasks;
using MailKit;
using MailKit.Net.Imap;
using MailKit.Security;
using Microsoft.Identity.Client;
namespace Codinglifestyle
{
public class ImapClientEx : IDisposable
{
#region Member variables
ImapClient _imapClient;
IMailFolder _imapFolder;
int _numMessages;
CancellationTokenSource _tokenCancel;
CancellationTokenSource _tokenDone;
Queue<OnImapCommand> _queueCommand;
bool _messagesArrived;
readonly string _imapServer;
readonly string _imapUser;
readonly string _authAppID;
readonly string _authAppSecret;
readonly string _authTenantID;
readonly SecureSocketOptions _sslOptions;
readonly int _port;
readonly FolderAccess _folderAccess;
protected DateTime _dtLastConnection;
readonly object _lock;
#endregion
#region Ctor
public ImapClientEx(string userEmail)
{
_queueCommand = new Queue<OnImapCommand>();
_numMessages = 0;
_lock = new object();
Config config = new Config("O365 Settings");
_authAppID = config["App ID"];
_authAppSecret = config.Decrypt("App Secret");
_authTenantID = config["Tenant ID"];
config = new Config("Mail Settings");
_imapServer = config["IMAP Server"];
_imapUser = userEmail;
_sslOptions = SecureSocketOptions.Auto;
_port = 993;
_folderAccess = FolderAccess.ReadWrite;
}
#endregion
#region Public Events
public delegate void OnImapCommand(ImapClient imapClient, IMailFolder imapFolder);
public event OnImapCommand NewMessage;
private void OnNewMessageEvent(ImapClient imapClient, IMailFolder imapFolder)
{
if (NewMessage != null)
NewMessage(_imapClient, _imapFolder);
}
#endregion
#region Public Methods
public async Task RunAsync()
{
try
{
QueueCommand(OnNewMessageEvent);
await DoCommandAsync((_imapClient, _imapFolder) =>
{
IdleAsync().Wait();
});
Log.Debug(Identifier + "IMAP client exiting normally.");
}
catch (OperationCanceledException)
{
Log.Debug(Identifier + "IMAP operation cancelled...");
}
catch (Exception ex)
{
Log.Err(ex, Identifier + "RunAsync");
}
finally
{
Dispose();
}
}
public bool IsConnected => _imapClient?.IsConnected == true && _imapFolder?.IsOpen == true;
public string Identifier => string.Format("IMAP {0} [{1}]: ", _imapUser, Thread.CurrentThread.ManagedThreadId);
public void Stop()
{
_tokenDone?.Cancel();
_tokenCancel?.Cancel();
}
public void Dispose()
{
Stop();
DisconnectAsync().Wait();
if (_imapFolder != null)
{
_imapFolder.MessageExpunged -= OnMessageExpunged;
_imapFolder.CountChanged -= OnCountChanged;
}
_imapFolder = null;
_imapClient?.Dispose();
_imapClient = null;
_tokenCancel?.Dispose();
_tokenCancel = null;
_tokenDone?.Dispose();
_tokenDone = null;
}
#endregion
#region IMAP Connect / Idle
private async Task ConnectAsync()
{
if (_imapClient != null)
Dispose();
_imapClient = new ImapClient();
_tokenCancel = new CancellationTokenSource();
Log.Debug(Identifier + "Connecting to IMAP server: " + _imapServer);
if (!_imapClient.IsConnected)
await _imapClient.ConnectAsync(_imapServer, _port, _sslOptions, _tokenCancel.Token);
if (!_imapClient.IsAuthenticated)
{
var app = ConfidentialClientApplicationBuilder
.Create(_authAppID)
.WithClientSecret(_authAppSecret)
.WithAuthority(new System.Uri($"https://login.microsoftonline.com/{_authTenantID}"))
.Build();
var scopes = new string[] { "https://outlook.office365.com/.default" };
var authToken = await app.AcquireTokenForClient(scopes).ExecuteAsync();
Log.Debug(Identifier + "Creating OAUTH2 tokent for {0}: {1}", _imapUser, authToken.AccessToken);
var oauth2 = new SaslMechanismOAuth2(_imapUser, authToken.AccessToken);
Log.Debug(Identifier + "Authenticating user: " + _imapUser);
await _imapClient.AuthenticateAsync(oauth2, _tokenCancel.Token);
}
if (!_imapClient.Inbox.IsOpen)
await _imapClient.Inbox.OpenAsync(_folderAccess, _tokenCancel.Token);
_imapFolder = _imapClient.Inbox;
_imapFolder.CountChanged += OnCountChanged;
_imapFolder.MessageExpunged += OnMessageExpunged;
_numMessages = _imapFolder.Count;
}
private async Task DisconnectAsync()
{
try
{
if (_imapClient?.IsConnected == true)
await _imapClient.DisconnectAsync(true);
Log.Debug(Identifier + "Disconnected.");
}
catch (Exception)
{
}
}
private async Task IdleAsync()
{
do
{
try
{
await DoCommandsAsync();
await WaitForNewMessages();
if (_messagesArrived)
{
Log.Debug(Identifier + "New message arrived. Queueing new message event...");
QueueCommand(OnNewMessageEvent);
_messagesArrived = false;
}
}
catch (OperationCanceledException)
{
Log.Debug(Identifier + "IMAP Idle stopping...");
break;
}
} while (_tokenCancel != null && !_tokenCancel.IsCancellationRequested);
}
private async Task WaitForNewMessages()
{
try
{
Log.Debug(Identifier + "IMAP idle for 1 minute. Connection age: {0}", DateTime.Now - _dtLastConnection);
if (_imapClient.Capabilities.HasFlag(ImapCapabilities.Idle))
{
_tokenDone = new CancellationTokenSource(new TimeSpan(0, 1, 0));
await _imapClient.IdleAsync(_tokenDone.Token, _tokenCancel.Token);
}
else
{
await Task.Delay(new TimeSpan(0, 1, 0), _tokenCancel.Token);
await _imapClient.NoOpAsync(_tokenCancel.Token);
}
}
catch (OperationCanceledException)
{
Log.Debug(Identifier + "WaitForNewMessages Idle cancelled...");
throw;
}
catch (Exception ex)
{
Log.Warn(ex, Identifier + "WaitForNewMessages errored out...");
throw;
}
finally
{
_tokenDone?.Dispose();
_tokenDone = null;
}
}
#endregion
#region Command Queue
public async Task<bool> DoCommandAsync(OnImapCommand command, int retries = -1)
{
int attempts = 1;
int errors = 0;
int connections = 0;
_dtLastConnection = DateTime.Now;
DateTime errorStart = DateTime.Now;
bool bReturn = false;
do
{
try
{
if (!IsConnected)
{
Log.Debug(Identifier + "Connection attempt #{0}; retries: {1}; errors: {2}; conns: {3}; total age: {4})",
attempts++,
(retries-- < 0) ? "infinite" : retries.ToString(),
errors,
connections,
DateTime.Now - _dtLastConnection);
await ConnectAsync();
if (!IsConnected)
throw new ServiceNotConnectedException();
Log.Debug($"{Identifier}Server Connection: {IsConnected}");
attempts = 1;
errors = 0;
_dtLastConnection = DateTime.Now;
connections++;
}
Log.Debug("{0}Run IMAP command: {1}", Identifier, command.Method);
await Task.Run(() => command(_imapClient, _imapFolder), _tokenCancel.Token);
Log.Debug(Identifier + "Command completed successfully.");
bReturn = true;
break;
}
catch (OperationCanceledException)
{
Log.Debug(Identifier + "Command operation cancelled...");
break;
}
catch (Exception ex)
{
if (retries == 0 && IsConnected)
Log.Err(ex, "{0}Error IMAP command: {1}", Identifier, command.Method);
if (errors++ == 0)
{
errorStart = DateTime.Now;
Log.Debug(Identifier + "Error detected - attempt immediate reconnection.");
await DisconnectAsync();
}
else
{
TimeSpan errorAge = (DateTime.Now - errorStart);
Log.Debug(Identifier + "Connect failure (attempting connection for {0})", errorAge);
if (errorAge.TotalMinutes < 10)
{
Log.Debug(Identifier + "Cannot connect. Retry in 1 minute.");
await Task.Delay(new TimeSpan(0, 1, 0), _tokenCancel.Token);
}
else if (errorAge.TotalMinutes < 60)
{
Log.Info(Identifier + "Cannot connect. Retry in 10 minutes.");
await Task.Delay(new TimeSpan(0, 10, 0), _tokenCancel.Token);
}
else
{
Log.Err(ex, Identifier + "Cannot connect. Retry in 1 hour (total errors: {0}).", errors);
await Task.Delay(new TimeSpan(1, 0, 0), _tokenCancel.Token);
}
}
}
} while (retries != 0 && _tokenCancel != null && !_tokenCancel.IsCancellationRequested);
return bReturn;
}
public async Task<bool> DoCommandsAsync(int retries = -1)
{
while (_queueCommand.Count > 0 && _tokenCancel != null && !_tokenCancel.IsCancellationRequested)
{
try
{
var command = _queueCommand.Peek();
if (await DoCommandAsync(command, retries))
{
lock (_lock)
_queueCommand.Dequeue();
}
if (_imapClient.IsConnected && !_imapFolder.IsOpen)
_imapFolder.Open(_folderAccess);
}
catch (Exception ex)
{
Log.Warn(ex, Identifier + "DoCommands errored out...");
throw;
}
}
return _queueCommand.Count == 0;
}
public void QueueCommand(OnImapCommand command)
{
lock (_lock)
_queueCommand.Enqueue(command);
_tokenDone?.Cancel();
}
#endregion
#region IMAP Events
private void OnCountChanged(object sender, EventArgs e)
{
var folder = (ImapFolder)sender;
Log.Debug(Identifier + "{0} message count has changed from {1} to {2}.", folder, _numMessages, folder.Count);
if (folder.Count > _numMessages)
{
Log.Debug(Identifier + "{0} new messages have arrived.", folder.Count - _numMessages);
_messagesArrived = true;
_tokenDone?.Cancel();
}
_numMessages = folder.Count;
}
private void OnMessageExpunged(object sender, MessageEventArgs e)
{
var folder = (ImapFolder)sender;
Log.Debug(Identifier + "{0} message #{1} has been expunged.", folder, e.Index);
_numMessages = folder.Count;
}
#endregion
}
}
It is worth studying the “robustness pattern” in the DoCommandAsync code. The most likely reason for an exception to be thrown, provided your own code is well written with error handling, is due to a problem with the server connection. This pattern is meant to allow a daemon to reestablish a connection even if it takes hours to do so. The idea is that on the first error it will immediately reconnect and try again. If there is still a connection issue it will wait 1 minute between retries, then 10 minutes, and then ultimately wait an hour before trying to reconnect and run the command. There is also a way of retrying indefinitely or for a specified number of retries.
It should also be noted that, as in the author’s example, there are two cancellation tokens being used. These can be accessed via the wrapper by calling Stop or Dispose the wrapper instance. When a command is queued we’ll wake up if idling. When a server event is received we should do likewise.
First, let’s demonstrate the simple case of connecting and running an IMAP command (such as deleting an email, searching for or fetching details, or moving a message, etc).
using (var imapClient = new ImapClientEx(_imapUser))
{
imapClient.DoCommandAsync(MoveEmail, 5).Wait();
}
Notice the using statement for scoping the ImapClientEx wrapper. This code is being executed by its own thread, when the command is run this is done in another thread, and the pointer of the function to run is shunted over from your thread to the IMAP thread and executed by the IMAP client. It will automatically connect before running the command. While async is supported in this case we will wait otherwise we will dispose of our IMAP connection too soon.
private void MoveEmail(ImapClient imapClient, IMailFolder imapFolder)
{
}
The command queue takes a delegate with parameters for the MailKit client and folder arguments. It is run by the IMAP client wrapper thread but it is the instance of your object so you have full access to member variables, etc. Again, this is a simple use case but shows how easily the client can connect and run code.
Now let’s move on to the use case where you want to have a long-standing connection to an inbox to monitor new messages. This requires asynchronously launching and storing the IMAP client wrapper. As the client is running it will remain connected and monitor two events as per the author’s example: inbox.CountChanged and inbox.MessageExpunged. By monitoring this we can expose our single event in the wrapper: NewMessage. With the IMAP client running, all we have to do is keep the instance in a member variable to queue additional IMAP commands, receive the NewMessage event, or stop the client when we are done.
protected void ImapConnect()
{
if (_imapClient != null)
{
_imapClient.NewMessage -= IMAPProcessMessages;
_imapClient.Stop();
_imapClient = null;
}
_imapClient = new ImapClientEx(_imapUser);
_imapClient.NewMessage += IMAPProcessMessages;
var idleTask = _imapClient.RunAsync();
_dtLastConnection = DateTime.Now;
}
Now it should be noted that a NewMessage event will fire once the IMAP client connects at startup. This is because our daemon needs to be capable of shutting down and therefore must track the last processed message. The best way to do this is to track the last UID processed. This way, whenever the event is fired, you will just search for new UIDs since the last tracked UID was seen.
private void IMAPProcessMessages(ImapClient imapClient, IMailFolder imapFolder)
{
LogSvc.Debug(this, "IMAP: Checking emails...");
_dtLastConnection = DateTime.Now;
if (_currentUid == 0)
_currentUid = (uint)TaskEmailData.FetchLastUID(_taskType);
LogSvc.Debug(this, "IMAP: Last email index from DB: " + _currentUid.ToString());
int currentIndex = imapFolder.Count - 1;
if (currentIndex >= 0)
{
var range = new UniqueIdRange(new UniqueId((uint)_currentUid + 1), UniqueId.MaxValue);
var uids = imapFolder.Search(range, SearchQuery.All);
if (uids.Count > 0)
{
LogSvc.Info(this, "IMAP: Processing {0} missed emails.", uids.Count);
foreach (var uid in uids)
{
var email = imapFolder.GetMessage(uid);
ImapProcessMessage(imapClient, imapFolder, uid, email);
}
Pulse();
}
else
{
LogSvc.Debug(this, "IMAP: No missed emails.");
}
}
else
{
LogSvc.Debug(this, "IMAP: No missed emails.");
}
}
I won’t show you but, suffice it to say, I have one extra level of redundancy in my daemon where it tracks the connection age and simply recycles the connection after a specified amount of time of inactivity. This was done because, while it was more usable, our old IMAP API became disconnected quite regularly although it falsely reported it was still connected.
Lastly, when the daemon is being shut down for any reason, we need to Stop or Dispose to disconnect and clean up the IMAP connection. The Stop will trigger the cancellation tokens such that the IMAP task thread shuts down in its own time. Calling Dispose directly will synchronously do the same. Also, Dispose can be called repeatedly on the wrapper instance and still be safe to use as it will reconnect as necessary.
_imapClient?.Dispose();
This took a couple of weeks to write and test. My boss was cool about it being shared so I hope to save someone else the pain of writing this from scratch. While MailKit may be on the basic end of the spectrum we built a solution that is very robust and will no doubt have a better up-time metric than before. Many thanks to the author and the MailKit user community for all the insight and knowledge necessary to have written this.
