Introduction
I have been working on a .NET project that needed to send email notifications via an SMTP server. It worked just fine, but sometimes the notifications just did not arrive. The cause was simple, but also annoying: when the SmtpClient could not reach the server (for any reason), there was no means to postpone the operation and try to re-send it later. Thus an exception was thrown, and the message abandoned.
Then I searched the web for a suitable solution, but found only fragments of what I imagined. One of these is the article by Amol Rawke Pune: Use of MSMQ for Sending Bulk Mails [1].
Well, sounds great but has a lots of limitations. But among these, the three major ones:
- Not all fields of the
MailMessage object are passed - The queue consumer "service" is a simple console application
- It has no solution to handle error conditions when sending mail
It is mainly a proof of concept, but I have to admit, this article was the starting point for me. After some Googling, I found a great solution [2] for the first one - which I incorporated in my solution after some minor adaptation. I also used the ideas of Mark Gravell [3] of making the service installer not to depend on installutil.exe.
I also found Rami Vemula's work [4] which is a great demonstration of how MailMessage can be populated with rich content. Thus it was ideal for me to test the abilities of serialization and of the solution itself.
So thanks to these good people for their contribution. All the other things I had to come up with myself... but it was fun, and I hope it will be of use for many.
Background
Message queue is a very basic concept of Windows. The so-called MSMQ (Microsoft Message Queuing) technology is not new, and is a really useful and rock solid IPC (Inter-Process Communication) tool - able to transaction ally communicate even between hosts over a network.
It has its own limitations, first of all, there is a limit of 4MB for messages. Considering the overhead of serialization, Unicode, and so on, we have to deal with even smaller messages. Well, I have to live with this - but I never intended to send large files.
Where Are the Message Queues?
Should be that simple to find them:
- Right click on “My Computer”
- Click on “Manage”
- Expand Services and Application
- Now you can see “Message Queuing”
- Expand “Private Queue”, click on “New Private queue”
Well, it is that simple on a Windows 2008 Server. But you will probably not find it on Windows 7. You will have to install Message Queuing service. It is (a standard) Windows component, so it's not a big deal.
As any other object since Windows NT, message queues have ACL-s (Access Control List). I will come back to this one later, since it can be tricky.
So my piece of code is made of four parts:
- A test application :)
- The library for posting messages on the queue
- A service to consume the queue and to send emails
- A
MailMessage serializer-deserializer
I will start with this later one shortly.
What About the Error Conditions?
Well, my approach was to treat the queue not like a regular FIFO storage - MSMQ has the tools to do it. I am using a property of the message object to set up a TTL-like counter (Time To Live), and a property to store a schedule timestamp. When a message is posted, the TTL counter is set to a user-defined value, and the timestamp to the current one. The service will use the timestamp to pick only the messages that are "expired" - every message starts its life as expired. If a possibly recoverable error condition arises during email sending, a message is re-posted in the queue and re-processed after a predefined waiting time. Every time a message is re-posted, the TTL is decreased. If the TTL is consumed, the message is abandoned. While a message is waiting to be re-processed, other messages can be processed.
Using the Code
Mailmessage Serializer-Deserializer
Serialization is something very important if we want to create distributed applications. I think actually any class should be serializable - but they are not. Well, System.Net.Mail.MailMessage is not serializable by default.
Actually, SmtpClient uses some sort of serialization to save messages in .elm format (see this article), but it would be a lot of work to make MailMessage again from such an RFC compliant file - and a lot of unnecessary overhead too.
The author of [1] created a serializable class that encapsulated a small subset of MailMessage properties. But in the blog-post [2], we can see a complete binary serializable replacement (SerializeableMailMessage) of the original class.
Since MailMessage has lot of properties of different unserializable types, the author implemented the SerializeableMailAddress, SerializeableAttachment, SerializeableAlternateView, SerializeableLinkedResource, SerializeableContentDisposition, SerializeableContentType, and SerializeableCollection classes.
Finally, it is that simple to put the serializabable version of MailMessage in an MSMQ Message:
public void QueueMessage(MailMessage emailMessage)
{
Message message = new Message();
message.Body = new SerializeableMailMessage(emailMessage);
}
and to get it back:
Message message = msgQueue.Receive();
MailMessage mailMessage = (message.Body as SerializeableMailMessage).GetMailMessage();
My version is an adaptation of it to .NET 4.0, I have changed to generic collections, and so on. Great stuff after all, but if you find something missing, he is to blame :). No, actually, since I intend to use this in real-life applications, any comment is appreciated.
Message Sender Library
This piece of code is intended to be called by the application willing to send a mail. It contains a MailSender class. When constructed, it tries to attach to the MSMQ specified in the constructor parameter. If it does not exist, the construction is aborted.
if (!MessageQueue.Exists(queueName))
{
throw new ArgumentException("Cannot instantiate. MSM queue does not exist.");
}
msgQueue = new MessageQueue(queueName, QueueAccessMode.Send);
msgQueue.Formatter = new BinaryMessageFormatter();
msgQueue.DefaultPropertiesToSend.Recoverable = true;
Oh, and by the way, there is logging all around the code: you will need to install NLog if you want to compile the code. Logging is a must even in production environments, and with NLog, you need only to change a configuration file to have the logs where you want them: one logger for all purposes. Really.
logger.Info("Successfully attached to MSM queue: '{0}'", queueName);
But I will omit logging statements in this article when possible.
If construction is successful, you may call the QueueMessage method.
public void QueueMessage(MailMessage emailMessage, int deliveryAttempts = 3)
{
try
{
Message message = new Message();
message.Body = new SerializeableMailMessage(emailMessage);
message.Recoverable = true;
message.Formatter = new BinaryMessageFormatter();
message.AppSpecific = deliveryAttempts;
message.Extension = System.BitConverter.GetBytes(DateTime.UtcNow.ToBinary());
message.Label = Guid.NewGuid().ToString();
message.UseAuthentication = useAuthentication;
msgQueue.Send(message);
}
catch (Exception ex)
{
throw ex;
}
}
The Extension property is a byte array, so I had to convert the current timestamp to bytes. I am using UTC to be sure to have the same time on both sides, if producer and consumer are not on the same host.
A queue can be set up to require authentication to access it. Only in this case are the ACL-s of the queue effective. I have not tested it yet, but it seems that authentication is working only in AD environments. Well, the code is prepared.
The Service
Let's talk first about the service installer. I wrote earlier that I have adopted a solution [3] to get rid of installutil.exe and to have the possibility to run the service application as a regular console application. This little trick is really useful during debugging.
[RunInstaller(true)]
public sealed class MSMQ_MailRelyServiceProcessInstaller : ServiceProcessInstaller
{
public MSMQ_MailRelyServiceProcessInstaller()
{
this.Account = ServiceAccount.NetworkService;
this.Username = null;
this.Password = null;
}
}
[RunInstaller(true)]
public class MSMQ_MailRelyServiceInstaller : ServiceInstaller
{
public MSMQ_MailRelyServiceInstaller()
{
this.DisplayName = "MSMQ Mail processor service";
this.StartType = ServiceStartMode.Automatic;
this.DelayedAutoStart = true;
this.Description =
"Service is designed to send email messages posted in a messaging queue.";
this.ServicesDependedOn = new string[] { "MSMQ" };
this.ServiceName = "MSMQ Mail Rely";
}
}
class Program
{
static void Install(bool undo, string[] args)
{
try
{
Console.WriteLine(undo ? "uninstalling" : "installing");
using (AssemblyInstaller inst = new AssemblyInstaller(typeof(Program).Assembly, args))
{
IDictionary state = new Hashtable();
inst.UseNewContext = true;
try
{
if (undo)
{
inst.Uninstall(state);
}
else
{
inst.Install(state);
inst.Commit(state);
}
}
catch
{
try
{
inst.Rollback(state);
}
catch { }
throw;
}
}
}
catch (Exception ex)
{
Console.Error.WriteLine(ex.Message);
}
}
static int Main(string[] args)
{
bool install = false, uninstall = false, console = false, rethrow = false;
try
{
foreach (string arg in args)
{
switch (arg)
{
case "-i":
case "-install":
install = true; break;
case "-u":
case "-uninstall":
uninstall = true; break;
case "-c":
case "-console":
console = true; break;
default:
Console.Error.WriteLine("Argument not expected: " + arg);
break;
}
}
if (uninstall)
{
Install(true, args);
}
if (install)
{
Install(false, args);
}
if (console)
{
MSMQ_MailRelyService service = new MSMQ_MailRelyService();
Console.WriteLine("Starting...");
service.StartUp(args);
Console.WriteLine("Service '{0}' is running in console mode. " +
"Press any key to stop", service.ServiceName);
Console.ReadKey(true);
service.ShutDown();
Console.WriteLine("System stopped");
}
else if (!(install || uninstall))
{
rethrow = true;
ServiceBase[] services = { new MSMQ_MailRelyService() };
ServiceBase.Run(services);
rethrow = false;
}
return 0;
}
catch (Exception ex)
{
if (rethrow) throw;
Console.Error.WriteLine(ex.Message);
return -1;
}
}
}
So if you want to install the service, just call MSMQ_MailRelyService.exe -i as administrator. To debug, just add -c as a command line argument in the Debug page of the application properties, and run it from the IDE.
Next, I will talk about the service itself. For the complete source code, please browse the code here or download it.
protected override void OnStart(string[] args)
{
try
{
if (!MessageQueue.Exists(settings.QueueName))
{
msgQueue = MessageQueue.Create(settings.QueueName);
msgQueue.Authenticate = settings.UseAuthentication;
msgQueue.Label = "MSMQ Mail Rely message queue";
}
else
{
msgQueue = new MessageQueue(settings.QueueName);
}
msgQueue.Formatter = new BinaryMessageFormatter();
msgQueue.MessageReadPropertyFilter.SetAll();
msgQueue.DenySharedReceive = true;
MSMQMessageProcessor = new Thread(new ThreadStart(this.ProcessMSMQMessages));
MSMQMessageProcessor.Start();
}
catch (Exception ex)
{
throw ex;
}
}
Remember: You may create the queue manually, but do not forget to set up the ACL. By default, everyone can post messages in the queue, but retrieving from it requires rights granted. The creator will have this by default. So if the service creates it, it will work just fine, but you need to take ownership to be able to manipulate the ACL or any other property via the MMC plug-in. If you create the queue, you have to grant necessary rights to the user running the service.
As extension methods are great things, I wrote one to get the next message to be processed. The method will traverse all messages in the queue, and take all eligible ones according to their schedule timestamp. From those eligible ones, the method will pick the oldest one, and return its ID. If no eligible message is found, the method will return null.
public static String GetScheduledMessageID(this MessageQueue q)
{
DateTime OldestTimestamp = DateTime.MaxValue;
String OldestMessageID = null;
using (MessageEnumerator messageEnumerator = q.GetMessageEnumerator2())
{
while (messageEnumerator.MoveNext())
{
DateTime ScheduledTime = DateTime.FromBinary(
BitConverter.ToInt64(messageEnumerator.Current.Extension, 0));
if (ScheduledTime < DateTime.UtcNow)
{
if (ScheduledTime < OldestTimestamp)
{
OldestTimestamp = ScheduledTime;
OldestMessageID = messageEnumerator.Current.Id;
}
}
}
}
return OldestMessageID;
}
The main thread will do the real work, so let's see:
private void ProcessMSMQMessages()
{
try
{
while (true)
{
Message message = msgQueue.Peek();
String ID = msgQueue.GetScheduledMessageID();
if (ID != null)
{
message = msgQueue.ReceiveById(ID);
MailMessage mailMessage =
(message.Body as SerializeableMailMessage).GetMailMessage();
Exception CachedException = null;
RetryReason retry = RetryReason.NoRetry;
try
{
using (var smtpClient = new SmtpClient())
{
smtpClient.Send(mailMessage);
}
}
catch (SmtpFailedRecipientsException ex)
{
CachedException = ex;
for (int i = 0; i < ex.InnerExceptions.Length; i++)
{
SmtpStatusCode status = ex.InnerExceptions[i].StatusCode;
if (status == SmtpStatusCode.MailboxBusy ||
status == SmtpStatusCode.MailboxUnavailable ||
status == SmtpStatusCode.InsufficientStorage)
{
retry = RetryReason.Postmaster;
}
}
}
catch (SmtpException ex)
{
CachedException = ex;
if (ex.InnerException != null)
{
WebExceptionStatus status = (ex.InnerException as WebException).Status;
if (status == System.Net.WebExceptionStatus.NameResolutionFailure ||
status == System.Net.WebExceptionStatus.ConnectFailure)
{
retry = RetryReason.Network;
}
}
}
catch (Exception ex)
{
CachedException = ex;
}
if (CachedException != null)
{
if (retry != RetryReason.NoRetry)
{
if (message.AppSpecific > 0)
{
DateTime OriginalScheduledTime =
DateTime.FromBinary(BitConverter.ToInt64(message.Extension, 0));
int retryDelaySeconds;
if (retry == RetryReason.Network)
{
retryDelaySeconds = settings.NetworkRetryDelay_s;
}
else
{
retryDelaySeconds = settings.PostmasterRetryDelay_s;
}
message.Extension = System.BitConverter.GetBytes(
DateTime.UtcNow.ToUniversalTime().AddSeconds(retryDelaySeconds).ToBinary());
message.AppSpecific--;
msgQueue.Send(message);
}
else
{
logger.ErrorException("Failed to deliver, no more attempts.", CachedException);
}
}
else
{
logger.ErrorException("Failed to deliver, but no use to retry", CachedException);
}
}
}
else
{
Thread.Sleep(settings.SleepInterval);
}
}
}
catch (ThreadAbortException)
{
logger.Info("Thread aborted.");
}
}
It might be worth reviewing error conditions where retries are applied. Actually, it depends on the SMTP server and the addresses.
The service has several settings stored in the app.config. The defaults might be suitable for you, but you may change them as you like.
First of all, we have the queue name. As this is the service, using a local private queue is straightforward. Technically, a public queue could also be used.
<applicationSettings>
<MSMQ_MailRelyService.Properties.Settings>
<setting name="QueueName" serializeAs="String">
<value>.\Private$\EmailQueue</value>
</setting>
This is how long (in milliseconds) the thread sleeps if there was no suitable message to process:
<setting name="SleepInterval" serializeAs="String">
<value>5000</value>
</setting>
Use or not use authentication:
<setting name="UseAuthentication" serializeAs="String">
<value>False</value>
</setting>
The amount of time (in seconds) the postponed delivery attempt should be delayed in case of a network error:
<setting name="NetworkRetryDelay_s" serializeAs="String">
<value>120</value>
</setting>
The delayed time in seconds in case of an SMTP error.
<setting name="PostmasterRetryDelay_s" serializeAs="String">
<value>3600</value>
</setting>
</MSMQ_MailRelyService.Properties.Settings>
</applicationSettings>
Do not forget to set up your SMTP environment in the app.config as well. This is how it can be done with GMail:
<system.net>
<mailSettings>
<smtp deliveryMethod="Network" from="validatedsender@gmail.com">
<network defaultCredentials="false" enableSsl="true"
host="smtp.gmail.com" port="587"
userName="username@gmail.com" password="password"/>
</smtp>
</mailSettings>
</system.net>
See MSDN for future details.
The Test Application
I will not say much about this one, the code will be self-explaining. As I mentioned earlier, I have adapted [4] to have a nearly complete feature-test.
MailSender sender = new MailSender(@".\Private$\EmailQueue");
class Program
{
static void Main(string[] args)
{
MailSender sender = new MailSender(@".\Private$\EmailQueue");
MailMessage m = new MailMessage();
m.From = new MailAddress("sender@mailserver.com", "Sender Display Name");
m.To.Add(new MailAddress("to@mail.com", "To Display Name"));
m.CC.Add(new MailAddress("cc@mail.com", "CC Display Name"));
m.Subject = "Sample message";
m.IsBodyHtml = true;
m.Body = @"<h1>This is sample</h1><a " +
@"href=""http://http://www.codeproject.com"">See this link</a>";
FileStream fs = new FileStream(@"C:\Windows\Microsoft.NET\Framework" +
@"\v4.0.30319\SetupCache\Client\SplashScreen.bmp", FileMode.Open, FileAccess.Read);
Attachment a = new Attachment(fs, "image.bmp", MediaTypeNames.Application.Octet);
m.Attachments.Add(a);
string str = "<html><body><h1>Picture</h1>" +
"<br/><img src=\"cid:image1\"></body></html>";
AlternateView av =
AlternateView.CreateAlternateViewFromString(str, null, MediaTypeNames.Text.Html);
LinkedResource lr = new LinkedResource(@"C:\Windows\Microsoft.NET\Framework" +
@"\v4.0.30319\ASP.NETWebAdminFiles\Images\ASPdotNET_logo.jpg", MediaTypeNames.Image.Jpeg);
lr.ContentId = "image1";
av.LinkedResources.Add(lr);
m.AlternateViews.Add(av);
sender.QueueMessage(m);
}
}
That's all folks.
Points of interest
I think this solution can be used as is in a real-life application. I have tried to deal with as many situations that must be handled as possible. I think MSMQ has even more possibilities to be explored in such an application, like using report typed messages and administration queues to give asynchronous feedback to the sender, introducing some timeouts. As I will test it in an AD environment soon, I will come back and update the article with my findings.
History
- 2012.02.05: First release, but with some improvements during the writing of this article
