Understanding the Insides of the SMTP Mail Protocol: Part 1

• Download source and sample

Introduction

Anyone who has a computer or mobile device has used mail. The mail system is an old, traditional simple protocol. The purpose of this article (Part 1) is to explore the insides of the SMTP protocol and show you how to implement it in C#.

• Part 2 about POP3 protocol
• Part 3 about IMAP protocol
• You can get all libraries from http://https://github.com/higty/higlabo.netstandard

SMTP

In the beginning, mail had two protocols: SMTP and POP3. Afterwards IMAP was added. This article describes SMTP. SMTP is a protocol to send mail. The basic flow is as given below:

• Open connection
• Authenticate
• Hello
• Authenticate
• From
• RcptTo
• Data
• Quit

Open Connection

To communicate to a mail server from the mail client, you must initialize the Socket object. SocketClient.cs exists in HigLabo.Net project.

SocketClient.cs

protected Socket GetSocket()
{
    Socket tc = null;
    IPHostEntry hostEntry = null;
    hostEntry = this.GetHostEntry();
    if (hostEntry != null)
    {
        foreach (IPAddress address in hostEntry.AddressList)
        {
            tc = this.TryGetSocket(address);
            if (tc != null) { break; }
        }
    }
    return tc;
}
private Socket TryGetSocket(IPAddress address)
{
    IPEndPoint ipe = new IPEndPoint(address, this._Port);
    Socket tc = null;

    try
    {
        tc = new Socket(ipe.AddressFamily, SocketType.Stream, ProtocolType.Tcp);
        tc.Connect(ipe);
        if (tc.Connected == true)
        {
            tc.ReceiveTimeout = this.ReceiveTimeout;
            tc.SendBufferSize = this.SendBufferSize;
            tc.ReceiveBufferSize = this.ReceiveBufferSize;
        }
    }
    catch
    {
        tc = null;
    }
    return tc;
}
private IPHostEntry GetHostEntry()
{
    try
    {
        return Dns.GetHostEntry(this.ServerName);
    }
    catch { }
    return null;
}

Now you can send a command with this Socket object. Here is the SMTP command flow to send mail:

This is a response text after opening a connection.

220 mx.google.com ESMTP pp8sm11319893pbb.21

Single line response format is shown here:

[responseCode][whitespace][message]

220 is ServiceReady.

Send hello command:

helo xxx@xxx.com

Request command format is:

[commandName][whitespace][message]

You can get a response text like below:

250-mx.google.com at your service, [61.197.223.240]
mx.google.com at your service, [61.197.223.240]
SIZE 35882577
8BITMIME
AUTH LOGIN PLAIN XOAUTH
ENHANCEDSTATUSCODES

Multiline response format is shown here:

[responseCode]-[message]
[message]
...
[message]

The Response message includes the supported authentication type. This mail server supports AUTH, LOGIN, PLAIN, and XOAUTH.

SmtpCommandResult represents both singleline and multiline response text.

Plain Authenticate

Here is the plain authentication command flow:

You must send a base64 encoded text.

String text = MailParser.ToBase64String
		(String.Format("{0}\0{0}\0{1}", this.UserName, this.Password));

Login Authenticate

Here is the login authentication command flow:

You must send the username and password encoded as base64.

CRAM-MD5 Authenticate

Here is a CRAM-MD5 authentication command flow:

You must send a text created with username, password, challenge text as response from the server.

MailParser.cs

public static String ToCramMd5String(String challenge, String userName, String password)
{
    StringBuilder sb = new StringBuilder(256);
    Byte[] bb = null;
    HMACMD5 md5 = new HMACMD5(Encoding.ASCII.GetBytes(password));
    bb = md5.ComputeHash(Convert.FromBase64String(challenge));
    for (int i = 0; i < bb.Length; i++)
    {
        sb.Append(bb[i].ToString("x02"));
    }
    bb = Encoding.ASCII.GetBytes(String.Format("{0} {1}", userName, sb.ToString()));
    return Convert.ToBase64String(bb);
}

From, RcptTo, Data, Quit

After authentication, you will send the mail command like this:

mail from:xxx@xxx.com

You will get a response like this:

250 2.1.0 OK mt9sm7913789pbb.14

Next, you send the rcpt command with the mail address you want to send the mail to.

rcpt to:yyy@yyy.com

To send to three users, you must send the rcpt command three times. To prepare to send the mail entity body, send the data command.

Data

You will get a response like this:

354 Go ahead mt9sm7913789pbb.14

And send the mail entity data. A line only including period is the end of the mail entity.

Date: Fri, 25 May 2012 14:43:46 +0900
From: <xxx@xxx.com>
Subject: TheTestMail
Content-Transfer-Encoding: Base64
Content-Disposition: inline
X-Priority: 3
To: yyy@xxx.com
Content-Type: text/plain; charset="iso-2022-jp"

GyRCS1xKOCVGJTklSBsoQg==

.

As the last process, you send a quit command to finish the send mail process. And close the connection.

SSL

When you use SSL, you must create a stream object using the SslStream class. Here is the code to create a SslStream object.

this.TcpSocket = this.GetSocket();
SslStream ssl = new SslStream(new NetworkStream(this.Socket), 
		true, this.RemoteCertificateValidationCallback);
ssl.AuthenticateAsClient("myserver.com");

TLS

Here is a diagram of SMTP over TLS process.

After getting a response starttls command, you must create an SslStream object and communicate to it.

SmtpClient.cs

private Boolean StartTls()
{
    SmtpCommandResult rs = null;

    if (this.EnsureOpen() == SmtpConnectionState.Connected)
    {
        rs = this.Execute("STARTTLS");
        if (rs.StatusCode != SmtpCommandResultCode.ServiceReady)
        { return false; }

        this.Ssl = true;
        this._Tls = true;
        SslStream ssl = new SslStream(new NetworkSream(this.Socket)
        , true, this.RemoteCertificateValidationCallback, null);
        ssl.AuthenticateAsClient(this.ServerName);
        this.Stream = ssl;
        return true;
    }
    return false;
}

Implementation by C#

To send mail using HigLabo, you must create instances of the SmtpClient and SmtpMessage objects. The SmtpClient class inherits from the MailClient class, and the SmtpMessage class inherits from the InternetTextMessage class.

The SmtpClient class is designed so that you can send each command to the server using the ExecuteXXX method and easily send mail by calling the SendMail method.

Here is a sample code to send mail using SSL:

using (var cl = new SmtpClient("smtp.gmail.com"))
{
    cl.Port = 465;
    cl.Ssl = true;
    cl.AuthenticateMode = SmtpAuthenticateMode.Auto;
    cl.UserName = "xxx@xxx.com";
    cl.Password = "???????";

    SmtpMessage mg = new SmtpMessage();
    mg.ContentEncoding = Encoding.GetEncoding("iso-8859-1");
    mg.ContentTransferEncoding = TransferEncoding.QuotedPrintable;
    mg.HeaderEncoding = Encoding.GetEncoding("iso-8859-1");
    mg.HeaderTransferEncoding = TransferEncoding.QuotedPrintable;
    mg.Date = DateTime.Now.ToUniversalTime();
    mg["Mime-Version"] = "1.0";
    mg.From = "xxx@xxx.com";
    mg.ReplyTo = "xxx1@xxx.com";
    mg.To.Add(new MailAddress("yyy@yyy.com"));
    mg.Subject = "Sample mail";
    mg.BodyText = "This is a sample mail!";
    SendMailResult rs = cl.SendMail(mg);
    if (rs.SendSuccessful == true)
    {
        //Do something. ex)show a message
    }
}

Quite intuitive to write code!

Here is a sample code to send mail using TLS:

using (var cl = new SmtpClient("smtp.gmail.com"))
{
    cl.Port = 587;
    cl.Tls = true;
    ...same as above code
}

Send mail with an attachment file:

using (var cl = new SmtpClient("smtp.gmail.com"))
{
    //Set property of SmtpClient...
    SmtpMessage mg = new SmtpMessage();
    //Set property of SmtpMessage...

    //Attachment file
    SmtpContent ct = new SmtpContent();
    ct.LoadFileData("C:\\MyPicture.png");
    //Load Html format text as file
    //ct.LoadHtml("<html>....</html>");
    //Load text data as file
    //ct.LoadText("This is a text file.");
    //Load from byte data 
    //ct.LoadData(new Byte[0]);
    mg.Contents.Add(ct);

    SendMailResult rs = cl.SendMail(mg);
    if (rs.SendSuccessful == true)
    {
        //Do something. ex)show a complete message
    }
}

You can attach a file by loading from filepath, text, HTML, or raw byte data.

Inside the SendMail Method

Here is the code for the SendMail method:

public SendMailResult SendMail(String from, String to, String cc, String bcc, String text)
{
    List<MailAddress> l = new List<MailAddress>();
    String[] ss = null;

    ss = to.Split(',');
    for (int i = 0; i < ss.Length; i++)
    {
        if (String.IsNullOrEmpty(ss[i]) == true)
        { continue; }
        l.Add(MailAddress.Create(ss[i]));
    }
    ss = cc.Split(',');
    for (int i = 0; i < ss.Length; i++)
    {
        if (String.IsNullOrEmpty(ss[i]) == true)
        { continue; }
        l.Add(MailAddress.Create(ss[i]));
    }
    ss = bcc.Split(',');
    for (int i = 0; i < ss.Length; i++)
    {
        if (String.IsNullOrEmpty(ss[i]) == true)
        { continue; }
        l.Add(MailAddress.Create(ss[i]));
    }
    return this.SendMail(new SendMailCommand(from, text, l));
}
public SendMailResult SendMail(String from, SmtpMessage message)
{
    return this.SendMail(new SendMailCommand(from, message));
}
public SendMailResult SendMail(SmtpMessage message)
{
    return this.SendMail(new SendMailCommand(message));
}
public SendMailListResult SendMailList(IEnumerable<SmtpMessage> messages)
{
    List<SendMailCommand> l = new List<SendMailCommand>();
    foreach (var mg in messages)
    {
        l.Add(new SendMailCommand(mg));
    }
    return this.SendMailList(l.ToArray());
}
public SendMailResult SendMail(SendMailCommand command)
{
    var l = this.SendMailList(new[] { command });
    if (l.Results.Count == 1)
    {
        return new SendMailResult(l.Results[0].State, command);
    }
    return new SendMailResult(l.State, command);
}
public SendMailListResult SendMailList(IEnumerable<SendMailCommand> commandList)
{
    SmtpCommandResult rs = null;
    Boolean HasRcpt = false;

    if (this.EnsureOpen() == SmtpConnectionState.Disconnected)
    { return new SendMailListResult(SendMailResultState.Connection); }

    if (this.State != SmtpConnectionState.Connected &&
        this.State != SmtpConnectionState.Authenticated)
    {
        return new SendMailListResult(SendMailResultState.InvalidState);
    }
    if (this.State != SmtpConnectionState.Authenticated)
    {
        rs = this.ExecuteEhloAndHelo();
        if (rs.StatusCode != SmtpCommandResultCode.RequestedMailActionOkay_Completed)
        { return new SendMailListResult(SendMailResultState.Helo); }
        if (this._Tls == true)
        {
            if (this.StartTls() == false)
            { return new SendMailListResult(SendMailResultState.Tls); }
            rs = this.ExecuteEhloAndHelo();
            if (rs.StatusCode != SmtpCommandResultCode.RequestedMailActionOkay_Completed)
            { return new SendMailListResult(SendMailResultState.Helo); }
        }
        if (SmtpClient.NeedAuthenticate(rs.Message) == true)
        {
            if (this.Authenticate() == false)
            { return new SendMailListResult(SendMailResultState.Authenticate); }
        }
    }

    List<SendMailResult> results = new List<SendMailResult>();

    foreach (var command in commandList)
    {
        rs = this.ExecuteMail(command.From);
        if (rs.StatusCode != SmtpCommandResultCode.RequestedMailActionOkay_Completed)
        {
            results.Add(new SendMailResult(SendMailResultState.MailFrom, command));
            continue;
        }
        List<MailAddress> mailAddressList = new List<MailAddress>();
        foreach (var m in command.RcptTo)
        {
            String mailAddress = m.ToString();
            if (mailAddress.StartsWith("<") == true && mailAddress.EndsWith(">") == true)
            {
                rs = this.ExecuteRcpt(mailAddress);
            }
            else
            {
                rs = this.ExecuteRcpt("<" + mailAddress + ">");
            }
            if (rs.StatusCode == SmtpCommandResultCode.RequestedMailActionOkay_Completed)
            {
                HasRcpt = true;
            }
            else
            {
                mailAddressList.Add(m);
            }
        }
        if (HasRcpt == false)
        {
            results.Add(new SendMailResult
			(SendMailResultState.Rcpt, command, mailAddressList));
            continue;
        }
        rs = this.ExecuteData();
        if (rs.StatusCode == SmtpCommandResultCode.StartMailInput)
        {
            this.SendCommand(command.Text + MailParser.NewLine + ".");
            rs = this.GetResponse();
            if (rs.StatusCode == SmtpCommandResultCode.RequestedMailActionOkay_Completed)
            {
                results.Add(new SendMailResult
			(SendMailResultState.Success, command, mailAddressList));
                this.ExecuteRset();
            }
            else
            {
                results.Add(new SendMailResult
			(SendMailResultState.Data, command, mailAddressList));
            }
        }
        else
        {
            results.Add(new SendMailResult
			(SendMailResultState.Data, command, mailAddressList));
        }
    }
    rs = this.ExecuteQuit();
    if (results.Exists(el => el.State != SendMailResultState.Success) == true)
    {
        return new SendMailListResult(SendMailResultState.SendMailData, results);
    }
    return new SendMailListResult(SendMailResultState.Success, results);
}

There are two requirements for the SmtpClient class and the SendMail method:

1. Send multiple mails with once open connection
2. Low level layer functionality about the send mail command

The SendMailCommand object is created inside the SendMail method.

From is a property that includes the sender mail address. The (xxx@xxx.com).RcptTo property has the destination address list. The Text property will return raw text data that is sent by the Data command.

The value of the Text property of the SendMailCommand object:

Date: Fri, 25 May 2012 14:43:46 +0900
From: <xxx@xxx.com>
Subject: TheTestMail
Content-Transfer-Encoding: Base64
Content-Disposition: inline
X-Priority: 3
To: yyy@xxx.com
Content-Type: text/plain; charset="iso-2022-jp"

GyRCS1xKOCVGJTklSBsoQg==

.

You can see the actual code that represents the SMTP protocol inside the SendMailList(IEnumerable<SendMailCommand> commandList) method overload version.

Open connection,Helo,Tls(optional),Authenticate(optional),MailFrom,RcptTo(multiple),Data,Quit. Exactly the same as the diagram.

The method will return the SendMailListResult object.

The Results property is a list of SendMailResult objects.

The SendMailResult object represents the result of the send mail process. The SendSuccessful property shows you if the process is completely successful or not. InvalidMailAddressList is a list that the RcptTo command was failed to. You can find which process failed using the State property. You can get the SendMailCommand object using the Command property that executes inside the SendMailList method.

History

• 2012/05/31: First post
• 2012/06/07: Modified link
• 2012/07/06: Modified source code and article