Understanding the Insides of the POP3 Mail Protocol: Part 2

• Download source and sample

Introduction

Everyone who has a computer or mobile device has used mail. The mail system is an old, traditional simple protocol. The purpose of this article (Part 2) is to explore the insides of the POP3 protocol and show you how to implement it with C#.

• Part 1 about SMTP protocol
• Part 3 about IMAP protocol

You can get the complete library from https://github.com/higty/higlabo.netstandard.

POP3

You can select two protocols to receive mail from a mailbox. This article describes the POP3 protocol. Here is the basic flow to receive mail:

• Open connection
• Authenticate
• List
• Retr
• Quit

Open Connection

See Open connection section of this post.

At first, you must authenticate the mailbox with your username and password. Here is a pop authenticate flow diagram:

public Boolean AuthenticateByPop()
{
    String s = "";
    if (this.EnsureOpen() == Pop3ConnectionState.Connected)
    {
        s = this.Execute("user " + this.UserName, false);
        if (s.StartsWith("+OK") == true)
        {
            s = this.Execute("pass " + this.Password, false);
            if (s.StartsWith("+OK") == true)
            {
                this._State = Pop3ConnectionState.Authenticated;
            }
        }
    }
    return this._State == Pop3ConnectionState.Authenticated;
}

This is the response text of the POP3 protocol:

+OK send PASS please

This is the response text when failed:

-ERR [AUTH] Username and password not accepted.

Single line response format is here:

When success:

+OK[whitespace][message]

When failed:

-ERR[whitespace][message]

POP3 authentication is pretty easy, but low security. To enhance the security level, you can use A-POP authentication.

public Boolean AuthenticateByAPop()
{
    String s = "";
    String TimeStamp = "";
    Int32 StartIndex = 0;
    Int32 EndIndex = 0;

    if (this.EnsureOpen() == Pop3ConnectionState.Connected)
    {
        s = this.Execute("user " + this.UserName, false);
        if (s.StartsWith("+OK") == true)
        {
            if (s.IndexOf("<") > -1 &&
                s.IndexOf(">") > -1)
            {
                StartIndex = s.IndexOf("<");
                EndIndex = s.IndexOf(">");
                TimeStamp = s.Substring(StartIndex, EndIndex - StartIndex + 1);
                s = this.Execute("pass " + MailParser.ToMd5DigestString
                                 (TimeStamp + this.Password), false);
                if (s.StartsWith("+OK") == true)
                {
                    this._State = Pop3ConnectionState.Authenticated;
                }
            }
        }
    }
    return this._State == Pop3ConnectionState.Authenticated;
}

Password Security

POP3 authentication is low-security since the username and password are send as raw text into the network. Malicious software or people can sniff your password and use it maliciously. A-POP is different from POP3 since the sent password is encoded by MD5Digest with the challenge text received from the server. The server compares the saved challenge text and text received from the client. There is no way to receive the actual password text. So A-POP is more secure than POP authentication.

Get Mail List

After authentication, you can get the mail list using the LIST command. The response text of the LIST command is as below:

+OK 250 messages (338834 bytes)
1 2762
2 1704
3 2259
4 1748
5 1799
6 1802
7 1109
8 1701
9 1075
10 1130
...
245 1310
246 1725
247 1092
248 1237
249 1161
250 1099
.

The format is like below:

+OK [message]
[mail index][white space][mail size]
[mail index][white space][mail size]
...repeat...
[mail index][white space][mail size]
[period]

You can get it by calling the ExecuteList method of the Pop3Client class:

public List<ListCommandResult> ExecuteList()
{
    ListCommand cm = new ListCommand();
    List<ListCommandResult> l = new List<ListCommandResult>();
    StringReader sr = null;
    String s = "";
    String line = "";

    this.CheckAuthenticate();
    s = this.Execute(cm);
    this.CheckResponseError(s);
            
    sr = new StringReader(s);
    while (sr.Peek() > -1)
    {
        line = sr.ReadLine();
        if (line == ".")
        { break; }
        if (line.StartsWith("+OK", StringComparison.OrdinalIgnoreCase) == true)
        { continue; }

        l.Add(new ListCommandResult(line));
    }
    return l;
}

This result is represented by the ListCommandResult class.

Get Message

You can get the actual mail message data by calling the GetMessage method of the Pop3Client class. Here is some sample code to receive a mail message:

private static void Pop3MailReceive()
{
    MailMessage mg = null;

    using (Pop3Client cl = new Pop3Client("pop.gmail.com"))
    {
        cl.Port = 995;
        cl.UserName = "xxxxx";
        cl.Password = "yyyyy";
        cl.Ssl = true;
        cl.AuthenticateMode = Pop3AuthenticateMode.Auto;
        var bl = cl.Authenticate();
        if (bl == true)
        {
            var l = cl.ExecuteList();
            for (int i = 0; i < l.Count; i++)
            {
                mg = cl.GetMessage(l[i].MailIndex);
            }
        }
    }
}

You must pass an index to the GetMessage method larger than 1. Please take care that the first value is 1, not zero.

The MailMessage class schema is shown here:

The MailMessage class inherits from InternetTextMessage.

You can get subject, body text using this sample code:

String subject = mg.Subject;
String body = mg.BodyText;

You can get raw text using the Data property that is defined on the InternetTextMessage class.

String rawText = mg.Data;

Attached File

Mail sometimes includes an attached file. You can get the attached files using the Contents property of the MailMessage class.

var mg = cl.GetMessage(1);
foreach (var ct in mg.Contents)
{
    ct.DecodeData("C:\\" + ct.FileName);
}

You can save attached files by calling the DecodeData method of the MailContent object.

In rare cases, MailMessage itself is an attached file. You can confirm whether MailMessage is an attached file or not by checking the IsAttachment property of the MailMessage object.

var mg = cl.GetMessage(1);
if (mg.IsAttachment == true)
{
    mg.DecodeData("C:\\MyFile.png");
}

HTML Mail

HTML mail is a text mail. You can get HTML text using the BodyText property of the MailMessage or MailContent object.

String htmlText = mg.BodyText;
foreach (var ct in mg.Contents)
{
    if (ct.IsHtml == false) { continue; }
    String htmlText1 = ct.BodyText;
}

To show HTML to your UI component, you must set this htmlText to your control. In a Windows Forms or WPF application, use the browser control to show it. In a web application, you can select some control, HtmlGenericControl, Label control, etc. Here is some sample code for a web application:

protected void Page_Load(object sender, EventArgs e)
{
    MailMessage mg = null;
    String htmlText = "";

    using (Pop3Client cl = new Pop3Client("pop.gmail.com"))
    {
        cl.Port = 995;
        cl.UserName = "xxxxx";
        cl.Password = "yyyyy";
        cl.Ssl = true;
        cl.AuthenticateMode = Pop3AuthenticateMode.Auto;
        if (cl.Authenticate() == false) { return; }
        if (mg.IsHtml == true)
        {
            htmlText = mg.BodyText;
        }
        else 
        {
            foreach (var ct in mg.Contents)
            {
                if (ct.IsHtml == true) 
                {
                    htmlText = ct.BodyText;
                    break; 
                }
            }
        }
    }
    this.Label1.Text = htmlText;
}

.eml file

Outlook and other mail client programs would create a .eml file. Actually, these files are simple text files, like below:

Return-Path: <xxxxx@gmail.com>
Received: from  (bf240.xxx.xxxx.com. [61.197.23.240])
        by mx.google.com with ESMTPS id pb4sm20464671pbc.55.2012.05.13.17.56.12
        (version=SSLv3 cipher=OTHER);
        Sun, 13 May 2012 17:56:13 -0700 (PDT)
Message-ID: <4fc05e2d.e4a9440a.302b.ffffe399@mx.google.com>
Date: Mon, 14 May 2012 00:56:10 +0900
From: xxxxx@gmail.com
Subject: Test
Content-Transfer-Encoding: Quoted-Printable
Content-Disposition: inline
Mime-Version: 1.0
Reply-To: xxxxx@hotmail.com
X-Priority: 3
To: yyyyy@gmail.com
Content-Type: text/plain; charset="iso-8859-1"

=E4=C4=F6=D6=FC=DC=DF


.

You can create a MailMessage object from text data.

String text = File.ReadAllText("C:\\MyMail.eml");
var mg = new MailMessage(text);

Delete Mail

Here is the delete process in POP3:

You can see that the response text of the DELE command is like below:

+OK marked for deletion

The Dele command marks the mail that you indicate as candidate for deletion. These marked mails will be deleted when you send the quit command. You can delete a mail by using the DeleteMail method of the Pop3Client object.

protected void Button1_Click(object sender, EventArgs e)
{
    MailMessage mg = null;
    String htmlText = "";

    using (Pop3Client cl = new Pop3Client("pop.gmail.com"))
    {
        cl.Port = 995;
        cl.UserName = "xxxxx";
        cl.Password = "yyyyy";
        cl.Ssl = true;
        cl.AuthenticateMode = Pop3AuthenticateMode.Auto;
        
        cl.DeleteMail(1, 2, 3);
    }
}

There is no authenticate required since all processes (open connection, authenticate, dele, quit) will automatically execute inside the DeleteMail method. Here is an implementation of the DeleteMail method:

public Boolean DeleteMail(params Int64[] mailIndex)
{
    DeleCommand cm = null;
    String s = "";

    if (this.EnsureOpen() == Pop3ConnectionState.Disconnected) { return false; }
    if (this.Authenticate() == false) { return false; }
    for (int i = 0; i < mailIndex.Length; i++)
    {
        cm = new DeleCommand(mailIndex[i]);
        s = this.Execute(cm);
        if (MailParser.IsResponseOk(s) == false) { return false; }
    }
    this.ExecuteQuit();
    return true;
}

You can see the open connection, authenticate, send dele command, and finally send quit command inside of the DeleteMail method.

Mail Format

Here is the actual raw text data of a POP3 message:

Return-Path: <xxxxx@gmail.com>
Received: from  (bf240.xxx.xxxx.com. [61.197.23.240])
        by mx.google.com with ESMTPS id pb4sm20464671pbc.55.2012.05.13.17.56.12
        (version=SSLv3 cipher=OTHER);
        Sun, 13 May 2012 17:56:13 -0700 (PDT)
Message-ID: <4fc05e2d.e4a9440a.302b.ffffe399@mx.google.com>
Date: Mon, 14 May 2012 00:56:10 +0900
From: xxxxx@gmail.com
Subject: Test
Content-Transfer-Encoding: Quoted-Printable
Content-Disposition: inline
Mime-Version: 1.0
Reply-To: xxxxx@hotmail.com
X-Priority: 3
To: yyyyy@gmail.com
Content-Type: text/plain; charset="iso-8859-1"

=E4=C4=F6=D6=FC=DC=DF


.

Message has two parts: header and body. Red is header and blue is body.

Header and body are separated by a blank line. Header has a list of values with a key value pair.

[key][colon][value]

The multi-line format is like below:

[key][colon][value]
[tab or white space][value]
...repeat...
[tab or white space][value]

The characters you can use in message body is 7bit char only. If you want to send "ありがとうございます" (it is a Japanese character that means thank you), you must encode it using Quoted-Printable or Base64 encoding. The mail format has been very complex since its history.

Important RCFs are 822, 2045-2049, 2231, 2282, 2407. Please check more details if you want to know.

Manage Read or Unread With the UIDL Command

POP3 does not have functionality to manage read state. You must use your own implementation to manage read or not. You can use the UIDL command to achieve read state management. Here is the response text of a UIDL command:

+OK
1 GmailId136106770caf88db
2 GmailId136106acda5c2ff1
3 GmailId136106ae7f1b631b
4 GmailId136106b691ad4a79
5 GmailId136106ba4df09d7f
6 GmailId136106bdfbd92cdb
7 GmailId136106c5168c87fe
...
245 GmailId13610723f4c51313
246 GmailId13610724acde6b5b
247 GmailId13610724dc2d28b0
248 GmailId136107261a4d3b0c
249 GmailId1361072652837e0f
250 GmailId136107269afc66ae
.

This format is like below:

+OK
[mail index][white space][unique id]
...repeat
[mail index][white space][unique id]
[period]

The UidlCommandResult class represents this response text:

To manage read state, you save this UID list for the next mail. I'll show you the concept code here:

private static void Pop3MailReceiveUnRead()
{
    MailMessage mg = null;

    using (Pop3Client cl = new Pop3Client("pop.gmail.com"))
    {
        cl.Port = 995;
        cl.UserName = "xxxxx";
        cl.Password = "yyyyy";
        cl.Ssl = true;
        cl.AuthenticateMode = Pop3AuthenticateMode.Auto;
        var bl = cl.Authenticate();
        if (bl == true)
        {
            var ul = cl.ExecuteUidl();
            String path = "C:\\MailUidl.txt";
            List<String> l = new List<String>();
            //Get read mail index list from file
            if (File.Exists(path) == true)
            {
                l.AddRange(File.ReadAllLines(path));
            }
            List<UidlCommandResult> unreadList = new List<UidlCommandResult>();
            for (int i = 0; i < ul.Count; i++)
            {
                if (l.Contains(ul[i].Uid) == true) { continue; }
                unreadList.Add(ul[i]);
            }
            //Save UIDL data to text file
            StringBuilder sb = new StringBuilder(ul.Count * 10);
            for (int i = 0; i < ul.Count; i++)
            {
                sb.AppendLine(ul[i].Uid);
            }
            File.WriteAllText(path, sb.ToString());
        }
    }
}

You can save UIDL data to a database or XML file depending on your requirements.

Forward Mail by SMTP

If you want to transfer your received mail, you can use CreateSmtpMessage method of MailMessage object.
Here is a sample code to do it:

private static void Pop3MailReceiveUnRead()
{
    MailMessage mg = null;

    using (Pop3Client cl = new Pop3Client("pop.gmail.com"))
    {
        cl.Port = 995;
        cl.UserName = "xxxxx";
        cl.Password = "yyyyy";
        cl.Ssl = true;
        cl.AuthenticateMode = Pop3AuthenticateMode.Auto;
        var bl = cl.Authenticate();
        if (bl == true)
        {
            mg = cl.GetMessage(1);
            SmtpMessage smg = mg.CreateSmtpMessage();
            // you can send smg by SmtpClient object
        }
    }
}

And you can send SmtpMessage class by SmtpClient class.
You can learn how to send a mail by SmtpClient class by reading the following article: Understanding the insides of the SMTP Mail protocol: Part 1.

Next article for IMAP: Understanding the insides of the IMAP Mail protocol: Part 3.

History

• 14^th June, 2012: First post
• 6^th July, 2012: Modified source code and article
• 24^th July, 2012: Modified article and added CreateSmtpMessage section
• 10^th December, 2012: Modified article