Emails Extractor

• Download demo - 579.2 KB
• Download source - 415.4 KB

Introduction

There are many companies/universities web sites contain a list of every employer working in this company, this list contains the name, email, telephone and even address for every employee .As an agent looking for cooperation with this company/university you need to add these people in your yahoo list to contact them or to ask about any information could help you, if they are interest in your product or if they are not?.

But how can you add them in your Email list?, this would take a lot of time and effort you may have not, Because you have to enter this person Name (First, Middle and last), Emails, Telephone and even there genders, Yes that’s right what if your product is only for women or only for men?, this mean that you will need more time to figure out if this person is man or women!!!.

Email Extractor is a simple program I have started when I faced this problem. by taking an advantage of what already available of Regex expressions on internet this program can scan a field of text looking for people and extract their names, telephone, emails, even their gender and a lot of other things you may need to know about this person to start a contact him.

This program could not just extract this person data but also order his data in a DataGridView control which contains rows for every detail available about this person , and over this it can save this data as a “.CSV” file which yahoo mail accept as importing file type to extract contacts and inserting these contacts in your mail. It also enable you specify what kind of data you really need by defining your own Regex expressions, you can save this expressions in a database under a specific Group name so you can use them after.

Handling data by executing Regex Expressions is also available to maintain and prepare mixed data to extract the specific data belongs to one person programmatically.

Extracting data belongs to one person from a pool of text is almost impossible, so every block of data belongs to one person must be separated from other data by more than two spaces “\n”, This will enable the code to split data into blocks every block belongs to a specific person, and also you can specify a Regex expression for text-block-splitting.

Background

Yahoo mail is one of the most mail provider popular in the internet it provide many tools to handle emails, yahoo also provide an important tool ,this tool is importing contacts ,which provide the ability to import contacts from other mail contact (yahoo, Gmail,…) or from files like CSV files.

How to add A Comma-separated values file (.CSV) to your Yahoo mail contact?

1. Open your yahoo mail.

2. Press “contacts”.

3. Add a new contact list.

4. Press import contacts.

5. Choose “others” to specify that you will import contacts from another method.

6. For step 1 unselect another yahoo account.

7. Select “a desktop email program (Outlook, Apple Mail, etc...)”.

8. Browse to the specific .CSV file you need to add.

9. Press “continue”. 

What Is Regular Expressions?

A regular expression is a set of characters that specify a pattern. The term "regular" has nothing to do with a high-fiber diet. It comes from a term used to describe grammars and formal languages.

Regular expressions are used when you want to search for specify lines of text containing a particular pattern. Most of the UNIX utilities operate on ASCII files a line at a time. Regular expressions search for patterns on a single line, and not for patterns that start on one line and end on another.

It is simple to search for a specific word or string of characters. Almost every editor on every computer system can do this. Regular expressions are more powerful and flexible. You can search for words of a certain size. You can search for a word with four or more vowels that end with an "s". Numbers, punctuation characters, you name it, a regular expression can find it. What happens once the program you are using find it is another matter. Some just search for the pattern. Others print out the line containing the pattern. Editors can replace the string with a new pattern. It all depends on the utility.

Regular expressions confuse people because they look a lot like the file matching patterns the shell uses. They even act the same way--almost. The square brackers are similar, and the asterisk acts similar to, but not identical to the asterisk in a regular expression. In particular, the Bourne shell, C shell, find, and cpio use file name matching patterns and not regular expressions.

Remember that shell meta-characters are expanded before the shell passes the arguments to the program. To prevent this expansion, the special characters in a regular expression must be quoted when passed as an option from the shell. You already know how to do this because I covered this topic in last month's tutorial.

You can know More about Regex Expressions here.

Search for People:

This is the main method for this program, it first splits text data derived from main RichtextBox(MixedEmailSRichtextBox) , by using the containing 2 spaces rule ,as different data related to different people must be preceded with spaces ([\n]{2} ,and then it import them to GetPeopleFromBlock Method which perform different Regex expressions to extract data related to specific person like Email, Name, Telephone number,….

  private void SearchForPeople()
        {
            WrongRegex.Clear();
            string Block = MixedEmailSRichtextBox.Text;  //control contains text data for different people
            string Regex = @"(\n){2,}(\w)*?";
            if (BlockSplitingText.Enabled == true && BlockSplitingText.Text.Length > 0)
                Regex = BlockSplitingText.Text;
            if (BlockSplitingText.Enabled == true && BlockSplitingText.Text.Length > 0)
                Regex = BlockSplitingText.Text;
            List<Person> People = GetPeopleFromBlock(Block, Regex);
            this.EmailsdataGridView.DataSource = PeopleTable(People, GetAdditionalRegexfrom(RegexDataGridView));
        
        }
 
        static List<Person> GetPeopleFromBlock(string Data,string RegexSplitting)
        {
            List<Person> MyPData = new List<Person>();
            try
            {
                string[] M = Regex.Split(Data, RegexSplitting); ;
 
                foreach (var Pdata in M)
                {
                    if (Pdata.Length > 20)
                    {
                        Person newPerson = new Person(Pdata);
 
                        MyPData.Add(newPerson);
                    }
                }
            }
            catch (Exception e)
            {
 
                MessageBox.Show(e.Message);
            }
            
            return MyPData;
        }  

People Table function:

This Method import the list of persons loaded with their block of data and then perform GetPersonFromBlock method which execute the already existed Regex expressions existed in the class and the user defined Regex expressions “AdditionalRegex”.

private DataTable PeopleTable(List<Person> People, List<AdditionalRegexExpression> AdditionalRegex)
        {
 
            DataTable PeopleTable = new DataTable();
            AddCoulmnsForPeopleTable(PeopleTable);
            for (int i = 0; i < People.Count; i++)
            {
                DataRow NewPerson = PeopleTable.NewRow();
                List<PersonData> newPErsonData =People[i].GetPersonFromBlock(AdditionalRegex,NoDataWithoutEmail);
                for (int P = 0; P < newPErsonData.Count; P++)
                {
                    if (newPErsonData[P].dataType == null)
                        continue;
                    string ColumnName = newPErsonData[P].dataType.ToString().Replace('_', ' ');
                    if(ColumnName=="Email Address")
                        ColumnName="E-mail Address";
                    if (ColumnName == "Email Display Name")
                        ColumnName = "E-mail Display Name";
                string AlreadyExistedData = NewPerson[ColumnName].ToString();//if there are existed data already
                NewPerson[ColumnName] = AlreadyExistedData + CleanString(newPErsonData[P].Data.ToString());//in case if data contains ',' this will confuse excel program 
                }
                if(newPErsonData.Count>0)
                  PeopleTable.Rows.Add(NewPerson);
            }
            return PeopleTable;
        }
 
         static string CleanString(string X)
        {
            
           return  Regex.Replace(X,RegexExpressionCollection.CleanStringForExcel," "); 
        }
 
 

The .CSV files exporting problem’s solution:

As you have noticed in the previous code there are method called Cleanstring ,this method play an important rule ,as it remove any sign could confuse the .csv reader program (excel,..) while he is ordering cell values in the rows,it perform a specific regex expression to remove coma –which .csv depending on as cell separating sign- and other signs.

public static string CleanStringForExcel { get { return @"\n|\s|\t|\f|\v|\e|,"; } } 

Class Person:

This class gets the block data in the constructor and performs the Regex expressions according to the methods which are calling.it also contain an enumaeration for some of the columns as a data type enum ,which can specify

class Person
        {
           public enum DataType
            {
                First_Name, Middle_Name, Last_Name, Department, Job_Title, Business_State, Home_Postal_Code, Business_Phone, Home_Phone, Birthday, Email_Address, Email_Display_Name,  Gender,Language, Notes, Web_Page, Business_Fax
            }
          
          
            string DataBlock { get; set; } 
            public Person(string DataBlock)  //gets the data which will be use in data exctraction
            {
                this.DataBlock = DataBlock;
            }
            public PersonData getEmailPersonFromBlock()
            {
 
 
                List<string> Emails = new List<string>();
                Regex emailRegex = new Regex(RegexExpressionCollection.Email,
                       RegexOptions.IgnoreCase);
                //find items that matches with our pattern
                MatchCollection emailMatches = emailRegex.Matches(this.DataBlock);
                foreach (Match emailMatch in emailMatches)
                {
                    Emails.Add(emailMatch.Value);
                }
                //retutn OnlyOneEmail
                if(emailMatches.Count==0)
                    return new PersonData(DataType.Email_Address, "");
                return new PersonData(DataType.Email_Address, Emails[0]);
            }
            public PersonData Gender()
            {
                string Name = GetPersonNameFromBlock()[0].Data;//just First Name;
                return Gender(Name);
            
            }
            // other Regex performing methods
            //.
            //.
            //.
            //.
  

Person class contain GetPersonFromBlock which perform all the Regex Expressions available at once and extract the needed data as Person DataType.

 public List<PersonData> GetPersonFromBlock(List<AdditionalRegexExpression> AddationalRegex, bool EmailIsMainData)
            {
 
                PersonData PEmailData = getEmailPersonFromBlock();
                if (!Regex.IsMatch( PEmailData.Data.ToString(),RegexExpressionCollection.Email)&& EmailIsMainData)//if this doesn't match Email Catagory and Email is Needed return empty
                    return new List<PersonData>();
                List<PersonData> PNameData = new List<PersonData>();
                if (PEmailData.Data != "")
                  PNameData =  GetPersonNameFromBlock(PEmailData.Data.ToString()); //if Name is Empty Take Email first part as name
                PersonData PPhone = GetPhonePersonFormBlock();
                PersonData PbirthDaY = GetPhonePersonBirthDate();
                PersonData PEMialDisplayName = new PersonData(); 
                    if(PNameData.Count>0)
                 PEMialDisplayName = new PersonData(DataType.Email_Display_Name, PNameData[0].Data);
                PersonData PPostalCode = GetPersonPostalCodeFromBlocK();
 
                PersonData Gender = new PersonData(); 
                if(PNameData.Count>0)
                    Gender=  this.Gender(PNameData[0].Data);
                PersonData Notes = new PersonData(DataType.Notes,this.DataBlock); //just in Case The Data Was Wrong U have to consider that every thing beside Email is Notes
                PersonData WebPage = GetPersonWebPageFromBlock();
                PersonData Business_Phone = GetPersonBussinessPhone();
                PersonData Lang = GetPersonLanguage();
                PersonData Business_Fax = getPersonBusinessFax();
                PersonData Business_State = GetPersonBusiness_State();
                PersonData Jop_Tile = new PersonData(DataType.Job_Title, Business_State.Data);
                PersonData Department = GetPersonDepartment();
                List<PersonData> AddaationalData = GetAddationalData(AddationalRegex);
                List<PersonData> TotalData = new List<PersonData>();
                TotalData.AddRange(PNameData);
                TotalData.AddRange(new List<PersonData> { Business_Fax, PEmailData, PPhone, PbirthDaY, PEMialDisplayName, PPostalCode, Gender, Notes, WebPage, Business_Phone, Lang, Business_State, Jop_Tile, Department });
                TotalData.AddRange(AddaationalData);//it must be at last becuase it maybe erase data taken by another regex
                return TotalData;
 
            }
            private string[] StringSplittededBySpace(string Data)
            {
                return Regex.Split(Data, @"[\s|\.|\,|\-]+?");
            }
            static List<int> WrongRegex = new List<int>();
 
 

This is very important as it perform the Regex expressions added the Regex Expression table which has been defined by user.

   public List<PersonData> GetAddationalData(List<AdditionalRegexExpression> AdditionalRegex)
            {
                List<PersonData> AdditionalData = new List<PersonData>();
                for (int i = 0; i < AdditionalRegex.Count; i++)
                {
                    if (WrongRegex.Contains(i))
                        continue;
                    try
                    {
                        MatchCollection RegexMatches = Regex.Matches(this.DataBlock, AdditionalRegex[i].RgExp.ToString());
                        int count = 0;
                        if (AdditionalRegex[i].NumOfResults == 0)
                        {
                            count = RegexMatches.Count;
                        }
                        else
                            count = AdditionalRegex[i].NumOfResults;
                        if (RegexMatches.Count < count)
                            count = RegexMatches.Count;
                        string Data = "";
                        for (int c = 0; c < count; c++)
                        {
                            Data += RegexMatches[c].Value.ToString();
                        }
                        string targetColumn = AdditionalRegex[i].RgNam;
                        AdditionalData.Add(new PersonData(targetColumn, Data));
                    }
                    catch (Exception e)
                    {
                        WrongRegex.Add(i);
                        throw e;
                    }
 
                }
                return AdditionalData;
            }
            
        } 

Class Regex Expression Type:

When user choose to add new Regex Expression in the Regex Expression Table ,the target column which the extracted data will be stored must be defined and also the number of results.

 class AdditionalRegexExpression
    {
        public AdditionalRegexExpression(string Name, string Regex,int NumberOfResults)
        {
            this.RgNam = Name;
            this.RgExp = Regex;
            this.NumOfResults = NumberOfResults;
        }
 
        public string RgExp { get; set; } // The Regex Expression 
 
        public string RgColumnTarget { get; set; }  //The targeted column
 
        public int NumOfResults { get; set; }  //the number of Results as defined by user
    } 

Opening text file in Rich textbox:

This method is used to load mixed data from a text file in Richtextbox control, this mixed data will be used to extract data related to different people. the ability to differentiate data related to specific person from another person controlling the accuracy of extracting results is depending on :

1) There must be more than 2 “\n” (spaces) between different related to different people; this will help program to split this data accurately and pushing them in data extracting process.

2) If there are another Regex entered by user it must be accurate and exist between person’s blocks.

3) There are data homogeneity –there must be like a general rule for different kind of data –like “Tel[:]+(\d+[-])+” for telephone number ,This will avoid you from writing different Regex to get only one kind of data like Emails.

 using (OpenFileDialog dlgOpen = new OpenFileDialog())
            {
                try
                {
                    // Available file extensions
                    dlgOpen.Filter = "All files(*.*)|*.*";
                    // Initial directory
                    dlgOpen.InitialDirectory = "D:";
                    // OpenFileDialog title
                    dlgOpen.Title = "Open";
                    // Show OpenFileDialog box
                    if (dlgOpen.ShowDialog() == DialogResult.OK)
                    {
                        // Create new StreamReader
                        StreamReader sr = new StreamReader(dlgOpen.FileName, Encoding.Default);
                        // Get all text from the file
                        string str = sr.ReadToEnd();
                        // Close the StreamReader
                        sr.Close();
                        // Show the text in the rich textbox rtbMain
                        
                        MixedEmailSRichtextBox.Text = str;
                    }
                }
                catch (Exception errorMsg)
                {
                    MessageBox.Show(errorMsg.Message);
                }
          } 

Different Regex for Data:

Class RegexExpressionCollection contains some of the available Regex Expressions which is important and will help you to identify important details about this person. Let’s agree that all Regex expressions here can be overwritten by just adding new Regex expression and adjust the column target to be as you wish.

1. Fax number Regex:

As the only difference between Fax and telephone number is that fax number is preceded with word fax ,so the Regex search for word fax as a general key to differentiate between fax and telephone number.

  public static string Fax{ get { return @"Fax:\s[+]*([\d+[-]*)+"; }} 

2. 2.Phone number Regex:

 public static string Phone { get { return @"(Tel|phone|telephone|mobile):\s[+]*([\d+[-]*)+"; }  }

3. Web page: 

There are different Regex expressions for web page this is the most simple and accurate I have found. 

public static string WebPage{get { return @"(http|https|ftp)\://[a-zA-Z0-9\-\.]+\.[a-zA-Z]{2,3}(:[a-zA-Z0-9]*)?/?([a-zA-Z 0-9\-\._\?\,\'/\\\+&%\$#\=~])*"; } } 

4. Birth Date:

public static string BirthDate{get { return @"Fax:\s[+]*([\d+[-]*)+"; } } 

5. Email:

As Email is the most important thing you have here , if this method didn’t reveal any value the program will erase it from results automatically, unless you check the check box titled” don’t erase the No email results”.

 public static string Email{get { return @"\w+([-+.]\w+)*@\w+([-.]\w+)*\.\w+([-.]\w+)*"; } }
  

6. Gender:

Gender Guessing for every client is very important ,this would determine if this person is really interest in your product or not. Suppose you were given a long mailing list and had to write a program to determine whether each name on the list represented a male or female, perhaps so you could preface each name with "Mr." or "Ms." in the salutation of a form letter. How would you do it? The first solution that comes to mind might be to create a table of every personal name and the gender normally associated with it. Then, for each name in the mailing list, look up the corresponding gender in the table, most likely using a hash function for speed.

Such an approach has two key problems. First, variations in spelling, no matter how slight, will defeat the algorithm. A human can tell that Caren, Caryn, Karyn, and Karin are all variations of the same name, but a table-lookup algorithm can't. If one of those four spellings aren't in the table, the program will be unable to match that name with a gender. Second, typing thousands of name-gender pairs into a table is time-consuming and error-prone. Moreover, you might never have a complete list because of variant spellings. What if I told you a gender detection program can be written in about 50 lines of code and data.

        public static bool IsMale(string Name)
        {
               string[] ArrayOfRegex = {
 
               "^[^S].*r[rv]e?y?$"                       //{ sex = ,"m," }  # Barry, Larry, Perry,...
              ,"^[^G].*v[ei]$"                           //{ sex = ,"m," }  # Clive, Dave, Steve,...
              ,"^[^AJKLMNP][^o][^eit]*([glrsw]ey|lie)$"  //{ sex = ,"m," }  # Dewey, Stanley, Wesley,...
              ,"^[CGJWZ][^o][^dnt]*y$"                   //{ sex = ,"m," }  # Gregory, Jeremy, Zachary,...
              ,"^.*[Rlr][abo]y$"                         //{ sex = ,"m," }  # Leroy, Murray, Roy,...
              ,"^.*[GRguw][ae]y?ne$"                     //{ sex = ,"m," }  # Duane, Eugene, Rene,...
              ,"^[CLMQTV].*[^dl][in]c.*[ey]$"            //{ sex = ,"m," }  # Lance, Quincy, Vince,...
              ,"^.*[ay][dl]e$"                           //{ sex = ,"m," }  # Clyde, Kyle, Pascale,...
              ,"^[^o]*ke$"                               //{ sex = ,"m," }  # Blake, Luke, Mike,...
              ,"^[^EL].*o(rg?|sh?)?(e|ua)$"              //{ sex = ,"m," }  # George, Joshua, Theodore,..
              ,"^[^JPSWZ].*[denor]n.*y$"                 //{ sex = ,"m," }  # Anthony, Henry, Rodney,...
              ,"^Br[aou][cd].*[ey]$"                     //{ sex = ,"m," }  # Bradley, Brady, Bruce,...
              ,"^[ILW][aeg][^ir]*e$"                     //{ sex = ,"m," }  # Ignace, Lee, Wallace,...
              ,"^[ABEIUY][euz]?[blr][aeiy]$"             //{ sex = ,"m," }  # Ari, Bela, Ira,...
              ,"^[ART][^r]*[dhn]e?y$"                    //{ sex = ,"m," }  # Randy, Timothy, Tony,...
              ,"^.*oi?[mn]e$"                            //{ sex = ,"m," }  # Antoine, Jerome, Tyrone,...
              ,"^D.*[mnw].*[iy]$"                        //{ sex = ,"m," }  # Danny, Demetri, Dondi,...
              ,"^[^BG](e[rst]|ha)[^il]*e$"               //{ sex = ,"m," }  # Pete, Serge, Shane,...
            
                                       };
            
            for (int i = 0; i < ArrayOfRegex.Length; i++)
               if(Regex.IsMatch(Name,ArrayOfRegex[i]))
                return true;
        
            return false;
        }

public static string[] Male{get{
return new string[]{
"^[^S].*r[rv]e?y?$"                       //{ sex = ,"m," }  # Barry, Larry, Perry,...
,"^[^G].*v[ei]$"                           //{ sex = ,"m," }  # Clive, Dave, Steve,...
,"^[^AJKLMNP][^o][^eit]*([glrsw]ey|lie)$"  //{ sex = ,"m," }  # Dewey, Stanley, Wesley,...
,"^[CGJWZ][^o][^dnt]*y$"                   //{ sex = ,"m," }  # Gregory, Jeremy, Zachary,...
,"^.*[Rlr][abo]y$"                         //{ sex = ,"m," }  # Leroy, Murray, Roy,...
,"^.*[GRguw][ae]y?ne$"                     //{ sex = ,"m," }  # Duane, Eugene, Rene,...
,"^[CLMQTV].*[^dl][in]c.*[ey]$"            //{ sex = ,"m," }  # Lance, Quincy, Vince,...
,"^.*[ay][dl]e$"                           //{ sex = ,"m," }  # Clyde, Kyle, Pascale,...
,"^[^o]*ke$"                               //{ sex = ,"m," }  # Blake, Luke, Mike,...
,"^[^EL].*o(rg?|sh?)?(e|ua)$"              //{ sex = ,"m," }  # George, Joshua, Theodore,..
,"^[^JPSWZ].*[denor]n.*y$"                 //{ sex = ,"m," }  # Anthony, Henry, Rodney,...
,"^Br[aou][cd].*[ey]$"                     //{ sex = ,"m," }  # Bradley, Brady, Bruce,...
,"^[ILW][aeg][^ir]*e$"                     //{ sex = ,"m," }  # Ignace, Lee, Wallace,...
,"^[ABEIUY][euz]?[blr][aeiy]$"             //{ sex = ,"m," }  # Ari, Bela, Ira,...
,"^[ART][^r]*[dhn]e?y$"                    //{ sex = ,"m," }  # Randy, Timothy, Tony,...
,"^.*oi?[mn]e$"                            //{ sex = ,"m," }  # Antoine, Jerome, Tyrone,...
,"^D.*[mnw].*[iy]$"                        //{ sex = ,"m," }  # Danny, Demetri, Dondi,...
,"^[^BG](e[rst]|ha)[^il]*e$"  };           //{ sex = ,"m," }  # Pete, Serge, Shane,...; }}}
 

        private void ExtractDataToCSV(DataGridView dgv)
        {
 
            // Don't save if no data is returned
            if (dgv.Rows.Count == 0)
            {
                return;
            }
            StringBuilder sb = new StringBuilder();
            // Column headers
            string columnsHeader = "";
            for (int i = 0; i < dgv.Columns.Count; i++)
            {
                columnsHeader += dgv.Columns[i].Name + ",";
            }
            sb.Append(columnsHeader + Environment.NewLine);
            // Go through each cell in the datagridview
            foreach (DataGridViewRow dgvRow in dgv.Rows)
            {
                // Make sure it's not an empty row.
                if (!dgvRow.IsNewRow)
                {
                    for (int c = 0; c < dgvRow.Cells.Count; c++)
                    {
                        // Append the cells data followed by a comma to delimit.
 
                        sb.Append(dgvRow.Cells[c].Value + ",");
                    }
                    // Add a new line in the text file.
                    sb.Append(Environment.NewLine);
                }
            }
            // Load up the save file dialog with the default option as saving as a .csv file.
            SaveFileDialog sfd = new SaveFileDialog();
            sfd.Filter = "CSV files (*.csv)|*.csv";
            if (sfd.ShowDialog() == System.Windows.Forms.DialogResult.OK)
            {
                // If they've selected a save location...
                using (System.IO.StreamWriter sw = new System.IO.StreamWriter(sfd.FileName, false))
                {
                    // Write the stringbuilder text to the the file.
                    sw.WriteLine(sb.ToString());
                }
            }
            // Confirm to the user it has been completed.
            MessageBox.Show("CSV file saved.");
        }