Validate a text file based on LINQ

• Download source code - 32.7 KB

Introduction

This article will introduce how to validate a text file based on LINQ.

Background

A flat file is being used as the data file to pass around. In order to make the file to be accepted from other systems, the file has to follow some data format that the other systems expect. Not only the data format, some business rules may also need to be pre-evaluated before the file is sent out. In this article, I'll build a validation engine based on LINQ to a text file to challenge if the file meets all the requirements which I defined for a system to process it.

Rules need to be validated

The valid file should have a HEADER record, a FOOTER record, and some CONTENT records. The first 3 characters will be used to describe the type of record it is. In this case, I am going to use 111 for HEADER record, 999 for FOOTER record, and 222 for CONTENT records. We could use 4-6 characters to describe the sub-record information. However, I am only going to demo the HEADER, FOOTER, and CONTENT records here. File example:

111this is header record
222allen.y.w#gmail.com 123-45-6789
222allen.y.w4gmail.com 123456789
222allen.y.w@gmail.com 123-45-6789
999totalrecord5

In the above file, here are some basic rules I want to validate:

1. File needs to have a HEADER record, which has a fixed length (16) --> raise error message when not met.
2. File needs to have one or many CONTENT records, which has a fixed length (40) --> raise error message when not met.
3. Position starting from 4 to 23 from CONTENT record is used for email information, and it needs to follow the email format (xxx@xxx.xxx) --> raise warning when not met.
4. Position starting from 24 to 40 from CONTENT record is used for SSN information, and it need to be like: xxx-xx-xxxx (x is number only) -> raise warning when not met.
5. File needs to have a FOOTER record. --> raise error message when not met.

Validation engine classes

Since the validation engine needs to validate multiple text file formats, I am going create an abstract class (BaseValidator) to structure the basic functionality. A child class (File1Validator) will implement those methods based on it own rules. An Event Handler in BaseValidator will handle the information passed through to the client when an error occurs.

Validation engine implementation

BaseValidator

BaseValidator is an abstract class. It has a Validate() method that drives all the validating process. In this case, it will always execute the ValidateHeader(), ValidateContent(), and ValidateFooter() processes when we invoke Validate(). The Validate() method can be overridden from the child class to implement more processes.

public virtual void Validate()
{
    ValidateHeader();
    ValidateContent();
    ValidateFooter();
}

protected abstract void ValidateHeader();
protected abstract void ValidateContent();
protected abstract void ValidateFooter();

BaseValidator also creates an event when an error occurs. The event call could provide the communication between the validation engine and the client.

public event EventHandler<validatingeventargs> EventOccured;

protected virtual void OnEventOccured(ValidatingEventArgs e)
{
    EventHandler<validatingeventargs> handler = EventOccured;
    if (handler != null)
    {
        handler(this, e);
    }
}

ValidatingEventArgs is an object that contains all the error information when the validator processes the rules. Since it's an EventArgs object, we can follow the .NET EventArgs design to directly inherit from the .NET EventArgs object.

namespace askbargains.com.txtValidator.Validation
{
    public class ValidatingEventArgs : EventArgs
    {
        public ValidatingEventArgs(string msg, 
               MessageType mType, RecordType rType)
        {
            this.Message = msg;
            this.MessageType = mType;
            this.RecordType = rType;
        }
        public string Message { get; set; }
        public MessageType MessageType { get; set; }
        public RecordType RecordType { get; set; }

    }
}

File1Validator

File1Validator is a child class from BaseValidator. It will only validate the rules I defined above. More children classes could be created for handling different types of file formats.

From our File1Validator, we want to call our BaseValidator Validate() method to drive the basic validation flow; we could also specify the additional validation process in File1Validator's Validate().

namespace askbargains.com.txtValidator.Validation
{
    public class File1Validator: BaseValidator
    {
        //constructor
        public File1Validator(string fileLocation)
            : base(fileLocation)
        {
        }
        public override void Validate()
        {
            //triger the BaseValidator's Validate
            base.Validate();

            //adding more rules for record1 if needed
            //ex: this.ValidateAdditionalRules();
        }
    }
}

File1Validator also handles the implementation for those BaseValidator abstract methods. In this case, that would be ValidateHeader(), ValidateContent(), and ValidateFooter().

In our File1Validator, we also use LINQ to query each type of record from our text file, and use Regular Expression to handle the format. Business rules could also be done in the same way.

protected override void ValidateHeader()
{
    //Select hearder rec from the file
    var heaRec = from str in RecStrs
                 where str.Substring(0, 3) == Convert.ToString((int)RecordType.HEADER)
                 select str;

    //add all the rules for header
    if (heaRec.Count() == 0)
    {
        base.OnEventOccured(new ValidatingEventArgs("File  DOES NOT " + 
             "contain Header record", MessageType.Error, RecordType.HEADER));
    }
    if (heaRec.Count() > 1)
    {
        base.OnEventOccured(new ValidatingEventArgs("Only one Header " + 
             "record allowed", MessageType.Error, RecordType.HEADER));
    }
    if (heaRec.Count() == 1)
    {
        if (heaRec.First().Length != 16)
        {
            base.OnEventOccured(new ValidatingEventArgs("Header record has " + 
                 "to be 16 charcter", MessageType.Error, RecordType.HEADER));
        }
        //add more rules for header 
    }

}

protected override void ValidateContent()
{
    //select content recs
    var conRec = from str in RecStrs
                 where str.Substring(0, 3) == Convert.ToString((int)RecordType.CONTENT)
                 select str;


    //add all the rules for content
    if (conRec.Count() == 0)
    {
        OnEventOccured(new ValidatingEventArgs("File  DOES NOT contain content record", 
                           MessageType.Error, RecordType.CONTENT));
    }

    int counter = 0;

    if (conRec.Count() > 0)
    {
        foreach (string address in conRec)
        {
            counter++;
            if (address.Length != 40)
            {
                OnEventOccured(new ValidatingEventArgs("Content record has to be 40", 
                               MessageType.Error, RecordType.CONTENT));
            }
            else
            {
                if (!email.IsMatch(address.Substring(3, 20)))
                {
                    OnEventOccured(new ValidatingEventArgs("Email is invliad " + 
                                   "for content " + counter.ToString(), 
                                   MessageType.Warning, RecordType.CONTENT));
                }

                if (!ssn.IsMatch(address.Substring(23, 11)))
                {
                    OnEventOccured(new ValidatingEventArgs("SSN is invliad " + 
                                   "fro content" + counter.ToString(), 
                                   MessageType.Warning, RecordType.CONTENT));
                }
            }

        }
    }
}

protected override void ValidateFooter()
{
    //check if file has Footer rec
    var fooRec = RecStrs
              .Where(rec => (rec.Substring(0, 3) == 
                                Convert.ToString((int)RecordType.FOOTER)))
              .Select(rec => rec.Substring(0, 3));

    if (fooRec.Count() == 0)
    {
        OnEventOccured(new ValidatingEventArgs("File  DOES NOT contain Footer Record", 
                                               MessageType.Warning, RecordType.FOOTER));
    }
}

Client to handle Error Messages

In this example, I am using a window application to represent the error information from File1Validator. Since an ErrorOccured event will raise whenever an error is caught, the Windows UI will delegate the EventOccured and print the error details.

See the code when we click StartValidate from the UI:

private void btnStart_Click(object sender, EventArgs e)
{
    this.listView1.Items.Clear();
    try
    {

        File1Validator recValidator = new File1Validator(textBox1.Text);
        recValidator.EventOccured += 
          new EventHandler<validatingeventargs>(recValidator_EventOccured);
        recValidator.Validate();

        MessageBox.Show("Validation completed");
    }
    catch (NullReferenceException ex)
    {
        MessageBox.Show("select an filepath", ex.Message);
    }
}

void recValidator_EventOccured(object sender, ValidatingEventArgs e)
{
    //handle error msg
    if (e.MessageType == MessageType.Error)
    {
        listView1.Items.Add(new ListViewItem(new string[] { e.MessageType.ToString(), 
           sender.ToString().Substring(sender.ToString().LastIndexOf(".") + 1), 
           e.RecordType.ToString(), e.Message }, "error.jpeg"));
    }
    //handle warning msg
    else if (e.MessageType == MessageType.Warning)
    {
        listView1.Items.Add(new ListViewItem(new string[] { e.MessageType.ToString(), 
           sender.ToString().Substring(sender.ToString().LastIndexOf(".") + 1), 
           e.RecordType.ToString(), e.Message }, "warning.jpeg"));

    }
}

Validator_EventOccured will take care of the ValidatingEventArgs object and display the result in the UI window.

Conclusion

In this application, we built a validation engine to evaluate a text file to process each line and generate errors when it's not meeting the requirements.