Automatically process incoming files

• Download source files - 4.55 KB

Introduction

Hello all. This is a little article on how to process incoming data automatically. In the short past, I've seen lots of people trying to use the FileSystemWatcher to trigger the data processing function in their software. However, many of them have to deal with the exception: "The process cannot access the file because it is being used by another process". We can fix this problem very easily. We only have to create a thread which reads the directory constantly, using the DirectoryInfo class. Next to that, we create a ProcessFile function, which processes the file and moves it into the 'processed' directory.

Using the code

Here we go:

private bool ProgramRunning = true;
private int FilesHandled = 1;

static void Main(string[] args)
{
    Program prog = new Program();
    prog.ReadDirectoryThread();
}

I use the field FilesHandled to count the number of files I've processed, just so that I can verify whether all files have been processed or not.

private void ReadDirectoryThread()
{
    while (this.ProgramRunning)
    {
        DirectoryInfo DirInfo = new DirectoryInfo(
            Path.GetDirectoryName(
            Process.GetCurrentProcess().MainModule.FileName)
            + "\\incoming\\");
        foreach (FileInfo FInfo in DirInfo.GetFiles("*.xml"))
        {
            Thread t = new Thread(new ParameterizedThreadStart(ProcessFile));
            t.Start(FInfo.FullName);
            while (t.ThreadState != System.Threading.ThreadState.Stopped)
            {
                //Thread.Sleep(5);
            }
        }
    }
}

We continuously check if new files are created. With the GetFiles() function, you can set a filter on a specific document type, for example, XML files. As soon as the directory contains documents, the ProcessFile function is called with the ParameterizedThreadStart() function. You can add a Thread.Sleep() in the t.ThreadState while loop to decrease the CPU usage; however, it is not recommended when you have to process a high amount of data files constantly.

void ProcessFile(object FileToProcess)
{
    string sFile = FileToProcess.ToString();
    bool isProcessed = false;
    // File processing
    try
    {
        // Process your file here.
    }
    catch
    {
        return;
    }
    // After the file is processed, we move it
    while (!isProcessed)
    {
        try
        {
            string ProcessedFilesDir = string.Format("{0}\\processed\\{1}_{2}.xml",
                Path.GetDirectoryName(Process.GetCurrentProcess().MainModule.FileName),
                DateTime.Now.ToString("dd_MM_yyyy_HH_mm_ss"), Guid.NewGuid().ToString());
            File.Move(sFile, ProcessedFilesDir);
            Console.WriteLine(FilesHandled + " - File " + 
                              sFile.ToString() + " processed\n");
            FilesHandled++;
        }
        catch
        {
            isProcessed = false;
        }
        finally
        {
            isProcessed = true;
        }
    }
}

In the first part in the code above, we can call the function to handle our data file, e.g., an import function or whatsoever. After the data has been handled successfully, we try to move the file to the 'processed' directory. In case the file has exclusive rights, the thread keeps trying to move the file until it's finally moved.

I've had a few cases where more than a hundred data files came in every second. Therefore, I added the DateTime.Now.ToString("dd_MM_yyyy_HH_mm_ss") and the Guid.NewGuid().ToString() functions, to create a proper filename for the processed file, without any risks on exceptions.

Because this is a console application, I used Path.GetDirectoryName(Process.GetCurrentProcess().MainModule.FileName). However, in Windows Forms, you can use Application.StartupPath instead.

History

• 27 Oct 2007 - First version published.