Writing Big Quantities of Files to HDD on Windows

• Download source - 31.8 KB

Introduction

If you write about thousand files into a folder and then try to open this folder with Explorer, you might notice that Explorer starts slowing down a bit. If you put 100,000 files, the creation of new ones in your program shall slow down all the performance. The solution to this problem is not to put too many files in one folder. The first thing that comes into mind is a hierarchical structure of folders, each folder containing no more than N files. This is exactly what the class presented here does.

The Hierarchical Structure

The proposed hierarchy is shown on the picture (composed on April 18, 2012):

First comes the root folder - the one you shall pass into the constructor. In this case, it's C:\FolderProviderTest. Then the folder with the current date in the name comes. The date format is "dd_MM_yyyy", and the lowest level of folders has the current hour, minute, and sequence number in the name in format "HH_mms" where s is the sequence number that is continuously incremented.

Using the Code

The Idea

Each time you need to write a file, you need the destination folder name. Somebody should take care of several things:

• The given folder must exist.
• The folder should contain only the allowed quantity of files. If this quantity exceeds, another one should be created and its name given.
• All the logic of the hierarchical structure should be hidden.

This is what the TreeFolderProvider class does.

The TreeFolderProvider Class

For each file you need to write on the disk, just call the TreeFolderProvider.NextFolder method - and you'll get the correct folder to write in. You can than use the Path.Combine method to get the full file name. You can also specify how many files should be there in one folder by setting the TreeFolderProvider.MaxAllowedInOneFolder property. Keep in mind that the method NextFolder will return the same folder name exactly MaxAllowedInOneFolder times. Then the folder name will change. Thus, for each file, you should call the NextFolder method exactly once.

public class TreeFolderProvider:IFolderProvider
{
    private string rootFolder;
    private DateTime creationTime = DateTime.Now;

    private int maxAllowedInOneFolder = 5;
    private string currentFolder;
    private int filesInCurrentFolder = 0;
    long folderChangeCount = 0;

    private DateTime todayFolder = DateTime.Now.Date;
    private System.Timers.Timer dateChecker = new System.Timers.Timer(60000);

    public static bool requireFolderWork = true;//set into false for testing purposes only
    
    void dateChecker_Elapsed(object sender, System.Timers.ElapsedEventArgs e)
    {
        CheckDate(DateTime.Now);
    }

    private static void requireFolder(string mustBe)
    {
        if (requireFolderWork == true)
            if (Directory.Exists(mustBe) == false)
                Directory.CreateDirectory(mustBe);
    }

    private void makeNextFolder()
    {
        long folderNumber = Interlocked.Increment(ref folderChangeCount);
        string date = todayFolder.ToString("dd_MM_yyyy");
        string time = DateTime.Now.ToString("HH_mm") + folderNumber.ToString();
        string datePath = Path.Combine(rootFolder, date);
        requireFolder(datePath);
        string nextPath = Path.Combine(datePath, time);
        requireFolder(nextPath);
        currentFolder = nextPath;

        filesInCurrentFolder = 1;
    }

    public TreeFolderProvider(string folderName)
    {
        rootFolder = folderName;
        requireFolder(folderName);
        makeNextFolder();
        filesInCurrentFolder = 0;
        dateChecker.Elapsed += new System.Timers.ElapsedEventHandler(dateChecker_Elapsed);
        dateChecker.Start();
    }
    
    public int MaxAllowedInOneFolder
    {
        get { return maxAllowedInOneFolder; }
        set { maxAllowedInOneFolder = value; }
    }

    public string RootFolder { get { return rootFolder; } }

    public string CurrentFolder { get { return currentFolder; } }

    public int FilesInCurrentFolder { get { return filesInCurrentFolder; } }

    public long FolderChangeCount { get { return folderChangeCount; } }
    
    public string NextFolder()
    {
        Interlocked.Increment(ref filesInCurrentFolder);
        if (filesInCurrentFolder > maxAllowedInOneFolder)
        {
            makeNextFolder();
        }
        return currentFolder;
    }

    public void CheckDate(DateTime toCheck)
    {
        if (todayFolder.Equals(toCheck.Date) == false)
        {
            todayFolder = toCheck.Date;
            makeNextFolder();
        }
    }

    public void Dispose()
    {
        dateChecker.Dispose();
    }
}

FolderChangeCount shows how many times the folder was actually changed, i.e., the private makeNextFolder method was called.

A class implements IDisposable because there's a timer that checks whether the next date has come. In this case, the folder is "recalculated" by the makeNextFolder method.

Testing Application

The console application in the example program creates a new instance of the TreeFolderProvider class, sets MaxAllowedInOneFolder property to 3, and then each five seconds, calls the NextFolder method, printing the result.

static void Main(string[] args)
{
    TreeFolderProvider provider = new TreeFolderProvider(@"C:\FolderProviderTest");
    provider.MaxAllowedInOneFolder = 3;
    for (int i = 0; i < 20; ++i)
    {
        Console.WriteLine(provider.NextFolder());
        Thread.Sleep(5000);
    }
    Console.WriteLine("Press any key to continue");
    Console.ReadKey();
}

The output is given below. The resulting folder structure is shown in the picture in the beginning of the article.

C:\FolderProviderTest\18_04_2012\19_591
C:\FolderProviderTest\18_04_2012\19_591
C:\FolderProviderTest\18_04_2012\19_591
C:\FolderProviderTest\18_04_2012\19_592
C:\FolderProviderTest\18_04_2012\19_592
C:\FolderProviderTest\18_04_2012\19_592
C:\FolderProviderTest\18_04_2012\19_593
C:\FolderProviderTest\18_04_2012\19_593
C:\FolderProviderTest\18_04_2012\19_593
C:\FolderProviderTest\18_04_2012\19_594
C:\FolderProviderTest\18_04_2012\19_594
C:\FolderProviderTest\18_04_2012\19_594
C:\FolderProviderTest\18_04_2012\20_005
C:\FolderProviderTest\18_04_2012\20_005
C:\FolderProviderTest\18_04_2012\20_005
C:\FolderProviderTest\18_04_2012\20_006
C:\FolderProviderTest\18_04_2012\20_006
C:\FolderProviderTest\18_04_2012\20_006
C:\FolderProviderTest\18_04_2012\20_007
C:\FolderProviderTest\18_04_2012\20_007
Press any key to continue

Points of Interest

The TreeFolderProvider class implements IFolderProvider interface with the NextFolder method only, giving you the opportunity to make some other strategy of folder providing (Strategy pattern). In the example program, TrivialFolderProvider is presented, which always returns the same folder name. It might be used for cases when you need to test your program and there won't be too many files.

History

• 18^th April, 2012 - First published