Dokan.Mem, A filesystem Prototype

• Download demo - 519.76 KB
• Download source - 532.52 KB

Introduction

Hiroki Asakawa created a device driver that enables an application in user-mode to simulate a file-system, and distributed it under a MIT-style license. This post shows how that's used in C# to create the functionality of a RAM-Disk, with some tips on rolling your own file-system application.

Before you can run the code, you have to run the installer to install a proxy driver (dokan.sys). This driver then acts as an intermediate between the kernel and our custom .NET solution. The installer for the proxy driver is also included with the source code here, but I'd recommend downloading the package from the original website as they contain two cool examples; The first implements a mirror of your C-drive, and the second makes the Registry available as a readonly drive. This article briefly explains how the Dokan-libraries are used to build something that resembles a RAM-disk in functionality

Having the option to expose data as a file-system has several advantages. It makes your data instantaneous available for other applications, without the need for a complicated UI.

Download the basic installer and .NET bindings to Dokan here[^].

Using the Code

Log in as administrator, run the installer (it's just 513 KB), open the project and hit F5; a trayicon should appear, and on doubleclick it should open the Windows-Explorer, pointing to a simulated harddisk. It'll mount on the first available free drive-letter. Paste a zip file on there and extract it. :)

I choose to implement a basic RAM-Disk to test the library (version 0.5.3), with the results documented here.

There's three projects in the solution:

• DokanNet - These are the .NET bindings to the Dokan libraries
• Buffers - Used to replace the MemoryStream
• Dokan.Mem - Example-implementation of the Dokan-interface, simulating a RAM-Disk

DokanNet

This project contains the bindings to the Dokan-libraries, provided by the DokanOperations-interface. It's a pretty straightforward definition of the actions that an application (like Word) can do on the file-system.

 1  public interface DokanOperations
 2  {
 3      int CreateFile(string filename, ..., DokanFileInfo info);
 4      int OpenDirectory(string filename, DokanFileInfo info);
 5      int CreateDirectory(string filename, DokanFileInfo info);
 6      int Cleanup(string filename, DokanFileInfo info);
 7      int CloseFile(string filename, DokanFileInfo info);
 8      int ReadFile(string filename, ..., DokanFileInfo info);
 9      int WriteFile(string filename, ..., DokanFileInfo info);
10      int FlushFileBuffers(string filename, DokanFileInfo info);
11      int GetFileInformation(string filename, ..., DokanFileInfo info);
12      int FindFiles(string filename, ArrayList files, DokanFileInfo info);
13      int SetFileAttributes(string filename, ..., DokanFileInfo info);
14      int SetFileTime(string filename, ..., DokanFileInfo info);
15      int DeleteFile(string filename, DokanFileInfo info);
16      int DeleteDirectory(string filename, DokanFileInfo info);
17      int MoveFile(string filename, ..., DokanFileInfo info);
18      int SetEndOfFile(string filename, long length, DokanFileInfo info);
19      int SetAllocationSize(string filename, long length, DokanFileInfo info);
20      int LockFile( string filename, long offset, long length, DokanFileInfo info);
21      int UnlockFile(string filename, long offset, long length, DokanFileInfo info);
22      int GetDiskFreeSpace(ref ulong freeBytesAvailable, ..., DokanFileInfo info);
23      int Unmount(DokanFileInfo info);
24  }

Once you have a class based on this interface (e.g. 'MyDokanOperations'), you can launch your new drive by calling the Dokan-main method. The drive will be available as long as this (blocking) task is running.

A skeleton-application is given below, implementing the DokanOperations in the MyDokanOperations class. Take note that the DokanOperations is an interface! It might not be named IDokanOperations, but that's what it should read.

 1  class MyDokanOperations : DokanOperations
 2  {
 3     // implementation of DokanOperations interface goes here..
 4  }
 5
 6  DokanOptions options = new DokanOptions
 7  {
 8     DriveLetter = 'Z',
 9     DebugMode = true,
10     UseStdErr = true,
11     NetworkDrive = false,
12     Removable = true,     // provides an "eject"-menu to unmount
13     UseKeepAlive = true,  // auto-unmount
14     ThreadCount = 0,      // 0 for default, 1 for debugging
15     VolumeLabel = "MyDokanDrive"
16  };
17
18  static void Main(string[] args)
19  {
20     DokanNet.DokanMain(
21         options,
22         new MyDokanOperations());
23  }

The examples all show a console, which is cool when you're developing your new file-system. It's very useful when debugging, and you can actually follow the interaction between the kernel and the simulated file-system.

Dokan.Mem

The RAM-Disk also shows the console-window, but you easily disable it by changing the output-type of the project to "Windows Application". I've provided a trayicon that lets one interact with the application in release-mode.

Back to that DokanOperations-interface; Most of these calls have actual API counterparts and you can find a description on MSDN. The API-description is very useful as it documents the general flow, and lists the error-codes that it might return.

Let's take a look at the actual implementation of the DeleteFile [^] method;

 1  public int DeleteFile(string filename, DokanFileInfo info)
 2  {
 3     // get parent folder
 4     MemoryFolder parentFolder = _root.GetFolderByPath(
 5         filename.GetPathPart());
 6
 7     // exists?
 8     if (!parentFolder.Exists())
 9         return -DokanNet.ERROR_PATH_NOT_FOUND;
10
11     // fetch file;
12     MemoryFile file = parentFolder.FetchFile(
13         filename.GetFilenamePart());
14
15     // exists?
16     if (!file.Exists())
17         return -DokanNet.ERROR_FILE_NOT_FOUND;
18
19     // delete it;
20     parentFolder.Children.Remove(file);
21     file.Content.Dispose();
22
23     return DokanNet.DOKAN_SUCCESS;
24  }

MSDN states for the DeleteFile function:

"If an application attempts to delete a file that does not exist, the DeleteFile function fails with ERROR_FILE_NOT_FOUND. If the file is a read-only file, the function fails with ERROR_ACCESS_DENIED."

The RAM-disk does check whether the file exists, but I didn't implement the check on the file-attributes yet. As far as setting the file-attributes go, that's not implemented at all;

1  public int SetFileAttributes(
2     string filename,
3     FileAttributes attr,
4     DokanFileInfo info)
5  {
6     return -DokanNet.DOKAN_ERROR;
7  }

If you take a look at the interface again, you'll notice that there's no OpenFile method. If the system wants to open a file, it will call the CreateFile method. It takes the same flags that the CreateFile [^] API uses. Depending on the FileMode, we create or open an existing or a non-existing file:

 1  public int CreateFile(
 2      string filename,
 3      FileAccess access,
 4      FileShare share,
 5      FileMode mode,
 6      FileOptions options,
 7      DokanFileInfo info)
 8  {
 9      [...]
10
11       // attempt to use the file
12      switch (mode)
13      {
14          // Opens the file if it exists and seeks to the end of the file,
15          // or creates a new file
16          case FileMode.Append:
17              if (!thisFile.Exists())
18                  MemoryFile.New(parentFolder, newName);
19              return DokanNet.DOKAN_SUCCESS;
20
21          // Specifies that the operating system should create a new file.
22          // If the file already exists, it will be overwritten.
23          case FileMode.Create:
24              //if (!thisFile.Exists())
25                  MemoryFile.New(parentFolder, newName);
26              //else
27              // thisFile.Content = new Thought.Research.AweBuffer(1024);
                //MemoryStream();
28              return DokanNet.DOKAN_SUCCESS;
29
30          // Specifies that the operating system should create a new file.
31          // If the file already exists, an IOException is thrown.
32          case FileMode.CreateNew:
33              if (thisFile.Exists())
34                  return -DokanNet.ERROR_ALREADY_EXISTS;
35              MemoryFile.New(parentFolder, newName);
36              return DokanNet.DOKAN_SUCCESS;
37
38          // Specifies that the operating system should open an existing file.
39          // A System.IO.FileNotFoundException is thrown if the file does not exist.
40          case FileMode.Open:
41              if (!thisFile.Exists())
42                  return -DokanNet.ERROR_FILE_NOT_FOUND;
43              else
44                  return DokanNet.DOKAN_SUCCESS;
45
46          // Specifies that the operating system should open a file if it exists;
47          // otherwise, a new file should be created.
48          case FileMode.OpenOrCreate:
49              if (!thisFile.Exists())
50                  MemoryFile.New(parentFolder, newName);
51              return DokanNet.DOKAN_SUCCESS;
52
53          // Specifies that the operating system should open an existing file.
54          // Once opened, the file should be truncated so that its size is zero bytes
55          case FileMode.Truncate:
56                  if (!thisFile.Exists())
57                      thisFile = MemoryFile.New(parentFolder, newName);
58                  thisFile.Size = 0;
59                  return DokanNet.DOKAN_SUCCESS;
60      }
61
62      return DokanNet.DOKAN_ERROR;
63  }

The MemoryFolder and MemoryFile classes are used to map the "files" in memory in a hierarchical structure. There's a rootnode that represents the root of the drive, and may contain objects that represent either a file or a folder:

You'll notice that the MemoryFile class is abstract.

Buffers

The RAM-Disk prototype was originally based on a MemoryStream. That would make it a "virtual memory disk", we're missing some functionality before we can call it a RAM-Disk.

I've replaced the MemoryStream with a buffer that's based on an idea [^] from David Pinch. He wrote a class that can be used to allocate memory, freeing it in the Dispose section. It's originally designed to provide the ability to protect an allocated region, hence the name ProtectedBuffer. This block of memory can be accessed as a stream, with the drawback that it can't be resized.

The AweBuffer class is based on that ProtectedBuffer-class, adding a call to the VirtualLock [^] API. This way the information is pinned into physical-memory, for as long as the thread is running*. Without this, the RAM-disks' maximum size would only be restricted by the amount of available virtual memory. In theory, it would speed up the access to the data.

In practice, the RAM-disk tends to push all other running applications into the swapfile, creating an even larger delay. It would require some testing on different machines to get a decent indication, but I think that this buffer performs worse than a simple MemoryStream.

*) See the VirtualLock entry on The Old New Thing [^]

Whether the system uses a MemoryStreamFile or a AweMemoryFile is determined when the application starts. I figured that there would be more people who'd want to try the difference, so it's easy to switch between RAM-Disk mode (using the AweBuffer) and Virtual Disk mode (using the MemoryStream). Simply toggle the bool in the Main-method of the Dokan.Mem project and recompile:

1  [STAThread()]
2  static void Main(string[] args)
3  {
4      // Set us up a tray-icon to interact with the user;
5      SetupNotifyIcon();
6
7      // Set to false if you want to test with the AweBuffers
8      // instead of a MemoryStream
9      MemoryFile.UseMemStream = true;

Points of Interest

In General

• You need to be logged in as an Administrator to run the file-system
• The code presented is of prototype-quality; there's no locking, no errorhandling, and just a very rudimentary UI.
• Paths and filenames are case-insensitive in Windows; different apps will use different casings when asking for your files.
• There's no special method to rename files (or folders), that's done by the MoveFile method.
• The folders are usually ordered in a hierarchy, but that's not required. One could omit the folderstructure completely and only show files (see the FindFiles method).
• You might want to disable any viruscheckers, as they tend to scan each file. The first file being requested after mounting is "AutoRun.Inf"
• The MemoryFile classes have the attribute FILE_ATTRIBUTE_NOT_CONTENT_INDEXED [^] to prevent Windows from content-indexing the files.
• The online readme [^] contains a more in-depth explanation on the inner structure of the Dokan-libraries and how they work.

If You are Interfacing with SQL Server

Because it's cool to be able to use Paint.NET over the explorer on a picture that's stored in SQL Server.

There's Stefan Delmarco's VarBinaryStream[^], providing convenient stream-based access to a VarBinary field. An implementation would go along these lines:

 1  public int ReadFile(string sourcePath, byte[] buffer, ref uint readBytes,
 2      long offset, DokanFileInfo info)
 3  {
 4      using (var con = new SqlConnection())
 5      {
 6          int fileId = GetFileIdByPath(sourcePath);
 7          using (var myVarBinarySource = new VarBinarySource(
 8              con,
 9              "[TableName]", // source table
10              "[Contents]",  // source column
11              "[Id]",        // key column
12              resourceId))
13          {
14              var dbStream = new VarBinaryStream(myVarBinarySource);
15              dbStream.Seek(offset, SeekOrigin.Begin);
16              readBytes = (uint) dbStream.Read(buffer, 0, buffer.Length);
17          }
18      }
19      return DokanNet.DOKAN_SUCCESS;
20  }

That's opening and closing database connections like crazy, but all in all, it performs quite well. Alternatively, you could open the connection in the CreateFile method, add it to a list, and close it again when the CloseFile method is called. If you need testdata; the AdventureWorks database contains a table called [Production].[Document], containing a few Word-documents as binary blobs.

Conclusion

Hiroki did a great job, the Dokan-libraries perform great. You can put breakpoints all over the place, and step through your code without any special settings :cool:

As for the RAM-Disk goes, it doesn't add much speed. The benchmark crashed when trying with the AweBuffer version, and it reported a mere 7 Mb/s when creating files. The maximum-speed during read-operations was around 40 Mb/s. As a comparison, my harddisk does around 64 Mb/s when creating files, with an average reading speed of 165 Mb/s.

History

• Initial version, 20-7-2010