ASP.NET file post direct to disk with upload file progress support

• Download source files - 431 KB
• Download demo project - 416 KB

Introduction

ASP.NET is a very nice and fast way to program small to enterprise level websites fast and efficiently. Most controls are perfect for any web development needed; however, the FileUpload control really has some shortcomings. Back when I used .NET 1.1, I did some testing to see how the control worked and handled large files, and was very unpleased.

I tried to upload a huge 1 or 2 gig file. I found myself with a problem where my web page would error, and the memory was never released. I left it at that, and did not mess with it anymore. I thought for sure that the 2.0 framework would have fixed that problem, but it appears that we are not that lucky (at least not that I could find).

Anyhow, the project I was working on requires that I have control of what my users do. You never know when someone is going to try and upload some huge file on purpose or by accident.

Purpose

This module was to provide a controlled way to process files and also to allow end users to see their current progress on the upload.

Code Examples

Below are the main source examples for this module. If you have searched the crap out of this subject on the internet, then you should see some similarities; however, there is one big difference that I have found that for some reason others are not doing. Let me explain:

Many examples on how to do this show developers converting byte data to strings and then searching for the string data. That is very inefficient. Most people make this module to use it for large file downloads. Most of the data that people are going to be posting is binary. Why tax your CPU with conversions to strings and then search the strings? I have no idea, but maybe that just sounded easier at the time. The way I do it is I search for byte patterns inside each byte buffer.

Another benefit is that it is possible for part of your file delimiters to be split up between multiple byte buffers. Most the examples I saw, people looked at the buffer once and then threw it away. I keep the last buffer in memory for reference, so if I did not find the start of a file in the buffer in my current buffer, then I merge the byte arrays together and then re-search.

I don't want to say that the way I am doing it is the best way, but I feel that just a few enhancements to the examples out there would make them worthy of deployment. Many of the examples that I tested would bring data in at approx 10 - 40 KB a second. Yeah.. like I am going to wait for a 100 - 2000 meg file to upload at that rate. With my module, I was able to bring in 1 - 4 Mbs per second with a low CPU.

Features

• Upload direct to disk support
• File progress bar (AJAX for the example)
• Automatic removal of files after a configured period of time
• Fast buffer parsing resulting in lower CPU requirements
• Configurable in app.config

How it works

The upload module implements IHttpModule and is hooked into the context.BeginRequest event. When that event is fired, I look to make sure that the page that is being requested is a page that may be uploading file(s) (configured in the web.config). I create my FileProcessor object and then start to get the buffered data from the worker object. My code comments will explain the process as it happens.

UploadModule.cs

using System;
using System.Collections.Generic;
using System.Text;
using System.Web;
using System.Reflection;
using System.IO;
using System.Diagnostics;
using System.Threading;

namespace TravisWhidden
{
    /// <summary>
    /// This module will provide alternate
    /// multi-part form posts abilities and also allow
    /// the developer to display current progress for the file being uploaded.
    /// </summary>
    public class UploadModule : System.Web.IHttpModule
    {
        #region "ClassLevelVars"

        /// <summary>
        /// This is the base path where all the files will be uploaded.
        /// This is set in the constructor. 
        /// Its value can be set by its default (built into the app),
        /// or it can be set in the web.config of the application
        /// </summary>
        private string _baseFileLocation = "";

        #endregion

        #region "Constructors"

        /// <summary>
        /// Default Constructor
        /// </summary>
        public UploadModule()
        {
            _baseFileLocation = 
              TravisWhidden.Properties.Settings.Default.BaseFileUpload;
        }

        #endregion

        #region "Properties"
        
        /// <summary>
        /// Used so we can read the base path from the config outside this assembly.
        /// </summary>
        public static string BasePath
        {
            get { return TravisWhidden.Properties.Settings.Default.BaseFileUpload; }
        }

        #endregion

        #region "Methods"

        /// <summary>
        /// Method checks to see if the page that is being requested
        /// is a page that this module will execute for. 
        /// </summary>
        /// <returns>Returns true if this page was
        /// listed in the web.config</returns>
        private bool IsUploadPages()
        {
            HttpApplication app = HttpContext.Current.ApplicationInstance;
            string[] uploadPages = (string[])
              TravisWhidden.Properties.Settings.Default.Pages.Split(
              new string[] {";"}, StringSplitOptions.RemoveEmptyEntries);
            for (int i = 0; i < uploadPages.Length; i++)
            {
                if (uploadPages[i].ToLower() == 
                            app.Request.Path.Substring(1).ToLower())
                    return true;
            }
            return false;
        }

        #endregion

        #region IHttpModule Members

        /// <summary>
        /// Cleanup interace object
        /// </summary>
        void System.Web.IHttpModule.Dispose()
        {
            
        }

        /// <summary>
        /// Interface method for Init
        /// </summary>
        /// <param name="context"></param>
        void System.Web.IHttpModule.Init(System.Web.HttpApplication context)
        {

            // Attach handler to BeginRequest so we can process the messages
            context.BeginRequest += new EventHandler(context_BeginRequest);

        }

        #endregion

        #region "Event Handlers / Events"

        /// <summary>
        /// Method handler used to process the byte data being pushed in.
        /// </summary>
        /// <param name="sender">HttpApplication</param>
        /// <param name="e">EventArgs</param>
        void context_BeginRequest(object sender, EventArgs e)
        {
            // Get the HttpApplication object
            HttpApplication httpApplication = (HttpApplication)sender;
            // Get the current Context object
            HttpContext context = httpApplication.Context;
            // Create an instance of the file processor

            // before we go any further, lets make sure we are to even watch this page
            if (!IsUploadPages())
            {
                // Exit the method.
                Debug.WriteLine("UploadModule - Not IsUploadPages().");
                return;
            }
            Debug.WriteLine("UploadModule - IsUploadPages().");

            // Create the file processor object
            // with the base path in its constructor. this object
            // will process all the byte data that we recive through the form post.
            FileProcessor fp = new FileProcessor(_baseFileLocation);

            // Get the current URL
            string rawURL = context.Request.RawUrl;

            // UniqueIdentifier to represent this post.
            Guid currentFileID = Guid.NewGuid();

            // the UploadStatus object is stored
            // in the Application memory that is global to the application.
            // this is what we will use to track the upload process
            UploadStatus uploadStatus = new UploadStatus(currentFileID);

            // Make sure we have an identifier. We use a post
            // identifier so we can track outside our upload the 
            // upload status. If it does not have postID,
            // we will add one to the current URL, and redirect the form
            // to itself with one in it.
            if (context.Request.QueryString["PostID"] != null && 
                context.Request.QueryString["PostID"].Length == 
                                Guid.NewGuid().ToString().Length)
            {
                currentFileID = new Guid(context.Request.QueryString["PostID"]);
            }
            else
            {
                // Redirect to this same page, but with a PostID
                if (rawURL.IndexOf("?") == -1)
                {
                    rawURL = rawURL + "?PostID=" + currentFileID.ToString();
                }
                else
                {
                    rawURL = rawURL + "&PostID=" + currentFileID.ToString();
                }
                context.Response.Redirect(rawURL);
            }

 

            // Make sure this is a multi-part form. If not, exit
            if (context.Request.ContentType.IndexOf("multipart/form-data") == -1)
            {
                // Not multi-part form
                return;
            }
            Debug.WriteLine("UploadModule Executing.");
            
            // Get the worker request object. This object provides us with the byte data 
            // as the user is uploading it. This is a very critical part of this module.
            HttpWorkerRequest workerRequest = 
              (HttpWorkerRequest)context.GetType().GetProperty(
               "WorkerRequest",  BindingFlags.Instance | 
               BindingFlags.NonPublic).GetValue(context, null);

            // Indicates if the worker request has a body
            if (workerRequest.HasEntityBody())
            {
                try
                {
                    // Add the upload status to the appliation object.
                    context.Application.Add(currentFileID.ToString(), uploadStatus);

                    // Get the byte size of the form post. We need this
                    // to detect how much we have left to go.
                    long contentLength = long.Parse((workerRequest.GetKnownRequestHeader(
                                                     HttpWorkerRequest.HeaderContentLength)));
                    
                    // the upload status object sets the length of the object. 
                    // This is used for progress tracking outside this module.
                    uploadStatus.TotalBytes = contentLength;

                    long receivedcount = 0;
                    // this is the default buffer size. It appears that the
                    // most you can get out of the worker request object
                    // so we might as well just use these chunks. 
                    long defaultBuffer = 8192;

                    // Get the preloaded buffer data. I have seen
                    // some problems with this running in the 
                    // Visual Studios 2005 built in webserver. 
                    // Sometimes it will return null, but I have not seen this happen on 
                    // my windows 2000 or 2003 server boxes. I researched this problem
                    // and could not find any solution to the built in VS webserver,
                    // but that does not matter because that is just for testing
                    // anyways. You could always user your local IIS to test this module.
                    byte[] preloadedBufferData = workerRequest.GetPreloadedEntityBody();

                    // Just a check to throw an error that is understandable
                    if (preloadedBufferData == null)
                    {
                        throw new Exception("GetPreloadedEntityBody() " + 
                                            "was null. Try again");
                    }

                    // Extract header information needed.
                    fp.GetFieldSeperators(ref preloadedBufferData);
                    
                    // Process this buffer.
                    fp.ProcessBuffer(ref preloadedBufferData, true);

                    // Update the status object.
                    uploadStatus.CurrentBytesTransfered += preloadedBufferData.Length;

                    // It is possible for all the data that was in the form post to be
                    // inside one buffer. if that is true,
                    // then the IsEntireEntityBodyIsPreloaded() 
                    // will return true. There is no reason to continue on.
                    if (!workerRequest.IsEntireEntityBodyIsPreloaded())
                    {
                        // Data is not all preloaded.
                        do
                        {
                            // Because of a problem where we will 
                            // be waiting for the ReadEntityBody to end
                            // when there is nothing left in the
                            // buffer, we have to only fill what is 
                            // needed to finish the buffer.
                            // When the last buffer comes up, we have to resize
                            // the buffer to finish the rest
                            // of the array. This will allow us to exit
                            // this loop faster. If we tried to fill
                            // the buffer when there was nothing left to be
                            // filled, it would just hang till what I assumed
                            // timed out waiting for more buffer data internally.
                            long tempBufferSize = (uploadStatus.TotalBytes - 
                                                   uploadStatus.CurrentBytesTransfered);
                            if (tempBufferSize < defaultBuffer)
                            {
                                defaultBuffer = tempBufferSize;
                            }

                            // Create the new byte buffer with the default size.
                            byte[] bufferData = new byte[defaultBuffer];
                            // Ask the worker request for the buffer chunk.
                            receivedcount = 
                              workerRequest.ReadEntityBody(bufferData, bufferData.Length);

                            // Process the buffered data.
                            fp.ProcessBuffer(ref bufferData, true);

                            // Update the status object.
                            uploadStatus.CurrentBytesTransfered += bufferData.Length;

                        } while (receivedcount != 0);
                    }
 
                }
                catch (Exception ex)
                {

                    Debug.WriteLine("Error: " + ex.Message);
                    Debug.WriteLine(ex.ToString());
                }
                finally
                {
                    // makes sure that any open streams are closed for safety
                    fp.CloseStreams();
                }

                // join the array of finished files.
                string finishedFiles = string.Join(";", 
                                         fp.FinishedFiles.ToArray()); 
                // Add to query string.
                if (rawURL.IndexOf("?") == -1 && 
                                           finishedFiles.Length > 0)
                {
                    rawURL = rawURL + "?finishedFiles=" + finishedFiles;
                }else{
                    rawURL = rawURL + "&finishedFiles=" + finishedFiles;
                }

                // dispose the FileProcessor object.
                fp.Dispose();

                // redirect the user back to the page it was
                // posted on, but with the new URL variables.
                context.Response.Redirect(rawURL);
            }
        }

        #endregion
    }
}

FileProcessor.cs

The FileProcessor class will be processing the buffers that come in off the WorkerRequest object. It is responsible for finding the files in the form post, extracting the data out, and keeping track of the files it extracted.

using System;
using System.Collections.Generic;
using System.Text;
using System.Diagnostics;
using System.Text;
using System.IO;
using System.Threading;

namespace TravisWhidden
{
    /// <summary>
    /// FileProcessor is used to process byte data from
    /// a multi-part form and save it to the disk.
    /// </summary>
    public class FileProcessor : IDisposable
    {

        #region "Class Vars"
        /// <summary>
        /// Form post id is used in finding field seperators in the multi-part form post
        /// </summary>
        private string _formPostID = "";
        /// <summary>
        /// Used to find the start of a file
        /// </summary>
        private string _fieldSeperator = "";
        
        /// <summary>
        /// Used to note what buffer index we are on in the collection
        /// </summary>
        private long _currentBufferIndex = 0;

        /// <summary>
        /// Used to flag each byte process if we have
        /// already found the start of a file or not
        /// </summary>
        private bool _startFound = false;

        /// <summary>
        /// Used to flag each byte process if we have already found the end of a file.
        /// </summary>
        private bool _endFound = false;

        /// <summary>
        /// Default string to where the files are to be uploaded.
        /// It will be overrided on the constructor
        /// </summary>
        private string _currentFilePath = @"C:\upload\";

        /// <summary>
        /// The way I wrote this is that I did not care
        /// about the file name in the form. I generate the filename
        /// to prevent anyone from posting two of the same
        /// files at the same time. Each file is unique
        /// and the form is redirected with the filenames
        /// as URL params so the .net application can handle
        /// the finished files as it wants.
        /// </summary>
        private string _currentFileName = Guid.NewGuid().ToString() + ".bin";

        /// <summary>
        /// FileStream object that is left open while a file
        /// is beting written to. Each file will open and 
        /// close its filestream automaticly
        /// </summary>
        private FileStream _fileStream = null;

        /// <summary>
        /// The following fields are used in the byte searching of buffer datas
        /// </summary>
        private long _startIndexBufferID = -1;
        private int _startLocationInBufferID = -1;

        private long _endIndexBufferID = -1;
        private int _endLocationInBufferID = -1;


        /// <summary>
        /// Dictionary array used to store byte chunks.
        /// Should store a max of 2 items. 2 history chunks are kept.
        /// </summary>
        private Dictionary<long, byte[]> _bufferHistory = 
                                new Dictionary<long, byte[]>();

        /// <summary>
        /// Object to hold all the finished filenames.
        /// </summary>
        private List<string> _finishedFiles = new List<string>();

        #endregion

        #region "Constructors"

        /// <summary>
        /// Default constructor with the path of where the files should be uploaded.
        /// </summary>
        public FileProcessor(string uploadLocation)
        {
            // This is the path where the files will be uploaded too.
            _currentFilePath = uploadLocation;
        }

        #endregion

        #region "Properties"

        /// <summary>
        /// Property used to get the finished files.
        /// </summary>
        public List<string> FinishedFiles
        {
            get { return _finishedFiles; }
        }

        #endregion

        #region "Methods"

        /// <summary>
        /// Method that is used to process the buffer. It may call itself several times
        /// as it looks for new files that may be possible in each buffer.
        /// </summary>
        /// <param name="bufferData">Byte data to scan</param>
        /// <param name="addToBufferHistory">If true,
        /// it will add it to the buffer history. If false, it will not.</param>
        public void ProcessBuffer(ref byte[] bufferData, bool addToBufferHistory)
        {
            int byteLocation = -1;

            // If the start has not been found, search for it.
            if (!_startFound)
            {
                // Search for start location
                byteLocation = GetStartBytePosition(ref bufferData);
                if (byteLocation != -1)
                {
                    // Set the start position to this current index
                    _startIndexBufferID = _currentBufferIndex + 1;
                    // Set the start location in the index
                    _startLocationInBufferID = byteLocation;

                    _startFound = true;

                }
            }

            // If the start was found, we can start to store the data into a file.
            if (_startFound)
            {
                // Save this data to a file
                // Have to find the end point.
                // Makes sure the end is not in the same buffer

                int startLocation = 0;
                if (byteLocation != -1)
                {
                    startLocation = byteLocation;
                }

                // Write the data from the start point to the end point to the file.

                int writeBytes = ( bufferData.Length - startLocation );
                int tempEndByte = GetEndBytePosition(ref bufferData);
                if (tempEndByte != -1)
                {
                    writeBytes = (tempEndByte - startLocation);
                    // not that the end was found.
                    _endFound = true;
                    // Set the current index the file was found
                    _endIndexBufferID = (_currentBufferIndex + 1);
                    // Set the current byte location
                    // for the assoicated index the file was found
                    _endLocationInBufferID = tempEndByte;
                }

                // Make sure we have something to write.
                if (writeBytes > 0)
                {
                    if (_fileStream == null)
                    {
                        // create a new file stream to be used.
                        _fileStream = new FileStream(_currentFilePath + 
                            _currentFileName, FileMode.OpenOrCreate);
                        
                        // this will create a time to live for the
                        // file so it will automaticly be removed
                        int fileTimeToLive = 
                         global::TravisWhidden.Properties.Settings.Default.FileTimeToLive;
                        
                        // if the form were not to handle the file and remove
                        // it, this is an automatic removal of the file
                        // the timer object will execute in x number
                        // of seconds (can override in the web.config file)
                        Timer t = new Timer(new TimerCallback(DeleteFile), 
                           _currentFilePath + _currentFileName, 
                           (fileTimeToLive * 1000), 0);

                    }
                    // Write the datat to the file and flush it.
                    _fileStream.Write(bufferData, startLocation, writeBytes);
                    _fileStream.Flush();
                }
            }

            // If the end was found, then we need
            // to close the stream now that we are done with it.
            // We will also re-process this buffer
            // to make sure the start of another file 
            // is not located within it.
            if (_endFound)
            {
                CloseStreams();
                _startFound = false;
                _endFound = false;

                // Research the current buffer for a new start location. 
                ProcessBuffer(ref bufferData, false);
            }

            // Add to buffer history
            if (addToBufferHistory)
            {
                // Add to history object.
                _bufferHistory.Add(_currentBufferIndex, bufferData);
                _currentBufferIndex++;
                // Cleanup old buffer references
                RemoveOldBufferData();
            }
        }

        /// <summary>
        /// Method used to clean up the internal buffer array. 
        /// Older elements are not needed, only a history of one is needed.
        /// </summary>
        private void RemoveOldBufferData()
        {
            for (long bufferIndex = _currentBufferIndex; 
                          bufferIndex >= 0; bufferIndex--)
            {
                if (bufferIndex > (_currentBufferIndex - 3))
                {
                    // Dont touch. preserving the last 2 items.
                }
                else
                {
                    if (_bufferHistory.ContainsKey(bufferIndex))
                    {
                        _bufferHistory.Remove(bufferIndex);
                    }
                    else
                    {
                        // no more previous buffers. 
                        bufferIndex = 0;
                    }
                }
            }
            GC.Collect();
        }

        /// <summary>
        /// Close the stream, and reset the filename.
        /// </summary>
        public void CloseStreams()
        {
            if (_fileStream != null)
            {
                _fileStream.Dispose();
                _fileStream.Close();
                _fileStream = null;

                // add the file name to the finished list.
                _finishedFiles.Add(_currentFileName);

                // Reset the filename.
                _currentFileName = Guid.NewGuid().ToString() + ".bin";
            }
        }

        /// <summary>
        /// This method should be ran on the bytes
        /// that are returned on GetPreloadedEntityBody().
        /// </summary>
        /// <param name="bufferData"></param>
        public void GetFieldSeperators(ref byte[] bufferData)
        {
            try
            {
                _formPostID = Encoding.UTF8.GetString(bufferData).Substring(29, 13);
                _fieldSeperator = 
                  "-----------------------------" + _formPostID;
            }
            catch (Exception ex)
            {
                Debug.WriteLine("Error in GetFieldSeperators(): " + ex.Message);
            }
        }

        /// <summary>
        /// Method used for searching buffer data, and if needed previous buffer data.
        /// </summary>
        /// <param name="bufferData">current
        /// buffer data needing to be processed.</param>
        /// <returns>Returns byte location of data to start</returns>
        private int GetStartBytePosition(ref byte[] bufferData)
        {

            int byteOffset = 0;
            // Check to see if the current bufferIndex
            // is the same as any previous index found.
            // If it is, offset the searching by the previous location
            if (_startIndexBufferID == (_currentBufferIndex + 1))
            {
                byteOffset = _startLocationInBufferID;
            }

            // Check to see if the end index was found before
            // this start index. That way we keep moving ahead
            if (_endIndexBufferID == (_currentBufferIndex +1))
            {
                byteOffset = _endLocationInBufferID;
            }

            int tempContentTypeStart = -1;
            // First see if we can find it in the current buffer batch.
            // Because there may be muliple posts
            // in a form, we have to make sure we do not
            // re-return any possible byte offsets
            // that we have returned before. This could lead
            // to an infinite loop.

            byte[] searchString = Encoding.UTF8.GetBytes("Content-Type: ");
            tempContentTypeStart = 
              FindBytePattern(ref bufferData, ref searchString, byteOffset);

            if (tempContentTypeStart != -1)
            {
                // Found content type start location
                // Next search for \r\n\r\n at this substring
                //int tempSearchStringLocation = 
                //  bufferDataUTF8.IndexOf("\r\n\r\n", tempContentTypeStart);
                searchString = Encoding.UTF8.GetBytes("\r\n\r\n");
                int tempSearchStringLocation = FindBytePattern(ref bufferData, 
                                      ref searchString, tempContentTypeStart);

                if (tempSearchStringLocation != -1)
                {
                    // Found this. Can get start of data here
                    // Add 4 to it. That is the number of positions
                    // before it gets to the start of the data
                    int byteStart = tempSearchStringLocation + 4;
                    return byteStart;
                }
            }
            else if((byteOffset - searchString.Length) > 0 ){

                return -1;
            }

            else
            {
                // Did not find it. Add this and previous
                // buffer together to see if it exists.
                // Check to see if the buffer index is at the start. 
                if (_currentBufferIndex > 0)
                {
                    // Get the previous buffer
                    byte[] previousBuffer = _bufferHistory[_currentBufferIndex - 1];
                    byte[] mergedBytes = MergeArrays(ref previousBuffer, ref bufferData);
                    // Get the byte array for the text
                    byte[] searchString2 = Encoding.UTF8.GetBytes("Content-Type: ");
                    // Search the bytes for the searchString
                    tempContentTypeStart = FindBytePattern(ref mergedBytes, 
                      ref searchString2, previousBuffer.Length - searchString2.Length);

                    //tempContentTypeStart = 
                    //         combinedUTF8Data.IndexOf("Content-Type: ");
                    if (tempContentTypeStart != -1)
                    {
                        // Found content type start location
                        // Next search for \r\n\r\n at this substring

                        searchString2 = Encoding.UTF8.GetBytes("Content-Type: ");
                        // because we are searching part of the previosu buffer,
                        // we only need to go back the length of the search 
                        // array. Any further, and our normal if statement
                        // would have picked it up when it first was processed.
                        int tempSearchStringLocation = FindBytePattern(ref mergedBytes, 
                           ref searchString2, (previousBuffer.Length - searchString2.Length));

                        if (tempSearchStringLocation != -1)
                        {
                            // Found this. Can get start of data here
                            // It is possible for some of this
                            // to be located in the previous buffer.
                            // Find out where the excape chars are located.
                            if (tempSearchStringLocation > previousBuffer.Length)
                            {
                                // Located in the second object. 
                                // If we used the previous buffer, we should
                                // not have to worry about going out of
                                // range unless the buffer was set to some
                                // really low number. So not going to check for
                                // out of range issues.
                                int currentBufferByteLocation = 
                                   (tempSearchStringLocation - previousBuffer.Length);
                                return currentBufferByteLocation;
                            }
                            else
                            {
                                // Located in the first object.
                                // The only reason this could happen is if
                                // the escape chars ended right
                                // at the end of the buffer. This would mean
                                // that that the next buffer would start data at offset 0
                                return 0;
                            }
                        }
                    }
                }
            }
            // indicate not found.
            return -1;
        }

        /// <summary>
        /// Method used for searching buffer data for end
        /// byte location, and if needed previous buffer data.
        /// </summary>
        /// <param name="bufferData">current
        /// buffer data needing to be processed.</param>
        /// <returns>Returns byte location of data to start</returns>
        private int GetEndBytePosition(ref byte[] bufferData)
        {

            int byteOffset = 0;
            // Check to see if the current bufferIndex
            // is the same as any previous index found.
            // If it is, offset the searching by the previous
            // location. This will allow us to find the next leading
            // Stop location so we do not return a byte offset
            // that may have happened before the start index.
            if (_startIndexBufferID == (_currentBufferIndex + 1))
            {
                byteOffset = _startLocationInBufferID;
            }

            int tempFieldSeperator = -1;

            // First see if we can find it in the current buffer batch.
            byte[] searchString = Encoding.UTF8.GetBytes(_fieldSeperator);
            tempFieldSeperator = FindBytePattern(ref bufferData, 
                                     ref searchString, byteOffset);

            if (tempFieldSeperator != -1)
            {
                // Found field ending. Depending on where the field
                // seperator is located on this, we may have to move back into
                // the prevoius buffer to return its offset.
                if (tempFieldSeperator - 2 < 0)
                {
                    //TODO: Have to get the previous buffer data.
                    
                }
                else
                {
                    return (tempFieldSeperator - 2);
                }
            }
            else if ((byteOffset - searchString.Length) > 0)
            {

                return -1;
            }
            else
            {
                // Did not find it. Add this and
                // previous buffer together to see if it exists.
                // Check to see if the buffer index is at the start. 
                if (_currentBufferIndex > 0)
                {

                    // Get the previous buffer
                    byte[] previousBuffer = _bufferHistory[_currentBufferIndex - 1];
                    byte[] mergedBytes = MergeArrays(ref previousBuffer, ref bufferData);
                    // Get the byte array for the text
                    byte[] searchString2 = Encoding.UTF8.GetBytes(_fieldSeperator);
                    // Search the bytes for the searchString
                    tempFieldSeperator = FindBytePattern(ref mergedBytes, 
                       ref searchString2, 
                       previousBuffer.Length - searchString2.Length + byteOffset);

                    if (tempFieldSeperator != -1)
                    {
                        // Found content type start location
                        // Next search for \r\n\r\n at this substring

                        searchString2 = Encoding.UTF8.GetBytes("\r\n\r\n");
                        int tempSearchStringLocation = FindBytePattern(ref mergedBytes, 
                            ref searchString2, tempFieldSeperator);
                        
                        if (tempSearchStringLocation != -1)
                        {
                            // Found this. Can get start of data here
                            // It is possible for some of this
                            // to be located in the previous buffer.
                            // Find out where the excape chars are located.
                            if (tempSearchStringLocation > previousBuffer.Length)
                            {
                                // Located in the second object. 
                                // If we used the previous buffer,
                                // we shoudl not have to worry about going out of
                                // range unless the buffer was set to some
                                // really low number. So not going to check for
                                // out of range issues.
                                int currentBufferByteLocation = 
                                  (tempSearchStringLocation - previousBuffer.Length);
                                return currentBufferByteLocation;
                            }
                            else
                            {
                                // Located in the first object. 
                                // The only reason this could happen is if
                                // the escape chars ended right
                                // at the end of the buffer. This would mean
                                // that that the next buffer would start data at offset 0
                                return -1;
                            }
                        }
                    }
                }
            }
            // indicate not found.
            return -1;
        }

        /// <summary>
        /// Method created to search for byte array patterns inside a byte array.
        /// </summary>
        /// <param name="containerBytes">byte array to search</param>
        /// <param name="searchBytes">byte
        ///   array with pattern to search with</param>
        /// <param name="startAtIndex">byte offset
        ///   to start searching at a specified location</param>
        /// <returns>-1 if not found or index
        /// of starting location of pattern</returns>
        private static int FindBytePattern(ref byte[] containerBytes, 
                ref byte[] searchBytes, int startAtIndex)
        {
            int returnValue = -1;
            for (int byteIndex = startAtIndex; byteIndex < 
                            containerBytes.Length; byteIndex++)
            {

                // Make sure the searchBytes lenght does not exceed the containerbytes
                if (byteIndex + searchBytes.Length > containerBytes.Length)
                {
                    // return -1.
                    return -1;
                }

                // First the first reference of the bytes to search
                if (containerBytes == searchBytes[0])
                {
                    bool found = true;
                    int tempStartIndex = byteIndex;
                    for (int searchBytesIndex = 1; searchBytesIndex < 
                                  searchBytes.Length; searchBytesIndex++)
                    {
                        // Set next index
                        tempStartIndex++;
                        if (!(searchBytes[searchBytesIndex] == 
                                     containerBytes[tempStartIndex]))
                        {
                            found = false;
                            // break out of the loop and continue searching.
                            break;
                        }
                    }
                    if (found)
                    {
                        // Indicates that the byte array has been found. Return this index.
                        return byteIndex;
                    }
                }
            }
            return returnValue;
        }

        /// <summary>
        /// Used to merge two byte arrays into one. 
        /// </summary>
        /// <param name="arrayOne">First byte array
        ///    to go to the start of the new array</param>
        /// <param name="arrayTwo">Second byte array
        ///    to go to right after the first array that was passed</param>
        /// <returns>new byte array of all the new bytes</returns>
        private static byte[] MergeArrays(ref byte[] arrayOne, ref byte[] arrayTwo)
        {
            System.Type elementType = arrayOne.GetType().GetElementType();
            byte[] newArray = new byte[arrayOne.Length + arrayTwo.Length];
            arrayOne.CopyTo(newArray, 0);
            arrayTwo.CopyTo(newArray, arrayOne.Length);

            return newArray;
        }

        /// <summary>
        /// This is used as part of the clean up procedures.
        /// the Timer object will execute this function.
        /// </summary>
        /// <param name="filePath"></param>
        private static void DeleteFile(object filePath)
        {
            // File may have already been removed from the main appliation.
            try
            {
                if (System.IO.File.Exists((string)filePath))
                {
                    System.IO.File.Delete((string)filePath);
                }
            }
            catch { }
        }

        #endregion

        #region IDisposable Members

        /// <summary>
        /// Clean up method.
        /// </summary>
        public void Dispose()
        {
            // Clear the buffer history
            _bufferHistory.Clear();
            GC.Collect();
        }

        #endregion
    }
}

Instructions for use

This module will be seen by every page; however, we don't want to use it unless it's the file upload page. You will see in the web.config a section for setting the pages to watch, along with some other settings:

<applicationSettings>
   <TravisWhidden.Properties.Settings>
      <setting name="BaseFileUpload" serializeAs="String">
         <value>C:\upload\</value>
      </setting>
      <setting name="FileTimeToLive" serializeAs="String">
         <value>3600</value>
      </setting>
      <setting name="Pages" serializeAs="String">
         <value>UploadModuleExample/Default.aspx;UploadModuleAtlasExample/
                  Default.aspx;/MembersOnly/ImageUploadManagement.aspx</value>
      </setting>
   </TravisWhidden.Properties.Settings>
</applicationSettings>

Closing

This module is to be distributed free of charge. If you plan to sell this module, at least send me some of the proceeds. He he.

Contact Info

You can email me at travis@lvfbody.com if you have any questions or comments. I would love to hear about how my module is being used and how it is acting.

References

1. http://forums.asp.net/thread/106552.aspx

Description: Very good information on how you can get this to work. Some code examples are extremely inefficient, but it gets the point across.

Other references (which pretty much look like stuff on ASP.NET forums) can be found on CodeProject.com. Some of the examples seen did not do all in one.

Additional Information

Note: as I was writing this up, I found this MSDN reference, and I don't know if this project could have been made simpler. I hope we did not re-invent the wheel. It may be nothing, but worth reading about anyways: http://msdn2.microsoft.com/en-us/library/system.web.httppostedfile.aspx.

This is what caught my eye: "Files are uploaded in MIME multipart/form-data format. By default, all requests, including form fields and uploaded files, larger than 256 KB are buffered to disk, rather than held in server memory."