Data Acquisition Library

• Download source for the article  

Introduction

In data acquisition software one needs to handle many challenges such as handling the data rate, handling the device, memory management and other issues. This library handles many of the issues related to data acquisition. In this article I am going to describe about the library that I have developed while working on many data acquisition and real-time data display projects. The library code is attached to this article

This library is entirely developed by me using .Net and C# programming. It is tested in many project and I have tested this library for handling data at more than 10 Mbps. But it can support higher data rate as well depending upon the acquisition device you are using. I am using this library in many projects or some modified form of this library in many data acquisition projects.

This library is highly extensible and flexible which is made by standard design patterns and principles. One basic principle is “program to interface and not to implementation” is utilize extensively in the library. By using this principle users of the library can write their own implementation of many interfaces provided in the library. In this way users can extend this library for their own applications. 

Features of the Library 

Data Acquisition

This library can acquire data for multiple data rates. Mostly this library is not tied to data rate but you can change the parameters for handling the data at your desire rate. This library provides data to the user immediately whatever present in the queue if no data is available it will return Null.  Data is hand-off to the user as soon as the call for data arrives, if data is not available during the call it will return Null. 

Data Saving

This library also supports the data saving feature simultaneously to acquisition. As soon as the data arrived it stores the data to a specified location on the hard disk. One can control this behaviour of the library by specifying the related parameters.  

Memory Management

If you are using 32 bit operating system it can only allow the program to utilize the memory limit up to 2 Gigabytes maximum. Now if your data is coming at the rate of about around 2 to 6 Mbps then it will take less than an hour to completely fill up your memory, provided, you are storing the data in the internal memory (RAM) for example in an array. In order to prevent this you need to empty the memory buffers after a certain amount of data is arrived. This library has this feature and handles the memory automatically for the user. This library empties and resizes the buffer after a particular size of data is arrived.  

Multi-Threaded and Thread Safe Code

Data acquisition and data saving are both performed in a separate thread than the calling thread. This allows us to gain performance and does not lock the calling thread.

Calls to the device are thread safe which means that another thread will not enter device calling routine if that routine is not completed. This allows us for data integrity and our data will not be corrupted during acquisition and saving.  

Multiple Devices 

One main feature of this library is that it can handle or work with many devices. User just needs to implement the simplest interface for that device and provide that implementation to this library and this library will start acquiring the data from that particular device.

Multiple Frame Formats 

This library can search and extract the specific frames from the incoming data. These data frames can be of any format. They can contain header to insure the start or end of the frames. This library can detect and extract these frames and transmit these frames to user instead of raw data. User can also provide the implementation for finding frames inside the data.

How to use this Library 

Library Class Diagram  

Let's talk about this class diagram.  DataAcquisisiton is the main class. It reference to interfaces to other classes. It does not know anything about the concrete implementation about the classes. Each concrete class implements the related interface. DataAcquisition class handles the writing of the data internally, whereas FrameSearcher and FrameFormatter passed to it externally. For demo I have included two device implementations and two frame searching algorithm implementations. 

One can use Idisplay to display some of the information here to user screen such as win-forms or console.  

General Steps for data acquisition 

How this library is implemented? There are some general steps which this library implements:

1. Acquire Data from device  
2. Search frames data if required
3. Format Data  if required
4. Hand-off the data to the user   

One thread inside the library continuously call the device for data acquisition, then search for frames in that data if found data then format that data and then hand-off that frame data to user. This thread continuously calls the device and performs aforementioned steps. Following code how this show the continuous calling to a method is implemented:   

 AutoResetEvent autoEvent = new AutoResetEvent(false);
  TimerCallback continuousTimerCallback = new TimerCallback(acquireData);
  device.Refresh();
         
  threadTimer = new System.Threading.Timer(continuousTimerCallback, autoEvent, 0, device.GetSubseque  ntReadTimeDelay() );

  autoEvent.WaitOne(-1);

 this code shows how the aforementioned steps are implemented: 

byte[] tempDataBuffer =  device.Read();

    if (tempDataBuffer != null)
    {

     // copy the data to the main buffer

    internalMemoryBuffer.AddRange(tempDataBuffer);

    // this will search and extract frames in the internal memory
    SearchFormatHandoff();

    EmptyInternalBuffer();// Empty the internal buffer and /or write the data chunk to hard disk        } 

  User can enable or disable some steps of data acquisition as shown in the following code :  

   public void EnableDataWriting(bool aval)
        {
            //check the user if data acquisition open do not set the control variables.
            if (!stopDataAcquisition)
            {

                isDataWritingRequired = aval;

                if (isDataWritingRequired)
                    this.dataWriter = new DataWriter();
            }
        }
        public void EnableFrameSearching(bool aval)
        {
            isFrameSearchingRequired = aval;
        }
        public void EnableDataFormatting(bool aval)
        {
            isDataFormattingRequired = aval;
        }

As you can see in the above code user can enable or disable data writing, frame searching and frame formatting. In this way user have the flexibility to control the data acquisition. 

How to Initialize, start and stop data acquisition?  

 NetworkAcquisitionDevice device  = new NetworkAcquisitionDevice("100.0.0.1",51212,30);
            FrameSearcherSecond frameSearcher = new FrameSearcherSecond();
            FormatterXorByte formatterCust = new FormatterXorByte();

            DataAcquisition daq = new DataAcquisition(device, frameSearcher, formatterCust);

            daq.EnableDataFormatting(true);
            daq.EnableFrameSearching(true);
            daq.EnableDataWriting(true); // data wil be in your harddisk.

            daq.StartAcquisition();
            bool stop =false;
            while (!stop)
            {

               byte[] dataFromDevice= daq.GetData();

                //process data here
            }

            daq.StopDataAcquisition();
            daq.Reset(); // it will reset all parameters.

In the above code I have use/setup for network data acquisition.  One can also use/setup for USB based data acquisition. One just need to change the NetworkAcquisitionDevice with USBAcquisitionDevice. 

How to consume data from the library?  

User can utilize the library in 2 ways. One is that the library itself pushes the data to calling user (also known as push method) and second one is that user call the library for data (also known as pop method). I have tested the application in both ways.

First method when you are popping data from the library make sure to get the data as soon as it is available. This will be extremely important when you are updating the user display with the data. In that case user will have to maintain the frame rate of the user screen. Also I have used the blocking collection and set the limit to 30 data frames. If you do not take fast enough then the data rate then blocking collection will not be blocked at 30 items. Hence when implementing the library keep this in mind.  

Second method is that implement the IDisplay interface because library has the reference to IDisplay interface it can call the display screen code. One point to note here is that in case of win-forms you will have to use Control.BegiInvoke to update the win forms control or dispatcher in case of WPF . This is because library is executing in a different thread than the thread from which you have created this library.

Following code will show the how to consume data. 

public byte[] GetData()
{

    byte[] dataResultWithTry = null;


    bool isDataAvailable = this.transferBuffer.TryTake(out dataResultWithTry);


    return dataResultWithTry;

}

and here is the code from where you can push the library. This is in DataAcquisition class and acquireData() method.  

       // now Hand off data..
             transferBuffer.TryAdd(locData);

       //publisher.UpdateUserInterface(locData);

 How this Library can be Extended? 

 How to Extend for Multiple Devices?  

One can extend the library for use with PCI interfacing, PCIe interfacing, parallel port interfacing and other interfacing as well. For that purpose user will need to implement a very simple interface for device acquisition such as open the device, close the device, refresh the device, get data bytes from the device. User can also configure after how much time (milliseconds) the library can call the device by implementing the method of IAcquisitionDevice interface and that method is GetSubsequentReadTimeDelay(). In this way one can use multiple devices for data acquisition with this library.     

I have tested this library with many devices such as serial port, FTDI USB device, National Instrument data acquisition device and with network(LAN and TCP/IP). For demonstration purpose I have included USB port interfacing and network interfacing.  

How to Extend other features  ? 

Other processing features which can be easily extended are frame searcher, frame data formatting and display interfaces.   

Frame searching can be extended by implementing a very simple interface IFrameSearcher. Similarly  frame data formatting can be extended by implementing a very simple interface for IFormatFrameData interface. Similarly users can extend for many user interface by implementing the IDisplay interface. User can provide implementation for either win-forms, console or WPF.  

References 

Check out this code at Github on  https://github.com/engrumair/DataAcquisitionLibraryDemo