AxPipe - Multi-Threaded Binary Stream C++ Class Library

• Download source with documentation and demo project - 450 Kb

Introduction

AxPipe is a very small and efficient class library implementing an extended stream and pipe abstraction similar to Unix Shell pipes. Both push and pull model processing, run-time built multi-stage pipes, optional multi-threading applied per stage, splits, joins and a very small footprint are distinguishing features. It's fully documented, with sample code and some stock transformations included.

The current implementation is for Win32 API with Visual Studio 6 or 2002, but porting to Unix should pose little difficulty, only the threading and co-routine support needs modification. AxPipe is a light-weight library easily learned and applied. Currently it is being used for development of the next version of AxCrypt, http://axcrypt.sourceforge.net.

It's still in active development, and I'd appreciate any and all constructive feedback and suggestions for improvements, or reasons for moving to an existing framework or library that does it better/faster/smaller/neater/safer/whatever.

Using the code

The source code comes with a complete demo application and complete documentation for every namespace, class, member, method, enumeration, define, template etc.

All is also available on the project home page, http://axpipe.sourceforge.net, where updated versions will be found as well. There you will find code samples and further overviews.

Introduction

The basic paradigm is taken from the Unix Shell pipe notation, where you might write:

crypt <file.txt | compress | tar >out.tar
     but you can also write, for example,
tar <file.txt | crypt | compress >out.z

The programs above are semi-ficticious, it's just to demonstrate the principle whereby input sources such as a file can be redirected into a processing program, which sends it on, where it can be connected to another processing program, or to a final destination.

I've frequently wanted to use the same principle for programming purposes in C++, but with minimal overhead and supporting different programming models. So I wrote this package.

The following is a minimal complete program demonstrating the basic usage.

/*! \file
    \brief A most simple AxPipe program

    Read an input file, pass it through a do-nothing
    pipe, and write it to a file. No error checking
    at all for clarity.
*/

#include "stdafx.h"

#include "AxPipe.h"

#include "CFileMap.h"


/// \brief Just pass data along - do nothing with it.

class CPipeDoNothing : public AxPipe::CPipe {
    void Out(AxPipe::CSeg *pSeg) {
        // Insert your code here...

        Pump(pSeg);
    }
};

int
_tmain(int argc, _TCHAR* argv[]) {
    AxPipe::CGlobalInit axpipeInit;    // It just has to be there.


    AxPipe::CSourceMemFile sourceFile; // The source is a memory mapped file


    sourceFile.Init(argv[1]);          // Set the source file name

    // Append a stage, in it's own thread

    sourceFile.Append(new AxPipe::CThread<CPipeDoNothing>);
    // Continue to append a sink to accept the output

    sourceFile.Append((new AxPipe::CSinkMemFile)->Init(argv[2]));

    // Initialize, Process the data, End process and Finalize.

    sourceFile.Open()->Drain()->Close()->Plug();

    // Check for any errors in any parts...

    if (sourceFile.GetErrorCode()) {
        // Print a clear text representation of the problem

        fprintf(stderr, sourceFile.GetErrorMsg());
        return sourceFile.GetErrorCode();
    }

    return 0;
}

Points of Interest

If you've never seen the use for co-routines (CreateFiber() et. al. in Win32 API) here's a pretty nice use, turning a push model processing filter into a pull model ditto. Contrary, I believe, to most similar packages AxPipe allows you to optionally, and in run-time, build processing pipes with different stages and with or without threading on a stage-by-stage basis.

It's very restrictive in it's use of external dependencies, and is suitable for very small applications even with minimal run-time support. It does not use exceptions for that reason, and certainly not MFC or similar large frameworks. There are also some 'stock' transformations included, developed on a needed basis for the projects where I use AxPipe, but the long-term idea is to gather a number of common sources, transformations and sinks into a very easily re-usable library of ready-made components.