Power of C++ - Developing a Portable HTTP Server with Python Interpreter using Boost, TinyXML

• Download demo (Windows version) - 351.79 KB

Remarks

• To start server, go to the server directory and run ahttpserver.exe start
• To see server statistics, run ahttpserver.exe stat
• To stop server, run ahttpserver.exe stop
• To play with Python interpreter, Python 2.5 should be installed
• To remove Python support, remove all <handlers> tags with content from server.config file

• Download sources (Visual Studio .NET 2005 solution/gcc makefile) - 142.92 KB

Remarks

• Project created against Boost 1.34.1 - you should have it to compile solution
• To compile python_handler, you need Python 2.5 installed
• To debug ahttpserver under Visual Studio .NET 2005, set ahttpserver->Properties->Debugging->Command Arguments: run

Introduction

First of all - this article is not a presentation of stable application - ahttpserver is just a result of my creative investigation of modern C++ programming approaches. At my current work, I cannot satisfy my curiosity and taste for new programming technologies learning and it was the root cause of this project starting - I have started it to refresh/extend and fix my C++ skills.

HTTP server application was selected as a complex set of functional parts:

• TCP sockets + HTTP protocol
• Parallel HTTP requests processing (multi-threading)
• Easy-to-modify server settings storage (XML, INI files)
• Interaction with server file system
• Modular architecture (plugins support)
• Server-side scripting support
• Server application features (service/daemon)
• Errors logging/server control

Also several targets were selected at the beginning of the project:

1. Code must be portable (project should be compiled/run at least under Windows and Linux). I have installed Ubuntu Linux 7.10 six months ago and admire it very much.
2. Code should be organized in independent parts which can be reused in other projects.
3. Project code has to be as small as possible (known C++ libraries can be utilized).

At present, the ahttpserver project contains three main parts:

1. ahttplib static library (aconnect and ahttp namespaces)
2. ahttpserver - server application core
3. python_handler - module for server-side Python scripts execution

Using the Code

So let's start from a simple example of aconnect library in action - the following code presents a very simple echo server.

#include "aconnect/aconnect.hpp"
#include "aconnect/util.hpp"

using namespace aconnect;

// global server instance
Server  server;

void threadProc (const ClientInfo& client) {
    static EndMarkSocketStateCheck check;
    string request = client.getRequest (check);
    request.erase ( request.find(check.endMark()) );

    string response;
    bool stopServer = false;

    if (util::equals (request, "STOP") ) {
        response = "Processed";
        stopServer = true;
    } else {
        response = "Echo: " + request;
    }
    // write response
    client.writeResponse(response);

    if (stopServer)
        exit (0);
}

// test it: http://localhost:8888/
int main (int argc, char* args[])
{
    Initializer init;
    FileLogger logger;
    // {timestamp} - will be replaced with generated timestamp
    // (example: 22_05_2008_20_17_35),
    // third parameter - max. size of log file - it will be rotated automatically
    logger.init (Log::Debug, "c:\\temp\\server_log_{timestamp}.log", 4194304);

    // init command server
    ServerSettings settings;
    settings.socketReadTimeout =
        settings.socketWriteTimeout = 300; // sec

    // init HTTP server
    server.setLog ( &logger);
    server.init (8888, threadProc, settings);

    server.start(); // started in child thread
    server.join();
}

Initializer is an RAII-style guard to init OS-dependant network functionality - under Windows it calls WSAStartup in the constructor and WSACleanup in the destructor.
Server is a main functional class - it creates TCP server socket, binds it to selected port (8888 in code) and start listening on this port. Server can be started in background thread (as in example) or in the main execution thread: server.start (true).
At server initialization, ServerSettings object is applied to server.

// server settings storage - used to setup default server settings
struct ServerSettings
{
    int    backlog;
    int    domain;
    bool    reuseAddr;
    bool    enablePooling;
    int    workersCount;

    int    workerLifeTime;            // sec
    int    socketReadTimeout;         // sec
    int    socketWriteTimeout;        // sec

    // default settings
    ServerSettings () :
        backlog (SOMAXCONN),    // backlog in listen() call
        domain (AF_INET),       // domain for 'socket' function call
        reuseAddr (false),      // SO_REUSEADDR flag setup on server socket
        enablePooling (true),   // show whether create worker-threads pool or not
        workersCount (500),     // maximum worker-threads count
        workerLifeTime (300),   // thread in pool lifetime
        socketReadTimeout (60), // server socket SO_RCVTIMEO timeout
        socketWriteTimeout (60) // server socket SO_SNDTIMEO timeout
    { }
};

Each accepted TCP connection is processed in the background worker thread - portable Boost.Thread library us used. Simple threads pool is implemented in aconnect::Server using boost::mutex and boost::condition. If the enablePooling field in server settings is true, then when initial TCP interaction is finished, worker thread starts waiting for a new request during workerLifeTime time. If no requests are found when timeout ends, then the thread is removed from the pool.

When server accepts client TCP connection, then it fills ClientInfo object with client related data.

struct ClientInfo
{
    port_type    port;            // int
    ip_addr_type    ip;           // unsigned char[4]
    socket_type    socket;        // OS-depended, under Win32 - SOCKET, Linux - int
    class Server    *server;
};

After client information loading execution is transferred to the worker thread (new or borrowed from pool), that executes the thread procedure (threadProc in code).

FileLogger - aconnect::Logger interface implementation to log messages to files. aconnect::Logger is a simple example of logging functionality developed in log4... manner - it contains a set of logging methods: info, warn, error to log message with the appropriate level.
ConsoleLogger writes messages to std::cout and FileLogger writes messages to file, FileLogger can rotate files when maximum file size achieved. FileLogger initialization is too simple - just define log level, path to file and maximum size of one log file (default size: 4 MB).

ahttp Library

aconnect library was developed at first turn - besides Server class, it contains a set of utility (types definitions, socket control, string comparison, date/time functions).

After TCP server implementation ahttp library was developed - this library contains HttpServer definition and a set of HTTP protocol related functionality. To start ahttp::HttpServer, one need have filled the HttpServerSettings instance (see listening) - even for this simple server, there are many settings. As a result, the preferred place to store these settings is an XML file which can be updated by hand and loaded quickly. See an example of settings file in sources ahttpserver sources (out\server.config file). Some settings are described in this file, the others can be understood from context.

class HttpServerSettings
{
public:

    HttpServerSettings();
    ~HttpServerSettings();

    void load (aconnect::string_constptr docPath) throw (settings_load_error);
    ...

protected:
    aconnect::ServerSettings settings_;
    aconnect::port_type port_;
    aconnect::port_type commandPort_;
    aconnect::string rootDirName_;

    aconnect::string appLocaton_;
    // logger
    aconnect::Log::LogLevel logLevel_;
    aconnect::string logFileTemplate_;
    size_t maxLogFileSize_;

    bool enableKeepAlive_;
    int keepAliveTimeout_;
    int commandSocketTimeout_;
    size_t responseBufferSize_;
    size_t maxChunkSize_;

    directories_map directories_;
    aconnect::str2str_map mimeTypes_;

    aconnect::Logger*    logger_;
    aconnect::string    serverVersion_;

    global_handlers_map registeredHandlers_;
    bool                firstLoad_;
    aconnect::string    directoryConfigFile_;
};

At present, ahttp::HttpServer has the following features:

• HTTP methods GET/POST/HEAD
• POST parameters parsing + uploaded files processing ("multipart/form-data" request type)
• Keep-alive mode
• "chunked" transfer encoding - support for dynamically created response
• MIME type evaluation by file extension (out\mime-types.config)
• Automatic URL mapping - (see sample in out\web\directory.config)
• Default documents loading (index.html)
• Directories browsing with customizable UI

Server has modular architecture - new functionality can be added as new handler. Handlers are connected to server by HttpServerSettings - <handlers> section in configuration file. Handlers are applied to file extensions (".py" for example, "." - empty extension) to all files from current virtual directory ("*"). Each handler has a main processing function:

HANDLER_EXPORT bool processHandlerRequest (ahttp::HttpContext& context);

If this function returns true (request completed) - processing of HTTP handler will be stopped after this function call. In the other case - next registered handler will be applied (this mode can be used in cache/authentication/extended logging modules).

See more details in code - I tried to write all code as simple as possible.

Points of Interest

As a result, at the time of ahttpserver development, I have investigated a set of useful libraries which I can use in the future in professional work. Firstly, I want to mention boost libraries Thread, Filesystem and String Algorithm - each of them is a great thing and contains many useful functionalities for everyday programming tasks. Boost Python Library is the perfect tool to write own Python modules - the library's use of advanced metaprogramming techniques simplifies its syntax for users, so that wrapping code takes on the look of a kind of declarative interface definition language (IDL).

Working on this application has not stopped yet - there are several uncompleted (planned) tasks:

• gzip/deflate content encoding support (with zlib or boost::iostreams)
• Server messages localization
• Basic authentication handler
• In-memory cache handler
• Python interpreter handler redevelopment to avoid crashes at mutlithreaded requests processing (at present, there are only several locks in critical places). I have tried several workarounds from Python documentation: PyGILState_Ensure/PyGILState_Release, PyEval_SaveThread... but a quick applicable solution has not been found yet. I planned to investigate existing Python modules (mod_wsgi, PyISAPIe)

History

• 2008-05-29: Version 0.1 published