Embedding Python in C/C++: Part II

• Download source files - 4.37 Kb
• Download demo project - 40.7 Kb

Introduction

This is part II of the article series. Part I had introduced Python/C API, a C library that helps to embed python modules into C/C++ applications. Basically, the API library is a bunch of C routines to initialize the Python Interpreter, call into your Python modules and finish up the embedding. In part I, I demonstrated how we can call functions, classes and methods defined within Python modules. Then, we discussed the details of multi-threaded embedding, an issue C/C++ programmers usually face during the integration stage. One question was raised during the discussion: How does our C/C++ code communicate with the embedded Python module when they are running on separate threads/processes? See the article: "Embedding Python in C/C++: Part I".

I am going to explore alternative solutions to this problem, IPC mechanisms in particular. As in part I, the article will not teach the Python language systematically, rather it will describe how the Python code works when it comes up. The discussion will be focused on how to integrate Python modules with your C/C++ applications. I will take the same practical approach as in Part I, but here I will present some limited theoretical discussions, thanks to the nature of the topics in this part. For example, the topic of shared memory deserves some theory. Still, I will leave most of the discussions to Jeffrey Richter's classic book.

Again the source code I provide is portable, meaning that it's aimed to run on both Windows and Linux. In order to use the source code, you should install a recent Python release, Visual C++ (or GCC compiler on Linux). The environment I have used to test is: Python 2.4 (Windows and Linux), Visual C++ 6.0 (Windows) or GCC 3.2 (RedHat 8.0 Linux). With Visual C++, select the Release configuration to build, as Debug configuration requires the Python debug library "python24_d.lib", which is not delivered with normal distributions.

Background

Python is a powerful interpreted language, like Java, Perl and PHP. It supports a long list of great features that any programmer would expect, two of my favorite features are "simple" and "portable". Along with the available tools and libraries, Python makes a good language for modeling and simulation developers. Best of all, it's free and the tools and libraries written for Python programmers are also free. For more details on the language, visit the official website.

TCP/IP sockets for embedding

Python has implemented several IPCs, including socket, memory mapped file (MMAP), queue, semaphore, event, lock, mutex, and so on. We are going to study two forms of IPC in the following: TCP/IP socket and MMAP. This section discusses a simple TCP/IP client/server model. The next section will describe how C/C++ and Python modules use MMAP to communicate with each other.

Let us start from a simple application, which implements a TCP client in the C code to communicate with a TCP server within a Python module. Here is the complete source "call_socket.c":

// call_socket.c - A sample program to 
// demonstrate the TCP client
//
#ifdef WIN32
#include <sys/types.h>
#include <Winsock2.h>
#define    WINSOCKVERSION    MAKEWORD( 2,2 )        
#else
#include <sys/socket.h>
#include <arpa/inet.h>
#include <unistd.h>
#include <time.h>
#endif

#include <stdio.h>
#include <string.h>

#define MAX_BUFFER    128
#define HOST        "127.0.0.1"
#define PORT         50007

int main ()
{
  int connectionFd, rc, index = 0, limit = MAX_BUFFER;
  struct sockaddr_in servAddr, localAddr;
  char buffer[MAX_BUFFER+1];

#ifdef WIN32
  // Start up WinSock2
  WSADATA wsaData;
  if( WSAStartup( WINSOCKVERSION, &wsaData) != 0 ) 
     return ERROR;        
#endif

  memset(&servAddr, 0, sizeof(servAddr));
  servAddr.sin_family = AF_INET;
  servAddr.sin_port = htons(PORT);
  servAddr.sin_addr.s_addr = inet_addr(HOST);

  // Create socket
  connectionFd = socket(AF_INET, SOCK_STREAM, 0);

  /* bind any port number */
  localAddr.sin_family = AF_INET;
  localAddr.sin_addr.s_addr = htonl(INADDR_ANY);
  localAddr.sin_port = htons(0);
  
  rc = bind(connectionFd, 
      (struct sockaddr *) &localAddr, sizeof(localAddr));

  // Connect to Server
  connect(connectionFd, 
      (struct sockaddr *)&servAddr, sizeof(servAddr));

  // Send request to Server
  sprintf( buffer, "%s", "Hello, Server!" );
  send( connectionFd, buffer, strlen(buffer), 0 );
 
  printf("Client sent to sever %s\n", buffer);

  // Receive data from Server
  sprintf( buffer, "%s", "" );
  recv(connectionFd, buffer, MAX_BUFFER, 0);
  printf("Client read from Server %s\n", buffer);

#ifdef WIN32
  closesocket(connectionFd);
#else
  close(connectionFd);
#endif
  
  printf("Client closed.\n");

  return(0);
}

The Python source file "py_socket_server.py" is as follows:

'''A sample of Python TCP server'''

import socket

HOST = '127.0.0.1'        # Local host
PORT = 50007              # Arbitrary port

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((HOST, PORT))
s.listen(1)

print 'Waiting for connection...'
conn, addr = s.accept()

print 'Connected by client', addr
while 1:
    data = conn.recv(1024)
    if not data: break
    print 'Received data from client', 
               repr(data), '...send it back...'
    conn.send(data)

conn.close()
print 'Server closed.'

On Windows, simply compile the C source and get the executable, which we call "call_socket.exe". To test this IPC, open a command window and start Python. Then import "py_socket_server" to start the Python TCP server. Open another command window, run "call_socket.exe". You will get the following outputs on the two windows:

Python 2.4.1 (#65, Mar 30 2005, 09:13:57) 
         [MSC v.1310 32 bit (Intel)] on win32
Type "help", "copyright", 
  "credits" or "license" for more information.
>>> import py_socket_server
Waiting for connection...
Connected by client ('127.0.0.1', 1482)
Received data from client 'Hello, 
              Server!' ...send it back...
Server closed.
>>>

C:\embedpython_2\embedpython_2_demo>call_socket
Client sent to sever "Hello, Server!"
Client read from Server "Hello, Server!"
Client closed.

C:\embedpython_2\embedpython_2_demo>

The C code can run on both Windows and Linux platforms. It could be further simplified by removing the portability. Note that the checks on the validity of returns are omitted for brevity. The C source is self-explanatory, except that we have to bind the local address with the client, which is usually unnecessary on the client side. In this integration, however, without the binding, the Python server reports the following error:

>>> import py_socket_server
Waiting for connection...
Connected by client ('127.0.0.1', 1479)
Received data from client Hello, Server! ...send it back...
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "py_socket_server.py", line 16, in ?
    data = conn.recv(1024)
socket.error: (10054, 'Connection reset by peer')

You might want to see how a Python TCP client looks like. Well, here is the Python source "py_socket_client.py". It is simple and clean. More importantly, it is portable from Windows to Linux!

'''A sample of Python TCP client'''

import socket

HOST = '127.0.0.1' # The localhost
PORT = 50007    # The same port as used by the server

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((HOST, PORT))

print 'Send to server', '\'Hello, world\''
s.send('Hello, world')

data = s.recv(1024)
print 'Received back', repr(data)

s.close()
print 'Client closed.'

It is fairly easy to use the above client/server model in multi-threaded embedding. Normally, we have two choices while running the Python module on a separate thread:

1. If you create the thread inside the Python module, place its server code in the Python thread function.
2. If you create the thread in the C/C++ code, call the Python server code from within the C thread function.

I leave this as an exercise to the reader. Refer to part I of this article sries for more details on the two approaches.

Shared memory (MMAP) for Python and C/C++

First, some theoretical preparation for this section. In Unix-like systems such as GNU/LINUX, shared memory segment and memory-mapped file (MMAP) are two different things. MMAP is memory-mapped file I/O. You can use MMAP as an IPC, but it is not very efficient, due to copying from each process' memory space to the disk file. In contrast, shared memory segment is a much faster form of IPC, because processes can share the memory segment in each of their address spaces. No disk copying or memory move-around.

Windows has implemented memory-mapped file (called "MMF") in a slightly different way. The MMF can be backed by either a user-defined disk file or by the system page file. When MMF is used as an IPC, Windows creates a named file-mapping (kernel) object. Through the kernel object, processes can map to the same disk file. This is the same as MMAP. But when MMF is backed by paging, this type of IPC can be very efficient. Because if you have got enough physical memory, paging will not be performed. It becomes a shared memory segment. Windows actually unifies MMAP and shared memory segment under the same cover of MMF! For more details on Windows implementation, refer to Jeffrey Richter's "Programming Applications for Microsoft Windows".

Now let's consider the following scenario. Somebody has written a Python module which is intended to run on a separate thread/process. It has defined an MMAP interface to communicate with the user of this module through MMAP. When we integrate it with our C/C++ application, we set up the MMAP interface for it and then start its execution. Our implementation on the client side is in "call_mmap.c". Here is the complete source:

// call_mmap.c - A sample of python embedding 
// (calling python functions 
// from within C code)
#include <Python.h>
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <errno.h>
#include <fcntl.h>
#include <string.h>

#ifdef WIN32    // Windows includes
#include <Windows.h>
#include <process.h>
#define sleep(x) Sleep(1000*x)
HANDLE hFile, handle, map;
#else        // POSIX includes
#include <pthread.h>
#include <sys/mman.h>
pthread_t mythread;
#endif

void myThread(void*);

#define NUM_ARGUMENTS 5
typedef struct 
{
   int argc;
   char *argv[NUM_ARGUMENTS]; 
} CMD_LINE_STRUCT;

int main(int argc, char *argv[])
{
    int i;
    char* indata = NULL;
    CMD_LINE_STRUCT cmd;
    
    cmd.argc = argc;
    for( i = 0; i < NUM_ARGUMENTS; i++ )
    {
        cmd.argv[i] = argv[i];
    }

    if (argc < 4) 
    {
        fprintf(stderr, "Usage: " + 
          "exe_name python_file class_name function_name\n");
        return 1;
    }

    /////////////////////////////////////////////////////////
    // Create a MMAP
    /////////////////////////////////////////////////////////
#ifdef WIN32
    // Create a memory-mapped file (MMF)
    hFile = CreateFile((LPCTSTR) "input.dat", 
        GENERIC_WRITE | GENERIC_READ, 
        FILE_SHARE_READ | FILE_SHARE_WRITE, NULL, 
        OPEN_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL);
    
    if(hFile == INVALID_HANDLE_VALUE) 
    {
    // Failed to create file 
    return 1;
    }

    map = CreateFileMapping(hFile, NULL, 
             PAGE_READWRITE, 0, 1024, "MMAPShmem");
    indata = (char *) MapViewOfFile (map, 
             FILE_MAP_READ | FILE_MAP_WRITE, 0, 0, 0); 
#else
    int fd;
    if((fd = open("input.dat", O_RDWR)) == -1)
    {
        printf("Couldn't open 'input.data'\n");
    }
    indata = mmap(    NULL, 1024, 
        PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
#endif

    if(indata != NULL)
    {    
        printf("Wrapper has created a MMAP " + 
                         "for file 'input.data'\n");
    }
 
    ///////////////////////////////////////////////
    // Create a thread
    ///////////////////////////////////////////////
#ifdef WIN32
    // Windows code
    handle = (HANDLE) _beginthread( myThread, 0, &cmd);
#else
    // POSIX code
    pthread_create( &mythread, NULL, myThread, 
                                         (void*)&cmd );
#endif

    // Random testing code
    for(i = 0; i < 10; i++)
    {
    memset(indata, 0, 1024);
    sprintf(indata, "%d", i);
    indata[3] = '\n';
      printf("The Main thread has writen %d to MMAP.\n", i);
    sleep(1);
    }

    printf("Main thread waiting for " + 
                "Python thread to complete...\n");

    // Join and wait for the created thread to complete...
#ifdef WIN32
    // Windows code
    WaitForSingleObject(handle,INFINITE);
    // Clean up
    UnmapViewOfFile(indata);
    CloseHandle(map);
    CloseHandle(hFile);
#else
    // POSIX code
    pthread_join(mythread, NULL);
    // Clean up
    munmap(indata, 1024);
    close(fd);
#endif
    
    printf("Main thread finished gracefully.\n");
    return 0;
}

void myThread( void *data )
{
    PyObject *pName, *pModule, *pDict, 
                            *pClass, *pInstance;
    PyThreadState *mainThreadState, 
                     *myThreadState, *tempState;
    PyInterpreterState *mainInterpreterState;
    
    CMD_LINE_STRUCT* arg = (CMD_LINE_STRUCT*)data;

    // Initialize python inerpreter
    Py_Initialize();
        
    // Initialize thread support
    PyEval_InitThreads();

    // Save a pointer to the main PyThreadState object
    mainThreadState = PyThreadState_Get();

    // Get a reference to the PyInterpreterState
    mainInterpreterState = mainThreadState->interp;

    // Create a thread state object for this thread
    myThreadState = PyThreadState_New(mainInterpreterState);

    // Swap in my thread state
    tempState = PyThreadState_Swap(myThreadState);

    // Now execute some python code (call python functions)
    pName = PyString_FromString(arg->argv[1]);
    pModule = PyImport_Import(pName);

    // pDict and pFunc are borrowed references 
    pDict = PyModule_GetDict(pModule);

    // Build the name of a callable class 
    pClass = PyDict_GetItemString(pDict, arg->argv[2]);

    // Create an instance of the class
    if (PyCallable_Check(pClass))
    {
    pInstance = PyObject_CallObject(pClass, NULL); 
    }

    // Call a method of the class with no parameters
    PyObject_CallMethod(pInstance, arg->argv[3], NULL);

    // Swap out the current thread
    PyThreadState_Swap(tempState);

    // Clean up thread state
    PyThreadState_Clear(myThreadState);
    PyThreadState_Delete(myThreadState);

    // Clean up
    Py_DECREF(pModule);
    Py_DECREF(pName);

    Py_Finalize();
    printf("My thread is finishing...\n");

    // Exiting the thread
#ifdef WIN32
    // Windows code
    _endthread();
#else
    // POSIX code
    pthread_exit(NULL);
#endif
}

The C application does the following:

• Create a disk file called "input.dat". Then, map the file to the memory space.
• Create a thread. The thread function executes the Python module.
• The C application thread writes to the MMAP file ten times. Note that the writing loop starts after the Python thread has been created and run.

Here is the Python source "py_mmap.py":

'''Python source designed to demonstrate '''
'''the use of python embedding'''

import sys
import os
import time
import mmap

INDATAFILENAME = 'input.dat'
LENGTHDATAMAP = 1024

class MMAPShmem:
    def run(self):
        inDataFile = open(INDATAFILENAME, 'r+')
        print 'inDataFile size: ', 
            os.path.getsize(INDATAFILENAME), 
            'MMAP size: ', LENGTHDATAMAP
        inDataNo = inDataFile.fileno() 
        
        inDataMap = mmap.mmap(inDataNo, LENGTHDATAMAP, 
                              access=mmap.ACCESS_WRITE) 
        inDataMap.seek(0)    # simple test of validity
    
        # write something into the mapped file
        x = 567
        inDataMap.write('%d' %x + '\n')

        for i in range(10):
            # read out from the file to verify
            inDataMap.seek(0)
            y = inDataMap.readline()
            print 'Python thread read from MMAP:', y 
            inDataMap.seek(0)
            inDataMap.write('%d' %x + '\n')
            print 'Python thread write back to MMAP:', x 
            time.sleep(1)

        inDataFile.close()

The Python module "py_mmap" defines one class "MMAPShmem", which has one method run(). All it does is opening the disk file created by the C code and mapping it to the memory. Then the module can use the mapped file just as you use a normal file I/O. In each for loop, Python reads MMAP and prints its contents. Then, it overwrites to the MMAP. Note that the ten reads/writes are running in parallel with the ten writes of the main C thread.

Open a command window and run "call_mmap py_mmap MMAPShmem run". You should get the output as shown below:

Wrapper has created a MMAP for file 'input.data'
The Main thread has writen 0 to MMAP.
inDataFile size:  1024 MMAP size:  1024
Python thread read from MMAP: 567

Python thread write back to MMAP: 567
The Main thread has writen 1 to MMAP.
Python thread read from MMAP: 1

Python thread write back to MMAP: 567
The Main thread has writen 2 to MMAP.
Python thread read from MMAP: 2

Python thread write back to MMAP: 567
The Main thread has writen 3 to MMAP.
Python thread read from MMAP: 3

Python thread write back to MMAP: 567
The Main thread has writen 4 to MMAP.
Python thread read from MMAP: 4

Python thread write back to MMAP: 567
The Main thread has writen 5 to MMAP.
Python thread read from MMAP: 5

Python thread write back to MMAP: 567
The Main thread has writen 6 to MMAP.
Python thread read from MMAP: 6

Python thread write back to MMAP: 567
The Main thread has writen 7 to MMAP.
Python thread read from MMAP: 7

Python thread write back to MMAP: 567
The Main thread has writen 8 to MMAP.
Python thread read from MMAP: 8

Python thread write back to MMAP: 567
The Main thread has writen 9 to MMAP.
Python thread read from MMAP: 9

Python thread write back to MMAP: 567
Main thread waiting for Python thread to complete...
My thread is finishing...
Main thread finished gracefully.

Apparently, the C and Python code running on two separate threads are communicating through the MMAP file "input.dat". In this case, since we have used text I/O (compared to binary I/O), you can actually check the contents.

Points of interest

Our MMAP has not implemented synchronization, which is usually required for data protection with the shared memory. In practice, you would want to coordinate access to the shared memory by multiple threads/processes. Otherwise, exclusiveness cannot be guaranteed and the data you get from the shared memory may be unpredictable.

Conclusion

We have demonstrated that we can utilize Python/C API to integrate Python modules with our C/C++ applications effectively. Our primary focus was how to embed Python in multi-threaded applications. The IPC as a communication mechanism between Python and C/C++ modules has been discussed in great depth. This is the concluding part of the article series.

History

• This is the first revision of the article and the source code.