Introduction
This is part II of the article series. Part I had introduced Python/C API, a C library that helps to embed python modules into C/C++ applications. Basically, the API library is a bunch of C routines to initialize the Python Interpreter, call into your Python modules and finish up the embedding. In part I, I demonstrated how we can call functions, classes and methods defined within Python modules. Then, we discussed the details of multi-threaded embedding, an issue C/C++ programmers usually face during the integration stage. One question was raised during the discussion: How does our C/C++ code communicate with the embedded Python module when they are running on separate threads/processes? See the article: "Embedding Python in C/C++: Part I".
I am going to explore alternative solutions to this problem, IPC mechanisms in particular. As in part I, the article will not teach the Python language systematically, rather it will describe how the Python code works when it comes up. The discussion will be focused on how to integrate Python modules with your C/C++ applications. I will take the same practical approach as in Part I, but here I will present some limited theoretical discussions, thanks to the nature of the topics in this part. For example, the topic of shared memory deserves some theory. Still, I will leave most of the discussions to Jeffrey Richter's classic book.
Again the source code I provide is portable, meaning that it's aimed to run on both Windows and Linux. In order to use the source code, you should install a recent Python release, Visual C++ (or GCC compiler on Linux). The environment I have used to test is: Python 2.4 (Windows and Linux), Visual C++ 6.0 (Windows) or GCC 3.2 (RedHat 8.0 Linux). With Visual C++, select the Release configuration to build, as Debug configuration requires the Python debug library "python24_d.lib", which is not delivered with normal distributions.
Background
Python is a powerful interpreted language, like Java, Perl and PHP. It supports a long list of great features that any programmer would expect, two of my favorite features are "simple" and "portable". Along with the available tools and libraries, Python makes a good language for modeling and simulation developers. Best of all, it's free and the tools and libraries written for Python programmers are also free. For more details on the language, visit the official website.
TCP/IP sockets for embedding
Python has implemented several IPCs, including socket, memory mapped file (MMAP), queue, semaphore, event, lock, mutex, and so on. We are going to study two forms of IPC in the following: TCP/IP socket and MMAP. This section discusses a simple TCP/IP client/server model. The next section will describe how C/C++ and Python modules use MMAP to communicate with each other.
Let us start from a simple application, which implements a TCP client in the C code to communicate with a TCP server within a Python module. Here is the complete source "call_socket.c":
#ifdef WIN32
#include <sys/types.h>
#include <Winsock2.h>
#define WINSOCKVERSION MAKEWORD( 2,2 )
#else
#include <sys/socket.h>
#include <arpa/inet.h>
#include <unistd.h>
#include <time.h>
#endif
#include <stdio.h>
#include <string.h>
#define MAX_BUFFER 128
#define HOST "127.0.0.1"
#define PORT 50007
int main ()
{
int connectionFd, rc, index = 0, limit = MAX_BUFFER;
struct sockaddr_in servAddr, localAddr;
char buffer[MAX_BUFFER+1];
#ifdef WIN32
WSADATA wsaData;
if( WSAStartup( WINSOCKVERSION, &wsaData) != 0 )
return ERROR;
#endif
memset(&servAddr, 0, sizeof(servAddr));
servAddr.sin_family = AF_INET;
servAddr.sin_port = htons(PORT);
servAddr.sin_addr.s_addr = inet_addr(HOST);
connectionFd = socket(AF_INET, SOCK_STREAM, 0);
localAddr.sin_family = AF_INET;
localAddr.sin_addr.s_addr = htonl(INADDR_ANY);
localAddr.sin_port = htons(0);
rc = bind(connectionFd,
(struct sockaddr *) &localAddr, sizeof(localAddr));
connect(connectionFd,
(struct sockaddr *)&servAddr, sizeof(servAddr));
sprintf( buffer, "%s", "Hello, Server!" );
send( connectionFd, buffer, strlen(buffer), 0 );
printf("Client sent to sever %s\n", buffer);
sprintf( buffer, "%s", "" );
recv(connectionFd, buffer, MAX_BUFFER, 0);
printf("Client read from Server %s\n", buffer);
#ifdef WIN32
closesocket(connectionFd);
#else
close(connectionFd);
#endif
printf("Client closed.\n");
return(0);
}
The Python source file "py_socket_server.py" is as follows:
'''A sample of Python TCP server'''
import socket
HOST = '127.0.0.1'
PORT = 50007
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((HOST, PORT))
s.listen(1)
print 'Waiting for connection...'
conn, addr = s.accept()
print 'Connected by client', addr
while 1:
data = conn.recv(1024)
if not data: break
print 'Received data from client',
repr(data), '...send it back...'
conn.send(data)
conn.close()
print 'Server closed.'
On Windows, simply compile the C source and get the executable, which we call "call_socket.exe". To test this IPC, open a command window and start Python. Then import "py_socket_server" to start the Python TCP server. Open another command window, run "call_socket.exe". You will get the following outputs on the two windows:
Python 2.4.1 (#65, Mar 30 2005, 09:13:57)
[MSC v.1310 32 bit (Intel)] on win32
Type "help", "copyright",
"credits" or "license" for more information.
>>> import py_socket_server
Waiting for connection...
Connected by client ('127.0.0.1', 1482)
Received data from client 'Hello,
Server!' ...send it back...
Server closed.
>>>
C:\embedpython_2\embedpython_2_demo>call_socket
Client sent to sever "Hello, Server!"
Client read from Server "Hello, Server!"
Client closed.
C:\embedpython_2\embedpython_2_demo>
The C code can run on both Windows and Linux platforms. It could be further simplified by removing the portability. Note that the checks on the validity of returns are omitted for brevity. The C source is self-explanatory, except that we have to bind the local address with the client, which is usually unnecessary on the client side. In this integration, however, without the binding, the Python server reports the following error:
>>> import py_socket_server
Waiting for connection...
Connected by client ('127.0.0.1', 1479)
Received data from client Hello, Server! ...send it back...
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "py_socket_server.py", line 16, in ?
data = conn.recv(1024)
socket.error: (10054, 'Connection reset by peer')
You might want to see how a Python TCP client looks like. Well, here is the Python source "py_socket_client.py". It is simple and clean. More importantly, it is portable from Windows to Linux!
'''A sample of Python TCP client'''
import socket
HOST = '127.0.0.1'
PORT = 50007
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((HOST, PORT))
print 'Send to server', '\'Hello, world\''
s.send('Hello, world')
data = s.recv(1024)
print 'Received back', repr(data)
s.close()
print 'Client closed.'
It is fairly easy to use the above client/server model in multi-threaded embedding. Normally, we have two choices while running the Python module on a separate thread:
- If you create the thread inside the Python module, place its server code in the Python thread function.
- If you create the thread in the C/C++ code, call the Python server code from within the C thread function.
I leave this as an exercise to the reader. Refer to part I of this article sries for more details on the two approaches.
Shared memory (MMAP) for Python and C/C++
First, some theoretical preparation for this section. In Unix-like systems such as GNU/LINUX, shared memory segment and memory-mapped file (MMAP) are two different things. MMAP is memory-mapped file I/O. You can use MMAP as an IPC, but it is not very efficient, due to copying from each process' memory space to the disk file. In contrast, shared memory segment is a much faster form of IPC, because processes can share the memory segment in each of their address spaces. No disk copying or memory move-around.
Windows has implemented memory-mapped file (called "MMF") in a slightly different way. The MMF can be backed by either a user-defined disk file or by the system page file. When MMF is used as an IPC, Windows creates a named file-mapping (kernel) object. Through the kernel object, processes can map to the same disk file. This is the same as MMAP. But when MMF is backed by paging, this type of IPC can be very efficient. Because if you have got enough physical memory, paging will not be performed. It becomes a shared memory segment. Windows actually unifies MMAP and shared memory segment under the same cover of MMF! For more details on Windows implementation, refer to Jeffrey Richter's "Programming Applications for Microsoft Windows".
Now let's consider the following scenario. Somebody has written a Python module which is intended to run on a separate thread/process. It has defined an MMAP interface to communicate with the user of this module through MMAP. When we integrate it with our C/C++ application, we set up the MMAP interface for it and then start its execution. Our implementation on the client side is in "call_mmap.c". Here is the complete source:
#include <Python.h>
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <errno.h>
#include <fcntl.h>
#include <string.h>
#ifdef WIN32 // Windows includes
#include <Windows.h>
#include <process.h>
#define sleep(x) Sleep(1000*x)
HANDLE hFile, handle, map;
#else // POSIX includes
#include <pthread.h>
#include <sys/mman.h>
pthread_t mythread;
#endif
void myThread(void*);
#define NUM_ARGUMENTS 5
typedef struct
{
int argc;
char *argv[NUM_ARGUMENTS];
} CMD_LINE_STRUCT;
int main(int argc, char *argv[])
{
int i;
char* indata = NULL;
CMD_LINE_STRUCT cmd;
cmd.argc = argc;
for( i = 0; i < NUM_ARGUMENTS; i++ )
{
cmd.argv[i] = argv[i];
}
if (argc < 4)
{
fprintf(stderr, "Usage: " +
"exe_name python_file class_name function_name\n");
return 1;
}
#ifdef WIN32
hFile = CreateFile((LPCTSTR) "input.dat",
GENERIC_WRITE | GENERIC_READ,
FILE_SHARE_READ | FILE_SHARE_WRITE, NULL,
OPEN_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL);
if(hFile == INVALID_HANDLE_VALUE)
{
return 1;
}
map = CreateFileMapping(hFile, NULL,
PAGE_READWRITE, 0, 1024, "MMAPShmem");
indata = (char *) MapViewOfFile (map,
FILE_MAP_READ | FILE_MAP_WRITE, 0, 0, 0);
#else
int fd;
if((fd = open("input.dat", O_RDWR)) == -1)
{
printf("Couldn't open 'input.data'\n");
}
indata = mmap( NULL, 1024,
PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
#endif
if(indata != NULL)
{
printf("Wrapper has created a MMAP " +
"for file 'input.data'\n");
}
#ifdef WIN32
handle = (HANDLE) _beginthread( myThread, 0, &cmd);
#else
pthread_create( &mythread, NULL, myThread,
(void*)&cmd );
#endif
for(i = 0; i < 10; i++)
{
memset(indata, 0, 1024);
sprintf(indata, "%d", i);
indata[3] = '\n';
printf("The Main thread has writen %d to MMAP.\n", i);
sleep(1);
}
printf("Main thread waiting for " +
"Python thread to complete...\n");
#ifdef WIN32
WaitForSingleObject(handle,INFINITE);
UnmapViewOfFile(indata);
CloseHandle(map);
CloseHandle(hFile);
#else
pthread_join(mythread, NULL);
munmap(indata, 1024);
close(fd);
#endif
printf("Main thread finished gracefully.\n");
return 0;
}
void myThread( void *data )
{
PyObject *pName, *pModule, *pDict,
*pClass, *pInstance;
PyThreadState *mainThreadState,
*myThreadState, *tempState;
PyInterpreterState *mainInterpreterState;
CMD_LINE_STRUCT* arg = (CMD_LINE_STRUCT*)data;
Py_Initialize();
PyEval_InitThreads();
mainThreadState = PyThreadState_Get();
mainInterpreterState = mainThreadState->interp;
myThreadState = PyThreadState_New(mainInterpreterState);
tempState = PyThreadState_Swap(myThreadState);
pName = PyString_FromString(arg->argv[1]);
pModule = PyImport_Import(pName);
pDict = PyModule_GetDict(pModule);
pClass = PyDict_GetItemString(pDict, arg->argv[2]);
if (PyCallable_Check(pClass))
{
pInstance = PyObject_CallObject(pClass, NULL);
}
PyObject_CallMethod(pInstance, arg->argv[3], NULL);
PyThreadState_Swap(tempState);
PyThreadState_Clear(myThreadState);
PyThreadState_Delete(myThreadState);
Py_DECREF(pModule);
Py_DECREF(pName);
Py_Finalize();
printf("My thread is finishing...\n");
#ifdef WIN32
_endthread();
#else
pthread_exit(NULL);
#endif
}
The C application does the following:
- Create a disk file called "input.dat". Then, map the file to the memory space.
- Create a thread. The thread function executes the Python module.
- The C application thread writes to the MMAP file ten times. Note that the writing loop starts after the Python thread has been created and run.
Here is the Python source "py_mmap.py":
'''Python source designed to demonstrate '''
'''the use of python embedding'''
import sys
import os
import time
import mmap
INDATAFILENAME = 'input.dat'
LENGTHDATAMAP = 1024
class MMAPShmem:
def run(self):
inDataFile = open(INDATAFILENAME, 'r+')
print 'inDataFile size: ',
os.path.getsize(INDATAFILENAME),
'MMAP size: ', LENGTHDATAMAP
inDataNo = inDataFile.fileno()
inDataMap = mmap.mmap(inDataNo, LENGTHDATAMAP,
access=mmap.ACCESS_WRITE)
inDataMap.seek(0)
x = 567
inDataMap.write('%d' %x + '\n')
for i in range(10):
inDataMap.seek(0)
y = inDataMap.readline()
print 'Python thread read from MMAP:', y
inDataMap.seek(0)
inDataMap.write('%d' %x + '\n')
print 'Python thread write back to MMAP:', x
time.sleep(1)
inDataFile.close()
The Python module "py_mmap" defines one class "MMAPShmem
", which has one method run()
. All it does is opening the disk file created by the C code and mapping it to the memory. Then the module can use the mapped file just as you use a normal file I/O. In each for
loop, Python reads MMAP and prints its contents. Then, it overwrites to the MMAP. Note that the ten reads/writes are running in parallel with the ten writes of the main C thread.
Open a command window and run "call_mmap py_mmap MMAPShmem run
". You should get the output as shown below:
Wrapper has created a MMAP for file 'input.data'
The Main thread has writen 0 to MMAP.
inDataFile size: 1024 MMAP size: 1024
Python thread read from MMAP: 567
Python thread write back to MMAP: 567
The Main thread has writen 1 to MMAP.
Python thread read from MMAP: 1
Python thread write back to MMAP: 567
The Main thread has writen 2 to MMAP.
Python thread read from MMAP: 2
Python thread write back to MMAP: 567
The Main thread has writen 3 to MMAP.
Python thread read from MMAP: 3
Python thread write back to MMAP: 567
The Main thread has writen 4 to MMAP.
Python thread read from MMAP: 4
Python thread write back to MMAP: 567
The Main thread has writen 5 to MMAP.
Python thread read from MMAP: 5
Python thread write back to MMAP: 567
The Main thread has writen 6 to MMAP.
Python thread read from MMAP: 6
Python thread write back to MMAP: 567
The Main thread has writen 7 to MMAP.
Python thread read from MMAP: 7
Python thread write back to MMAP: 567
The Main thread has writen 8 to MMAP.
Python thread read from MMAP: 8
Python thread write back to MMAP: 567
The Main thread has writen 9 to MMAP.
Python thread read from MMAP: 9
Python thread write back to MMAP: 567
Main thread waiting for Python thread to complete...
My thread is finishing...
Main thread finished gracefully.
Apparently, the C and Python code running on two separate threads are communicating through the MMAP file "input.dat". In this case, since we have used text I/O (compared to binary I/O), you can actually check the contents.
Points of interest
Our MMAP has not implemented synchronization, which is usually required for data protection with the shared memory. In practice, you would want to coordinate access to the shared memory by multiple threads/processes. Otherwise, exclusiveness cannot be guaranteed and the data you get from the shared memory may be unpredictable.
Conclusion
We have demonstrated that we can utilize Python/C API to integrate Python modules with our C/C++ applications effectively. Our primary focus was how to embed Python in multi-threaded applications. The IPC as a communication mechanism between Python and C/C++ modules has been discussed in great depth. This is the concluding part of the article series.
History
- This is the first revision of the article and the source code.