Introduction
Inspired by the article "Embedding Python in Multi-Threaded C/C++ Applications" (Linux Journal), I felt the need for a more comprehensive coverage on the topic of embedding Python. While writing this article, I had two objectives:
- This is written for programmers who are more experienced in C/C++ than in Python, the tutorial takes a practical approach and omits all theoretical discussions.
- Try to maintain Python's cross-platform compatibility when writing the embedding code.
Now, you have got some modules written in Python by others and you want to use them. You are experienced in C/C++, but fairly new to Python. You might wonder if you can find a tool to convert them to C code, like the conversion from FORTRAN. The answer is no. Some tools can help you generate the executable from a Python module. Is the problem solved? No. Converting code into executables usually makes things even more complicated, as you must figure out how your C/C++ application communicates with the executable "black-box".
I am going to introduce C/C++ programmers to Python/C API, a C library that helps to embed python modules into C/C++ applications. The API library provides a bunch of C routines to initialize the Python Interpreter, call into your Python modules and finish up the embedding. The library is built with Python and distributed with all the recent Python releases.
Part I of this article series discusses the basics of Python embedding. Part II will move on to more advanced topics. This tutorial does not teach the Python language systematically, but I will briefly describe how the Python code works when it comes up. The emphasis will be on how to integrate Python modules with your C/C++ applications. See article: "Embedding Python in C/C++: Part II".
In order to use the source code, you should install a recent Python release, Visual C++ (or GCC compiler on Linux). The environment that I have used to test is: Python 2.4 (Windows and Linux), Visual C++ 6.0 (Windows) or GCC 3.2 (RedHat 8.0 Linux). With Visual C++, select the Release configuration to build. Debug configuration requires Python debug library "python24_d.lib", which is not delivered with normal distributions.
Background
Python is a powerful interpreted language, like Java, Perl and PHP. It supports a long list of great features that any programmer would expect, two of my favorite features are "simple" and "portable". Along with the available tools and libraries, Python makes a good language for modeling and simulation developers. Best of all, it's free and the tools and libraries written for Python programmers are also free. For more details on the language, visit the official website.
Embedding basics: functions, classes and methods
First, let us start from a sample C program that calls a function within a Python module. Here is the source file "call_function.c":
#include <Python.h>
int main(int argc, char *argv[])
{
PyObject *pName, *pModule, *pDict, *pFunc, *pValue;
if (argc < 3)
{
printf("Usage: exe_name python_source function_name\n");
return 1;
}
Py_Initialize();
pName = PyString_FromString(argv[1]);
pModule = PyImport_Import(pName);
pDict = PyModule_GetDict(pModule);
pFunc = PyDict_GetItemString(pDict, argv[2]);
if (PyCallable_Check(pFunc))
{
PyObject_CallObject(pFunc, NULL);
} else
{
PyErr_Print();
}
Py_DECREF(pModule);
Py_DECREF(pName);
Py_Finalize();
return 0;
}
The Python source file "py_function.py" is as follows:
'''py_function.py - Python source designed to '''
'''demonstrate the use of python embedding'''
def multiply():
c = 12345*6789
print 'The result of 12345 x 6789 :', c
return c
Note that checks for the validity of objects are omitted for brevity. On Windows, simply compile the C source and get the executable, which we call "call_function.exe". To run it, enter the command line "call_function py_function multiply". The second argument is the name of the Python file (without extension), which when loaded becomes the module name. The third argument is the name of the Python function you are going to call within the module. The Python source does not take part in compiling or linking; it's only loaded and interpreted at run-time. The output from the execution is:
The result of 12345 x 6789 : 83810205
The C code itself is self-explanatory, except that:
- Everything in Python is an object.
pDict and pFunc are borrowed references so we don't need to Py_DECREF() them.
- All the Py_XXX and PyXXX_XXX calls are Python/C API calls.
- The code will compile and run on all the platforms that Python supports.
Now, we want to pass arguments to the Python function. We add a block to handle arguments to the call:
if (PyCallable_Check(pFunc))
{
if( argc > 3 )
{
pArgs = PyTuple_New(argc - 3);
for (i = 0; i < argc - 3; i++)
{
pValue = PyInt_FromLong(atoi(argv[i + 3]));
if (!pValue)
{
PyErr_Print();
return 1;
}
PyTuple_SetItem(pArgs, i, pValue);
}
pValue = PyObject_CallObject(pFunc, pArgs);
if (pArgs != NULL)
{
Py_DECREF(pArgs);
}
} else
{
pValue = PyObject_CallObject(pFunc, NULL);
}
if (pValue != NULL)
{
printf("Return of call : %d\n", PyInt_AsLong(pValue));
Py_DECREF(pValue);
}
else
{
PyErr_Print();
}
}
The new C source adds a block of "Prepare the argument list for the call" and a check of the returned value. It creates a tuple (list-like type) to store all the parameters for the call. You can run the command "call_function py_source multiply1 6 7" and get the output:
The result of 6 x 7 : 42
Return of call : 42
Writing classes in Python is easy. It is also easy to use a Python class in your C code. All you need to do is create an instance of the object and call its methods, just as you call normal functions. Here is an example:
#include <Python.h>
int main(int argc, char *argv[])
{
PyObject *pName, *pModule, *pDict,
*pClass, *pInstance, *pValue;
int i, arg[2];
if (argc < 4)
{
printf(
"Usage: exe_name python_fileclass_name function_name\n");
return 1;
}
pClass = PyDict_GetItemString(pDict, argv[2]);
if (PyCallable_Check(pClass))
{
pInstance = PyObject_CallObject(pClass, NULL);
}
if( argc > 4 )
{
for (i = 0; i < argc - 4; i++)
{
arg[i] = atoi(argv[i + 4]);
}
pValue = PyObject_CallMethod(pInstance,
argv[3], "(ii)", arg[0], arg[1]);
} else
{
pValue = PyObject_CallMethod(pInstance, argv[3], NULL);
}
if (pValue != NULL)
{
printf("Return of call : %d\n", PyInt_AsLong(pValue));
Py_DECREF(pValue);
}
else
{
PyErr_Print();
}
}
The third parameter to PyObject_CallMethod(), "(ii)" is a format string, which indicates that the next arguments are two integers. Note that PyObject_CallMethod() takes C variables types as its arguments, not Python objects. This is different from the other calls we have seen so far. The Python source "py_class.py" is as follows:
'''py_class.py - Python source designed to demonstrate'''
'''the use of python embedding'''
class Multiply:
def __init__(self):
self.a = 6
self.b = 5
def multiply(self):
c = self.a*self.b
print 'The result of', self.a, 'x', self.b, ':', c
return c
def multiply2(self, a, b):
c = a*b
print 'The result of', a, 'x', b, ':', c
return c
To run the application, you add a class name between the module and function names, which is "Multiply" in this case. The command line becomes "call_class py_class Multiply multiply" or "call_class py_class Multiply multiply2 9 9".
Multi-threaded Python embedding
With the above preparations, we are ready for some serious business. The Python module and your C/C++ application have to run simultaneously from time to time. This is not uncommon in simulation communities. For instance, the Python module to be embedded is part of a real-time simulation and you run it in parallel with the rest of your simulation. Meanwhile, it interacts with the rest at run-time. One conventional technique is multi-threading. There are a number of options for multi-threaded embedding. We are going to discuss two of them here.
In one approach you create a separate thread in C and call the Python module from the thread function. This is natural and correct, except that you need to protect the Python Interpreter state. Basically, we lock the Python Interpreter before you use it and release after the use so that Python can track its states for the different calling threads. Python provides the global lock for this purpose. Let us look at some source code first. Here is the complete content of "call_thread.c":
#include <Python.h>
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <errno.h>
#ifdef WIN32 // Windows includes
#include <Windows.h>
#include <process.h>
#define sleep(x) Sleep(1000*x)
HANDLE handle;
#else // POSIX includes
#include <pthread.h>
pthread_t mythread;
#endif
void ThreadProc(void*);
#define NUM_ARGUMENTS 5
typedef struct
{
int argc;
char *argv[NUM_ARGUMENTS];
} CMD_LINE_STRUCT;
int main(int argc, char *argv[])
{
int i;
CMD_LINE_STRUCT cmd;
pthread_t mythread;
cmd.argc = argc;
for( i = 0; i < NUM_ARGUMENTS; i++ )
{
cmd.argv[i] = argv[i];
}
if (argc < 3)
{
fprintf(stderr,
"Usage: call python_filename function_name [args]\n");
return 1;
}
#ifdef WIN32
handle = (HANDLE) _beginthread( ThreadProc,0,&cmd);
#else
pthread_create( &mythread, NULL,
ThreadProc, (void*)&cmd );
#endif
for(i = 0; i < 10; i++)
{
printf("Printed from the main thread.\n");
sleep(1);
}
printf("Main Thread waiting for My Thread to complete...\n");
#ifdef WIN32
WaitForSingleObject(handle,INFINITE);
#else
pthread_join(mythread, NULL);
#endif
printf("Main thread finished gracefully.\n");
return 0;
}
void ThreadProc( void *data )
{
int i;
PyObject *pName, *pModule, *pDict,
*pFunc, *pInstance, *pArgs, *pValue;
PyThreadState *mainThreadState, *myThreadState, *tempState;
PyInterpreterState *mainInterpreterState;
CMD_LINE_STRUCT* arg = (CMD_LINE_STRUCT*)data;
for(i = 0; i < 15; i++)
{
printf("...Printed from my thread.\n");
sleep(1);
}
Py_Initialize();
PyEval_InitThreads();
mainThreadState = PyThreadState_Get();
mainInterpreterState = mainThreadState->interp;
myThreadState = PyThreadState_New(mainInterpreterState);
PyEval_ReleaseLock();
PyEval_AcquireLock();
tempState = PyThreadState_Swap(myThreadState);
pName = PyString_FromString(arg->argv[1]);
pModule = PyImport_Import(pName);
pDict = PyModule_GetDict(pModule);
pFunc = PyDict_GetItemString(pDict, arg->argv[2]);
if (PyCallable_Check(pFunc))
{
pValue = PyObject_CallObject(pFunc, NULL);
}
else {
PyErr_Print();
}
Py_DECREF(pModule);
Py_DECREF(pName);
PyThreadState_Swap(tempState);
PyEval_ReleaseLock();
PyThreadState_Clear(myThreadState);
PyThreadState_Delete(myThreadState);
Py_Finalize();
printf("My thread is finishing...\n");
#ifdef WIN32
_endthread();
#else
pthread_exit(NULL);
#endif
}
The thread function needs a bit of explanation. PyEval_InitThreads() initializes Python's thread support. PyThreadState_Swap(myThreadState) swaps in the state for the current thread, and PyThreadState_Swap(tempState) swaps it out. The Python Interpreter will save what happens between the two calls as the state data related to this thread. As a matter of fact, Python saves the data for each thread that is using the Interpreter so that the thread states are mutually exclusive. But it is your responsibility to create and maintain a state for each C thread. You may wonder why we didn't call the first PyEvel_AcquireLock(). Because PyEval_InitThreads() does so by default. In other cases, we do need to use PyEvel_AcquireLock() and PyEvel_ReleaseLock() in pairs.
Run "call_thread py_thread pythonFunc" and you can get the output as shown below. The file "py_thread.py" defines a function called pythonFunc() in which a similar random testing block prints "print from pythonFunc..." to screen fifteen times.
Printed from the main thread.
...Printed from my thread.
Printed from the main thread.
...Printed from my thread.
Printed from the main thread.
...Printed from my thread.
Printed from the main thread.
...Printed from my thread.
Printed from the main thread.
...Printed from my thread.
...Printed from my thread.
Printed from the main thread.
Printed from the main thread.
...Printed from my thread.
Printed from the main thread.
...Printed from my thread.
Printed from the main thread.
...Printed from my thread.
Printed from the main thread.
...Printed from my thread.
Main Thread waiting for My Thread to complete...
...Printed from my thread.
...Printed from my thread.
...Printed from my thread.
...Printed from my thread.
...Printed from my thread.
My thread is finishing...
Main thread finished gracefully.
Obviously, the implementation is getting complicated because writing multi-threading code in C/C++ is not trivial. Although the code is portable, it contains numerous patches, which require detailed knowledge of the system call interfaces for specific platforms. Fortunately, Python has done most of this for us, which brings up the second solution to our question under discussion, namely, letting Python handle the multi-threading. This time the Python code is enhanced to add a threading model:
''' Demonstrate the use of python threading'''
import time
import threading
class MyThread(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
def run(self):
for i in range(15):
print 'printed from MyThread...'
time.sleep(1)
def createThread():
print 'Create and run MyThread'
background = MyThread()
background.start()
print 'Main thread continues to run in foreground.'
for i in range(10):
print 'printed from Main thread.'
time.sleep(1)
print 'Main thread joins MyThread and waits until it is done...'
background.join()
print 'The program completed gracefully.'
The C code does not handle threading any more. All it needs to do is just call createThread(). Try to use the previous "call_function.c". You can run "call_function py_thread createThread" to see how the output looks like. In this case, the second solution is cleaner and simpler. More importantly, Python threading model is portable. While the C threading code for Unix and Windows is different, it remains the same in Python.
The Python function createThread() is not required if our C code calls the thread class' start() and joint() methods. The relevant changes are listed in the following (from the C source file "call_thread_2.c"):
pInstance = PyObject_CallObject(pClass, NULL);
PyObject_CallMethod(pInstance, "start", NULL);
i = 0;
while(i<10)
{
printf("Printed from C thread...\n");
PyObject_CallMethod(pInstance, "join", "(f)", 0.001);
Sleep(1000);
i++;
}
printf(
"C thread join and wait for Python thread to complete...\n");
PyObject_CallMethod(pInstance, "join", NULL);
printf("Program completed gracefully.\n");
Basically, after you create the class instance, call its start() method to create a new thread and execute its run() method. Note that without frequent short joins to the created thread, the created thread can only get executed at the beginning and the main thread will not release any CPU to it until it's done. You may try this out by commenting the joint call within the while loop. The behavior is somehow different from that of the previous case where we had called start() from within the Python module. This seems to be a feature of multi-threading which is not documented in the Python Library Reference.
Points of interest
I have deliberately paid attention to writing generic, portable C code for Python embedding. By encapsulating low-level system calls, Python supports platform portability and makes writing portable code easier. Most Python modules can be ported between Unix-like environment and Windows with little effort. We should keep this portability in mind when writing C/C++ wrapping code for Python modules. It may not be always straightforward to write portable C code yourself. Python has done a lot of hard work, as in the above case. Try to explore easier, simpler and cleaner solutions. Anyway, I stop here. How to write portable Python code goes beyond the scope of this tutorial. It could well make a good title for a new article.
While embedding is a good option for utilizing Python modules in C/C++ applications, there are other alternative approaches. On Windows, some tools (e.g. "py2exe") can convert Python modules into Windows executables directly. Then, you can spawn a process to run the executable from within the C/C++ application. One drawback is that you cannot call the module directly. Instead, your application has to interact with the module through certain types of IPC. This requires that the Python module under discussion be "IPC ready", meaning that it should have implemented an IPC interface to process incoming and outgoing data. Part II of this article will discuss embedding-related IPC techniques.
All the source code provided in this article is simple C code for demonstration purpose. In practice, I recommend to put the Python embedding code in C++ wrapper classes. This way, high-level application developers don't have to deal with the embedding details.
Conclusion
In this part, I have covered Python embedding from the basics such as calling functions, classes and methods, to not so basic topics like multi-threaded embedding. Python/C API provides a consistent calling interface to ease the task of integration between C/C++ and Python modules.
The discussion of multi-threaded embedding has raised a question for us: how does your C/C++ application communicate with the embedded Python module? The second part of this article shall address this issue from the IPC point of view.
History
- This is the first revision of the article and the source code.
