- Simple Python ctypes Example
- 25.4.2009
I have been working with the massive data tables generated by my balloon flight and have run into a bit of trouble with Python. Everyone knows that Python isn't the fastest kid on the block. However, there are several ways you can speed it up. First off, Python 2.6 and 3.0 provide a new multiprocessing library that makes it much easier to divide a problem amongst concurrent workers. However, this is still not ideal for high throughput data requirements. I have previously worked with Python's C API, but found it's syntax quite undesirable. This new problem gave me some motivation to check out ctypes. With Python's ctypes, I can code in straight C and call the functions from Python. I had a hard time finding end-to-end examples online, so I thought it would be beneficial to share my findings.
To start, you must first create a C source file - let's call it 'libtest.c'. Say that you want to make a function that multiplies to integers, so insert this code into 'libtest.c':int multiply(int num1, int num2) { return num1 * num2; }
Next, we need to compile 'libtest.c' into a valid library. The commands that I used were:gcc -c -fPIC libtest.cgcc -shared libtest.o -o libtest.so
This creates the final version of the library that we can use in python.
From the directory containing 'libtest.so', fire up the python interpreter and try these commands:>>> from ctypes import * >>> import os >>> libtest = cdll.LoadLibrary(os.getcwd() + '/libtest.so') >>> print libtest.multiply(2, 2)
The function should print 4 if everything has gone correctly.
Sometime in the near future, I will expand upon how I applied this to my problem using POSIX pthreads.
Nice one, looking forward to the POSIX threads.
Reply
I took the liberty of converting this example to Windows (for the same reasons you state for posting it in the first place :))
http://www.vagabonds.info/wp/2009/04/simple-python-ctypes-example-in-windows/
Reply
Excellent! For some reason I can't bring myself to develop C on Windows, but I'm sure this is invaluable to many people.
Reply
Have you tried SWIG? http://www.swig.org/
Reply
No, but it looks very interesting. I may use it for the next, high performance pthreading tutorial in the near future. If anyones is interested, OpenMP (http://en.wikipedia.org/wiki/OpenMP) seems to be worth taking a look at as well.
Reply
If you're using OS X, you need to create the library as follows (one gcc step, not two):
gcc -dynamiclib libtest.c -o libtest.dylib
Then, import as above, except with '/libtest.dylib' instead of '/libtest.so'
Reply
Thanks for expanding. I would also like to note that it may be possible to generate the library in a more concise fashion on other UNIX systems as well. Unfortunately, I have to put this project on hold until finals are over in another week or so.
Reply
Can you do a more complicated example , one with a class?
Reply
I'm not quite sure what you mean by an example with a class, or what it would show that hasn't already been done here. If you mean Python class, then you would just put the 'libtest = ... print libtest.multiply(2,2)' somewhere within the class declaration. If you meant C++ class, I honestly have no idea how the process would change - I'm trying to stick to C due to what I have heard about C++.
Also, I will be adding a more complicated example later with some Python classes and more complicated multithreaded C code. However, this won't be for another week or so due to finals at my university.
Reply
very nice. Thx. How would you do it, if your c function works on an array?
Reply
For arrays, try this on for size:
lib.c
/* sum of a list of doubles */
double sum(double *data, int n)
{
int i=0;
double sum=0.0;
for (i=0; i<n; i++)
{
sum += data[i];
}
return sum;
}
/* mean of a list of doubles */
double mean(double *data, int n)
{
double s = sum(data, n);
return s/((double)n);
}
Then we compile it:
$ gcc -shared -fPIC lib.c -o lib.so
In python we use ctypes:
from ctypes import *
so = CDLL("lib.so")
# Set up interfaces
so.mean.argtypes= [POINTER(c_double), c_int]
so.mean.restype = c_double
so.sum.argtypes = [POINTER(c_double), c_int]
so.sum.restype = c_double
# Call the functions.
def cmean(self, dlist):
doubleArray = c_double*len(dlist) # Data Type (double[n])
cx = doubleArray(*dlist) # Actual array
n = len(cx) # Length of the data
result = so.mean(cx, n)
return float(result)
def csum(self, dlist):
doubleArray = c_double*len(dlist)
cx = doubleArray(*dlist)
n = len(cx)
result = so.sum(cx, n)
return float(result)
# We can now use the functions as if they were pure python!
data = [1,2,3,4,5,6]
print cmean(data)
print csum(data)
I used ctypes to create a transparent wrapper around my math library. When you import it, it checks for the .so file. If it exists, the .so functions (cmean / csum) are bound to the library. If not, then pure python modules are bound to it instead. You just "import mylib" and take advantage of the .so file if it exists.
Reply
By the way, for questions (I'm the above poster) email me: frikker@gmail.com
Reply
Thanks for contributing! I've been lazy posting lately (school stuff), so user examples are a big plus!
Reply
No problem. You're exactly correct that complete examples using ctypes are kind of rare. I literally just typed out that example as a reminder to myself so I figured I'd share the love.
The next problem I am tackling is how to access non-standard c types (either through pointers or other) with ctypes. Are you familiar with libusb? Currently I have a driver written in python using pyusb (a thin wrapper around the C interface to libusb). The syntax in python basically identical to C. I am curious if I am able to port my python over to C and then call the functions using python. I'm wondering if I could return a c-type "usb device handle" back into the python scope. It would not be a double or int, obviously, so I'm not sure how I would define the interface.
Nice captcha, by the way :)
Reply
Hi. I got successfully called a c++ routine using ctypes (using a c wrapper and the extern "C" directive). Working on a writeup now.
Reply
typo oops. I successfully called a c++ routine using ctypes is what I (obviously) meant to say :)
Reply
Thanks - although I would really rather not steal your thunder on this stuff. You seem to be much more active about creating short tutorials at the moment that I am, so why not create a few pages of your own?
I'm not at all opposed to you adding more content here, but I think a couple pages on a personal site would be much more user-friendly (and I would gladly link to them). I'm actually a bit embarrassed to have someone actively posting here - as the site is still very much a hobby work-in-progress.
However, if you think that's too much trouble, I completely understand (I obviously haven't had much time to do much web development lately either). Go ahead and add everything here. This type of work is definitely put to better use by sharing with others!
Reply
Believe it or not I know exactly what boat you're in. I made a cms engine about 5-6 years ago (which is still up and running) as a hobby. I'm kind of embarrassed to post on it (http://frikk.tk) because I haven't actually updated anything in almost two years. I'll be tearing it down hopefully in a couple weeks and putting up a simple django engine, but at the moment I have nothing :).
I'm posting here mostly because you're actually responding to my posts. Peace!
Reply
That's pretty impressive for 5-6 years ago. However, you'll love the switch to django. I haven't coded anything here for a while, but the python/django combination makes it easy to pick up regardless of how long it has been.
Reply
perhaps a silly question but are we still hindered by python GIL if using pthreads in our c bindings?
thanks
Reply
If you are doing *all* of the multithreaded stuff in C, then no. I still plan on providing an example/benchmark of this, but school has taken over my life ATM.
Reply
For a more portable way of compiling the C source file (which will invoke gcc with the right parameters on Linux/OSX or msvc on Windows), you can use setuptools or distutils. In your setup.py, you would add something like:
from setuptools import setup, Extension
module1 = Extension('libtest',
sources = ['libtest.c'],
export_symbols = ['multiply'])
[...]
setup(
[...]
ext_modules = [module1],
)
Reply