Alex Rivera | Logout

Transmission bytearray from Python to C and return it

Asked 2013-03-17T10:22:43.187
8

I need fast processing of XOR bytearray, In a variant on Python

for i in range(len(str1)): str1[i]=str1[i] ^ 55

works very slow
I wrote this module in C. I know C language very badly, before I on it wrote nothing.
In a variant

PyArg_ParseTuple (args, "s", &str))

everything works as expected, but I need to use instead of s s* because elements can contain embeded null, but if I change s to s* when calling python crash

PyArg_ParseTuple (args, "s*", &str)) // crash

Maybe some beginner like me want to use my example as a start to write something of his own, so bring all the information to be used in this example on Windows.
Parsing arguments and building values on page http://docs.python.org/dev/c-api/arg.html

test_xor.c

#include <Python.h>

static PyObject* fast_xor(PyObject* self, PyObject* args)
{
    const char* str ;
    int i;

    if (!PyArg_ParseTuple(args, "s", &str))
        return NULL;

    for(i=0;i<sizeof(str);i++) {str[i]^=55;};
    return Py_BuildValue("s", str);

}

static PyMethodDef fastxorMethods[] =
{
     {"fast_xor", fast_xor, METH_VARARGS, "fast_xor desc"},
     {NULL, NULL, 0, NULL}
};

PyMODINIT_FUNC

initfastxor(void)
{
     (void) Py_InitModule("fastxor", fastxorMethods);
}

test_xor.py

import fastxor
a=fastxor.fast_xor("World") # it works with s instead s*
print a
a=fastxor.fast_xor("Wo\0rld") # It does not work with s instead s*

compile.bat

rem use http://bellard.org/tcc/
tiny_impdef.exe C:\Python26\python26.dll
tcc -shared test_xor.c python26.def -IC:\Python26\include -LC:\Python26\libs -ofastxor.pyd
test_xor.py 
Edit
Report

1 Answer

5

This can be done very quickly using numpy. You're unlikely to get substantially faster hand-rolling your own xor routine in C:

In [1]: import numpy

In [2]: data = numpy.uint8(numpy.random.randint(0, 256, 10000))

In [3]: timeit xor_data = numpy.bitwise_xor(data, 55)
100000 loops, best of 3: 17.4 us per loop

If you're using a big dataset (say 100 million points), it's favourably comparable to the times you have quoted for your code:

In [12]: data = numpy.uint8(numpy.random.randint(0, 256, 100000000))

In [13]: timeit xor_data = numpy.bitwise_xor(data, 55)
1 loops, best of 3: 198 ms per loop
answered 2013-03-17T11:12:37.897

Your Answer