Making python fast

Everything todo with programming goes HERE.
Post Reply
User avatar
Lucifer
Project Developer
Posts: 8683
Joined: Sun Aug 15, 2004 3:32 pm
Location: Republic of Texas
Contact:

Making python fast

Post by Lucifer »

http://www-128.ibm.com/developerworks/l ... ?ca=dnt-64

Google pdf->html converter


Looks like an order of magnitude speedup can be reached easily with pyrex, whereas psyco looks to me mostly useful as a way to just throw another library module at it. When you compare pyrex's typical order of magnitude increase to the benchmarks I posted awhile back showing python to be 10-20 times slower than C, it looks like we can have our cake and eat it too. :)

In case anyone was curious. :)

Anyone know a good text-to-speech library with python bindings?
Image

Be the devil's own, Lucifer's my name.
- Iron Maiden
User avatar
Lucifer
Project Developer
Posts: 8683
Joined: Sun Aug 15, 2004 3:32 pm
Location: Republic of Texas
Contact:

Post by Lucifer »

This is a very interesting usage of pyrex. Build a standalone executable that embeds python.
Image

Be the devil's own, Lucifer's my name.
- Iron Maiden
User avatar
Z-Man
God & Project Admin
Posts: 11624
Joined: Sun Jan 23, 2005 6:01 pm
Location: Cologne
Contact:

Post by Z-Man »

Indeed, impressive performance. I did my own tests.
Raw python

Code: Select all

#!/usr/bin/python
num = 100000
min = 2.8
max = 4.0
x = 0.1;
for i in range(0,num):
    r = min + i * ( max - min )/num;
    for j in range(1,100):
        x = r * x * (1 - x)
gives

Code: Select all

time echo "import pychaos" | python

real    0m5.764s
user    0m5.625s
sys     0m0.045s
Full pyrex code:

Code: Select all

#!/usr/bin/python
cdef int num, i, j
cdef float min, max, r, x
num = 100000
min = 2.8
max = 4.0
x = 0.1;
for i from 0<= i < num:
    r = min + i * ( max - min )/num;
    for j from 1 <= j < 100:
        x = r * x * (1 - x)
gives

Code: Select all

time echo "import chaos" | python

real    0m0.120s
user    0m0.101s
sys     0m0.007s
Which is more than 50 times the performance, erring in favor of vanilla python because I ignore the constant startup time of the interpreter! Without replacing the range loops with integer loops it's

Code: Select all

time echo "import chaos" | python

real    0m0.495s
user    0m0.474s
sys     0m0.014s
Now, there's the TINY problem that pyrex has been designed to interface with C only, not C++... I'll read the docs and see whether this can be overcome with a minimal amout of C++ glue.
User avatar
Z-Man
God & Project Admin
Posts: 11624
Joined: Sun Jan 23, 2005 6:01 pm
Location: Cologne
Contact:

Post by Z-Man »

Of course, the test was a non-test because it's likely that my simple code can be transfered 1-to-1 to C by pyrex.
User avatar
Z-Man
God & Project Admin
Posts: 11624
Joined: Sun Jan 23, 2005 6:01 pm
Location: Cologne
Contact:

Post by Z-Man »

Hmm, I somehow was unable to come up with a good way to call Python code from the C side, pyrex seems to focus on the other direction.
User avatar
Lucifer
Project Developer
Posts: 8683
Joined: Sun Aug 15, 2004 3:32 pm
Location: Republic of Texas
Contact:

Post by Lucifer »

Pyrex is for building python extension modules. :) So to use C++ code you're probably going to need a C interface to it, and the assumption is that you're going to be mixing pyrex and regular python code, not that you're going to write standalone programs.
Image

Be the devil's own, Lucifer's my name.
- Iron Maiden
User avatar
Z-Man
God & Project Admin
Posts: 11624
Joined: Sun Jan 23, 2005 6:01 pm
Location: Cologne
Contact:

Post by Z-Man »

Uh, I didn't notice the forum this is on. I thought you suggested to use it for Arma :) which would be a bit difficult. So yeah, when I notice a python module I write and use runs too slow, I'll use pyrex.
User avatar
Lucifer
Project Developer
Posts: 8683
Joined: Sun Aug 15, 2004 3:32 pm
Location: Republic of Texas
Contact:

Post by Lucifer »

Heh. I wasn't sure actually, because I wanted to post the links because I thought it was of general interest to anybody using/considering using python but was concerned about performance, but it does have relevance to the roadmap. :)

Since you brought it up, though (heh heh), I was thinking that pyrex would help to address a lot of Luke's performance concerns and possibly some of his issues with python dependencies. I mean, we can go to write more core stuff with pyrex instead of python (like the default AIs), you know, stuff that on a server that uses additional actual python would need available but a user that just wants to play the game and doesn't want a python dependency. Then we can use pyrex to generate the C files and distribute those, so no python/pyrex needed to compile the game. :)

Also, we'd still want to use the python C api to initialize the library and so forth, we'd just have pyrex available to build extensions that'll be called from within Python and run (hopefully) at C speeds, with proper optimization.

Anyway, I'm trying to hurry through my current projects so I can play with pyrex some this weekend. It provides a potential solution to my perpetual problem of playing audio in python. I could use it to wrap SDL (pygame is a high-level wrapper, I want a low-level wrapper), or I could use it to wrap PortAudio (a C API). You know, I could just do cool stuff with it. :) Or I might use it to wrap Audiere. I found a link on the python website that tells you how to use GCC in either Cygwin or MingW to build extension modules that'll run in the officially distributed interpreter, which alleviates the need to get VS7 to build extension modules for python 2.4.
Image

Be the devil's own, Lucifer's my name.
- Iron Maiden
User avatar
Lucifer
Project Developer
Posts: 8683
Joined: Sun Aug 15, 2004 3:32 pm
Location: Republic of Texas
Contact:

Post by Lucifer »

Ok, so I've changed out pyAlarm to use pyrex to build the top-level script. It still depends on python, and I moved it to my own cvs server, and since I haven't made a new release I don't have any code you can download, but I thought some of you might be interested in the top-level program's source code. It uses pyrex to build to a c file and header, and then gcc builds it. I can post the Makefile too if you're interested, or I can probably throw together a quick distribution. Anyway, here's the pyrex file. It embeds python into it, but since pyAlarm uses a lot of Python standard libraries, it still requires a complete python 2.4 installation (I also started using some python 2.4 only features).
Attachments
pyAlarm.zip
Pyrex source file.
(1.84 KiB) Downloaded 294 times
Image

Be the devil's own, Lucifer's my name.
- Iron Maiden
Post Reply