Azimout's Linux weblog

Python code profiling and optimizations

leave a comment »

I’ve written before about ways to profile your Python code. Here’s also a good guide with tips on Python performance optimization.

Here’s a few more very specific tricks I discovered through trial-and-error:

  1. I was saving Python lists in MySQL by converting them to strings (e.g. ‘[1, 2, 3, 4]’) and putting them in a VARCHAR field. To recover them I would loop through the SQL SELECT results and then run eval() on each of them. Besides being a security loophole, eval () is rather slow, especially if you have to run it a few thousands of times. I found out the safer and much faster json.loads() function did the trick for me.
  2. I was plotting the results with matplotlib and saving to a .png file. To remove the surrounding whitespace, I would call savefig() with the bbox_inches=’tight’ argument. By profiling, I found out this added a whopping 1.5″ each time it was called. I managed to get around this by creating a figure object of the right size from the start, i.e. pyplot.figure(figsize=(9, 4), dpi=200) and adjusting the margins like this: fig.subplots_adjust(left = 0.05, right = 0.95, bottom = 0.1, top = 0.9)
  3. I tried to make use of multiple cores with multi-threading. Despite what the article on the Global Interpreter Lock (GIL) says, my I/O (MySQL select), image processing (matplotlib) and NumPy threads did not seem to run in parallel; my multi-threaded code was actually 5-10% slower (probably due to the context-switching overhead). Multi-processing did the trick for me.

Written by azimout

22/07/2011 at 13:46

Posted in Tricks

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: