Optimizing a Python program with profiling data

If you have been maintaining a Python program that has been growing for some time with new features you may notice at some point it can get a bit lagging and slow. For example, the app that is powered by hearham.com repeaters listing has grown from thousands of repeaters to now over 10,000 repeaters in the listing shown in the app. This hung things up especially on devices like Raspberry Pi or Librem phone while starting up the app.

Starting Python program with a profiler

To get a profile timing of functionality within the app, run something like

python3 -m cProfile -o profile.pstats ./your-program.py

After running all functionality you want to test that may be slow (including startup), close out the program and you should have a profile.pstats file in the working directory.

Reading and interpreting Pstats view

You can read pstats file with various tools you can install to view and interpret .pstats profile captures, but pstats-viewer online is an easy way to analyze it without installing anything on your computer.

Once loaded into the program (or uploaded on the Pstats-viewer online) you can look at the timings of various function calls. Ones that are recursively or repeatedly called are likely culprits for slowness in parts of the code. For example:

Unoptimized profile of repeaterstart.py

Here you can see that the addRepeaterIcon, called for each repeater shown on the map, was called and checked comparing min and max frequency to show – if you have UHF or VHF set in the settings for your UHF or VHF radio. Adjusting the code to not call this getMinFilter, getMaxFilter function each time, makes the code faster:

Optimized profile of repeaterstart.py

with the code adjusted as shown here, with the min and max number used as a parameter each time, rather than checking the config each time, the configparser library was called twice instead of thousands of times. Anything that reads or writes disk space (including reading the config file) is going to take some time especially on slower drives or hard disks! In the second profile above, you can see the configparser code does not take a significant portion of the running time now.

End result – new version

The latest RepeaterSTART 0.8 has the above fix for faster startup, and right click options for web link for the lower repeaters window. Check it out and please leave any bugs or feedback in the issues page.

Leave a Reply

Your email address will not be published.

twenty nine + = 30