Alex Rivera | Logout

How to do "performance-based" (benchmark) unit testing in Python

Asked 2009-03-22T19:36:55.467
16

Let's say that I've got my code base to as high a degree of unit test coverage as makes sense. (Beyond a certain point, increasing coverage doesn't have a good ROI.)

Next I want to test performance. To benchmark code to make sure that new commits aren't slowing things down needlessly. I was very intrigued by Safari's zero tolerance policy for slowdowns from commits. I'm not sure that level of commitment to speed has a good ROI for most projects, but I'd at least like to be alerted that a speed regression has happened, and be able to make a judgment call about it.

Environment is Python on Linux, and a suggestion that was also workable for BASH scripts would make me very happy. (But Python is the main focus.)

python linux unit-testing benchmarking
Edit
Report

1 Answer

7

You will want to do performance testing at a system level if possible - test your application as a whole, in context, with data and behaviour as close to production use as possible.

This is not easy, and it will be even harder to automate it and get consistent results.

Moreover, you can't use a VM for performance testing (unless your production environment runs in VMs, and even then, you'd need to run the VM on a host with nothing else on).

When you say doing performance unit-testing, that may be valuable, but only if it is being used to diagnose a problem which really exists at a system level (not just in the developer's head).

Also, performance of units in unit testing sometimes fails to reflect their performance in-context, so it may not be useful at all.

answered 2009-03-22T19:40:24.757

Your Answer