Day 16: Measure Performance Before Optimizing
Writing programs takes time, but writing correct and fast programs is even more difficult. That is why there is so much badly written software in the market. Bugs are the result of sloppy and rushed code. Programs are frequently slower than they need to be as a result.
In this situation, it sometimes seems smart to design systems that are efficient. But for many programmers, efficiency becomes an end in itself, and they end up trying to optimize code before understanding the real implications of these optimizations.
One of the most common problems is not understanding the real bottlenecks in the system. For example, when writing software for the web using standard architectures, accessing the data base is frequently the bottleneck of the system. When faced with this situation, it doesn’t matter if the code to handle string comparison is written in assembly or in Ruby, the result will be approximately the same.
Measuring Speed Up
There is an interesting result studied in computer architechture called the Ahmdal Law. This principle says that any optimization (say, getting twice the performance) applied to a part of the system that is responsible for only a small percentage of the execution time will result in a very small improvement.
For example, suppose that you improve the string handling code to be 10 times more efficient. This might give you hopes that the whole system will be a few times more efficient that it is. However, after measuring the system, you verify that the time spent on string comparison is 10% of the total time.
The result is that an operation that takes 10% of the total time now takes only 1% of the time. However, this means that the performance improvement for the whole system is just 9%.
If 9% still seems good for you, notice that I was very generous here. It is really hard to change something to be 10 times faster, unless you come up with a different algorithm. Also, the differences in percentage of usage are not commonly in the range 10 to 90%, but more frequently it is 1 to 99%. That is, most areas of your code contribute 1% or less to the running time, while bottlenecks such as network and database delays are responsible for 99% of the time.
Even for processor bound tasks this situation remains. There is probably a few small parts of your code that are responsible for 99% of the time spent in the program.
That is why measuring a system is so important. If you don’t know what is responsible for that figure of 99%, your efforts to improve the performance of the system are basicaly wasted. You should aim first at understanding what contributes to the bottlenecks in the system. Then, you can start planning on how to improve that area.
Further Reading
Read Computer Architecture: A Quantitative Approach to find everything you may want to know about Amdahl Law and how it applies to computer systems.
Similar Posts:
About the Author
Carlos Oliveira holds a PhD in Systems Engineering and Optimization from University of Florida. He works as a software engineer, with more than 10 years of experience in developing high performance, commercial and scientific applications in C++, Java, and Objective-C. His most Recent Book is Practical C++ Financial Programming.
Premature optimisation is the root of all evil!
Once I have uttered the usual optimisation quote, I may say that measuring is the first thing you need to do. In my fractal generator programs (I’m a mathematician and have done some work in complex dynamics, they were not for fun nor art), I didn’t optimise anything when I started out. Then, some day, more functionality was needed and then was the moment to optimise. Even simplifying code and cleaning the implementation, the last part of the research needed 60 hours in an 8 core system running in parallel. Prior to that, the same code would have taken around 250 hours (estimated). If I had started writing optimised code (which was not feasible as we were still not sure about how to implement it) it would have taken me a lot more than 190 hours to do it.
You always need to measure performance, and then think about the trade-off of your time vs your run time vs your programming time.
Cheers,
Ruben
By Ruben Berenguel on Feb 19, 2011
nice article. make real sense to development
By zhoubig on May 15, 2011