Lies, Damn Lies and Benchmarks. Software benchmarking is like trying to measure the individual performance of soccer players on a team and then using that information to predict how they will do in the World Cup. When there is a clear gap between the abilities of a novice and the abilities of a professional, the outcome is almost always predictable. Things get interesting, however, when you take professionals at the height of their careers and pit them against each other.
Benchmarking software follows the same philosophy. There are so many variables at play that it is often hard to predict how one set of benchmarks will play out in the real world. By forcing two state-of-the-art systems to their limits, we can learn about strengths and weaknesses in each system. Over the past week, we did a small project to benchmark the latest release of Monarch3 against Apache2 and PHP5 – the results surprised us…