Some time ago, routinely jumping through blogs of people I consider interesting on github, I found one great blog focused on Rust and its surroundings. Frankly, I don’t care about Rust that much right now. Mostly because I don’t like the ideas behind the language and one, which is even more important I haven’t even tried to write something meaningful using Rust. Neverthelse, it’s always beneficial to read different opinions on the topics that you both know well and superficially.
Surely this is a programmer blog, and it discusses subjects beyond only Rust ecosystem. One of the subjects that caught my eye was Async Benchmarks Index. It’s a type of live article popular in recent past where author lists a set of benchmarks and validates them against some claim. I don’t undertake to judge the claim the author made in the article. What I’m interested instead, to look more closely at one specific benchmark Async Python is not faster and rebuttal provided for it Ignore All Web Performance Benchmarks, Including This One. I agree with the rebuttal - original async python is not faster benchmark is too subjective. And I have no doubts that provided rebuttal does a good job of debunking it.
What got me thinking the most, though, more fundamental points in the rebuttal post that benchmarks are rigged and shouldn’t be trusted. And while I agree with those statements as they stand true most of the time, I dissagree with the statement that all web performance benchmarks should be ignored. I think, web performance benchmarks have value. They just need to be conducted by you, that is. Yes, I believe, if you run benchmarks tailored to your needs i.e. your specific: data requirements and constraints, hardware, etc. you’ll get the value out of them. Unfortunately, in practice, almost nobody conducts their own benchmarks. The majority is choosing the technology either from their past expirience or rigged benchmarks from the internet or even worse blindly. Contrary to how real decisions should be made - based on data-driven principle. You always want to evaluate something before starting to put stakes in it. Benchmarking is just a tool that gives you that abillity to meassure performance characteristic of a given system, in our example web frameworks. So in conclusion, I would suggest to ignore most of publicly available benchmarks and trust only the data verified by yourself or trusted thirdparty.