Some hours ago, during a discussion on Twitter, it came to light that a lot of people don’t understand the importance of a complete and clear review methodology.
Even geeks, engineers, journalists seem to be unable to see how bad an incomplete testing method could be. We know that in scientific fields a rigorous methodology is needed. According to scientific studies, a test shall be capable of being replicated: “Reliability is the degree of consistency of a measure. A test will be reliable when it gives the same repeated result under the same conditions” (Source).
To do this, we have to know exactly how the experiment was done. About a hardware review, we have to know every single hardware and software component was used, and which version of it. Thanks to these informations, if we want, we could be able to replicate the review tests and we could be able to confirm the achieved results.
So, during that Twitter discussion, someone questioned my results of this review, where I tested the power consumption of the Intel Core i9-9900K @ stock setting. As you can see, I meticulously described the configuration and the methodology I used. Also, I used default settings for everything: “out-of-the-box settings”. I did that in order to see how the PC of "average Joe" perform. We all know that average Joe purchases the pc, puts the plug and pushes the power button. Stop.
People complained that I had to change BIOS options in order to use a lower Power Limit 2, according to Intel specifications. The problem is that Intel itself allowed Asus, Gigabyte, MSI and other manufacturers to use that setting on high end main boards: PL2 @ Auto (About 210W). According to Intel, it’s not overclock: “The values on the data sheet for 1.52 V and 138 A are the maximum values that can be used on the Intel® processor” (Source). So, main boards manufacturers didn’t violate the specification. They just use the max PL2 value allowed.
But this is not the problem.
The problem concerns the methodology of the tests. These people, if they think my results are wrong, could remake the same tests with the same hardware and the same software. Despite that, they attacked B&C saying that other sites are more reliable. It could be. We absolutely don’t think B&C is the best site. However, I think B&C is the site with the best and more clear methodology tables. Here there are some examples.
Some Intel Core i9-9900 reviews (I can't list every 9900K review made)
| Site | Anandtech | Bits'n'Chips | Guru3D | TechPowerup | TechSpot | TomsHardware |
| Mainboard | Yes | Yes | No | Yes | Yes | Yes |
| BIOS | Yes | Yes | Yes | Yes | No | No |
| CPU Cooler | Yes | Yes | No | No | No | Yes |
| Thermal compound | No | No | No | No | No | No |
| Memory | Yes | Yes | No | Yes | Yes | Yes |
| Memory Timings | No | Yes | Yes | Yes | No | No |
| SSD/HDD | Yes | Yes | No | Yes | No | Yes |
| GPU | Yes | Yes | Yes | Yes | Yes | Yes |
| PSU | Yes | Yes | Yes | Yes | No | Yes |
| Operating System | Yes | Yes | Yes | Yes | No | Yes |
| Version OS | Yes | Yes | No | Yes | No | No |
| Build OS | No | Yes | No | No | No | No |
| Driver Chipset | No | Yes | No | No | No | No |
| Driver GPU | No | Yes | No | Yes | No | No |
| Software | Yes | Yes | Yes | Yes | Yes | Yes |
| Software version | Yes | Yes | No | No | No | Yes |
| OS Clean Installation | No | Yes | Np | No | No | No |
As you can see, B&C tests can be replicated without problems. We don’t fear about that. On the other hand, we could have a problem if we want to replicate the tests of other sites.
In conclusion, we think that review sites have to improve their methodology, in order to create more reliable reviews. Currently, the reviews are far far away from being considered “scientific”.
Please note: It's not an attack to the other sites. B&C is a small site, attended in our spare time, so even we want to read some "scientific" reviews. We are end users in first place, not journalists. ;)