View and vote on the article here: Benchmark Faking
Benchmark Faking| Category | | | Summary | | One of the oldest, and what is believed to be one of the most reliable ways to judging a piece of hardware's effectiveness, is benchmarking. While companies seem to crank out products that will crunch through greater amounts of polygons in a shorter amoun |
| | Body | In May of 2003, serious doubts were raised as to nVidia actually still putting out a competitive product. Well-known tech news organizations, such as ExtremeTech.com and Beyond3D, reported that nVidia benchmarks produced by 3DMark03 may be artificially inflated compared to cards using drivers other than the nVidia Detonator FX 44.03 and 43.51 drivers.
These claims were then investigated by Futuremark Corporation, makers of 3DMark03. According to their official investigation, when cards using the listed nVidia drivers were used on a developer's version, which would allow the actual testing to be looked at more closely, the test appeared to be odd, as beta testers discovered. They would reduce the amount of work that the card needed to do, hence making the benchmark scores appear higher than what they should be when compared to other cards.
Once they circumvented the nVidia benchmark cheats, they found a drop in scores as high as 24.1% on a GeForce FX5900 Ultra using the Detonator FX 44.03 drivers. When they tested the same version on cards produced by ATi, one of nVidia's leading competitors, they experienced only a 1.9% drop. (The tested card was a Radeon 9800 Pro, using Catalyst 3.4 drivers. All tests were done using DirectX 9.) It is believed that the Catalyst drivers did alter the results, but not to the severe degree that nVidia drivers did. ATi did come forward and say certain parts of their drivers were "inflating" the scores, but they said that the components that caused a rise in their scores were practical parts used to optimize for their architecture, but said they would remove them in the next driver version.
How are these benchmarks faked? Well, first the driver recognizes the benchmark program, in this case 3DMark03, as the loading screen comes up. This is easily circumvented by slightly changing that screen, however. Once it realizes the benchmark program is running, the drivers detect a vertex shader used in one of the tests. It will ignore the requests to carry through with the "back buffer clear" command. In another test, it detects another shader and cuts off parts of the sky that can't be seen, hence reducing the workload. Similar methods were used, and when they all add up, they cause a significant reduction in workload, causing higher benchmark scores.
Sources:
FutureMark Report - http://www.futuremark.com/companyinfo/3dmark03_audit_report.pdf
Author: Clickman
Edited By: Skerzhner |
|
This article was imported from zZine. (original author: Prothis)
There are no replies to this post yet.
|