Benchmark Faking |
Article is yet to be rated |
|
| Author:
| Prothis
|
|
| Submitted: |
21-Nov-2003 00:46:40 |
| Imported From: |
zZine (original author: Prothis)
|
| One of the oldest, and what is believed to be one of the most reliable ways to judging a piece of hardware's effectiveness, is benchmarking.
|
|
In May of 2003, serious doubts were raised as to nVidia actually still putting out a competitive product. Well-known tech news organizations, such as ExtremeTech.com and Beyond3D, reported that nVidia benchmarks produced by 3DMark03 may be artificially inflated compared to cards using drivers other than the nVidia Detonator FX 44.03 and 43.51 drivers.
These claims were then investigated by Futuremark Corporation, makers of 3DMark03. According to their official investigation, when cards using the listed nVidia drivers were used on a developer's version, which would allow the actual testing to be looked at more closely, the test appeared to be odd, as beta testers discovered. They would reduce the amount of work that the card needed to do, hence making the benchmark scores appear higher than what they should be when compared to other cards.
Once they circumvented the nVidia benchmark cheats, they found a drop in scores as high as 24.1% on a GeForce FX5900 Ultra using the Detonator FX 44.03 drivers. When they tested the same version on cards produced by ATi, one of nVidia's leading competitors, they experienced only a 1.9% drop. (The tested card was a Radeon 9800 Pro, using Catalyst 3.4 drivers. All tests were done using DirectX 9.) It is believed that the Catalyst drivers did alter the results, but not to the severe degree that nVidia drivers did. ATi did come forward and say certain parts of their drivers were "inflating" the scores, but they said that the components that caused a rise in their scores were practical parts used to optimize for their architecture, but said they would remove them in the next driver version.
How are these benchmarks faked? Well, first the driver recognizes the benchmark program, in this case 3DMark03, as the loading screen comes up. This is easily circumvented by slightly changing that screen, however. Once it realizes the benchmark program is running, the drivers detect a vertex shader used in one of the tests. It will ignore the requests to carry through with the "back buffer clear" command. In another test, it detects another shader and cuts off parts of the sky that can't be seen, hence reducing the workload. Similar methods were used, and when they all add up, they cause a significant reduction in workload, causing higher benchmark scores.
Sources:
FutureMark Report
Author: Clickman
This article was originally published by CyberArmy.net in the CyberArmy Library.
|
|
You must be logged in to vote on an article
|
About Us | Privacy Policy | Mission Statement | Help
|