Winning By Cheating
October 3rd, 2013When I first set up a Samsung Galaxy S4, I decided to run a set of benchmarks just to see how well it did. As already reported by the media, it scored more than twice as fast as its predecessor, the Galaxy S3. But it didn’t feel that much faster, if at all. It still bogged down when loads of apps were running, and it was still necessary to use a utility that quit all open apps, or just restart.
So where was the advantage?
Well, I’m not much of a gamer, so I suppose frame rates were noticeably higher, as they should be considering that more powerful graphics hardware had been installed. Otherwise, I didn’t see any real difference. The display was only a tad larger, and Samsung’s penchant for junkware didn’t really deliver a better user experience.
So maybe I wasn’t terribly surprised to discover that Samsung may have “cooked the books” on the Galaxy S4, and the Galaxy Note S3 “phablet.” How? Well it appears they inserted code on these handsets that would automatically switch them into a higher speed mode when certain benchmark apps were run. Consider it overclocking for the sake of raising the scores. Of course, if these handsets were run at such speeds all the time, reliability might suffer, and battery life would be less. It was all in the sake of faking the scores.
Now I wasn’t the one to determine how this scheme was being played. Such respectable sites as AnandTech and Ars Technica confirmed this dirty truth. In one test, the name of a benchmarking app was changed from the default apparently set by Samsung. Guess what? The results were run without overclocking and were thus 20% lower. In the real world, of course, a 20% performance boost really isn’t that noticeable.
So does that sound fair to you?
Well, it appears Samsung is not the only manufacturer of Android gear that’s faking it. In an article from AppleInsider’s Daniel Eran Dilger, “According to testing by Anand Lal Shimpi and Brian Klug of AnandTech, benchmark cheats are not limited to Samsung, but rather pandemic among Android licensees.
The site asserts that, “With the exception of Apple and Motorola, literally every single OEM we’ve worked with ships (or has shipped) at least one device” that to produced false benchmark scores. That means Asus, HTC, LG — well you get the picture, and it’s not pretty.
In the personal computer world, having superior benchmarks is a badge of honor, same as the 0-60 mile per hour ratings on a car. The faster, the better. Sure, in the automotive world, few actually attempt to accelerate as fast as possible, unless there’s no traffic and no waiting police officer to catch you for speeding or driving in a dangerous fashion. But on a PC, it theoretically means that your apps will not only launch faster, but run faster. This means that you should be able to get your work done faster, although your typing speed won’t increase proportionately. But the things your computer does for which you have to wait will be done faster. That’s a good thing.
Certainly, if you’re a gamer, you will cherish achieving higher frame rates along with the ability to run more and more of the high-end animation effects offered by an app. Yes, I suspect there’s a practical limit to how many frames are too much to make a difference, but at least you have figures you can compare.
That is unless the figures are bogus.
Sure, there are lots of ways to check computing hardware, and different characteristics to measure. There are ways to conduct a test that will show off the best a device can produce. But you want to believe that the results are real and not faked.
All right, some of those note-book and mobile gadget battery test results might be a little screwy. To get the maximum amount of time between charges, you may run the gadget with all unneeded services turned off and brightness set to a minimum level. You may run only one app rather than many to consume less system resources. The manufacturer’s claims may be proven, in theory at any rate, but they don’t represent a real world experience.
But the results can be duplicated if you follow the same operating conditions. On the other hand, if the hardware is run in a special mode just to complete benchmarks faster, that’s not fair. It’s just a cheap trick, and you’d think multibillion dollar multinational corporations are above such schemes, right?
Imagine a car that switched on the turbo only when you ran a 0-60 test.
Does Samsung or one of the other blatant offenders really deserve your business because they cheat on the benchmarks of their hardware? Sure, I suppose people who own a Galaxy S4 can claim they have one of the fastest smartphones on the planet, although it does seem that the iPhone 5s generally fares better against Samsung’s fake scores.
However, there’s no accountability in the industry. Most people who believe those high test scores usually don’t read the stories that demonstrate they are fake. The numbers afford bragging rights. False bragging rights, but bragging rights nonetheless.
Print This Post
[…] “Winning By Cheating: When I first set up a Samsung Galaxy S4, I decided to run a set of benchmarks just to see how well it did. As already reported by the media, it scored more than twice as fast as its predecessor, the Galaxy S3. But it didn’t feel that much faster, if at all. It still bogged down when loads of apps were running, and it was still necessary to use a utility that quit all open apps, or just restart.” — “The Tech Night Owl” […]
I have heard of these types of cheating strategies before. Internet carriers boost performance to websites that test your upload/download speed, but throttle anything else. Even though, as you say, most people don’t read the reports of the cheating, I think they should really pay attention. To me it indicates a company’s willingness to deceive, and if it is done for testing, then why not in the technical specifications, materials, and most other claims they make? It is sad that we live in a time where basic honesty is sidestepped simply for bragging rights.
@Len Swierski, Thanks for bringing this up. I caught Cox Communications doing this. You get one figure from their benchmark tool; another from the rest.
CenturyLink (the other broadband provider in these parts) is more honest with their speed ratings.
Peace,
Gene
As with certain individuals in sports, all cheaters should be called out and labelled as such.
So what else is new? Manufacturers of a lot of products have “cheated”, if you want to use that word, by using favorable, non-standardized testing methods to generate impressive benchmark specs. We’re familiar with this tactic in stats ranging from audio amplifier distortion to automobile mileage, and we’ve encountered it in the fiercely competitive smartphone market as well (I seem to recall one mfr. who boasted its smartphone came with a camera with some megapixel capacity which would be jaw-dropping in a Hasselblad). That’s why many consumers are smart enough to rely on test results reported by independent and disinterested third-party reviewers. (This site has a history of knocking Consumer Reports, but it’s worth remembering that a major reason for the long-term success of that publication has to do with public mistrust of advertising claims).
@dana sutton, My problem with Consumer Reports is that, at times, they are just plain wrong.
Peace,
Gene