Why Do You Believe Consumer Reports?
April 6th, 2011Just this week, Consumer Reports posted an online review covering the iPad 2, and a small number of also-rans in the tablet PC world. This report was meant as an upgrade to an article that appeared in the May 2011 newsstand issue, which was prepared before the iPad 2 was available to evaluate.
The conclusion was predictable. The iPad 2 gets an “excellent” rating in nearly every category. Curiously, the original iPad was placed in a tie with the Motorola Xoom, though heaven knows why. Well, at least CR didn’t find any bogus antenna issue with which to downgrade Apple’s iconic gadget, but they aren’t entitled to much more credit than that, simply because they still haven’t a clue how to properly review products of this sort.
They have, for example, an ephemeral “ease of use” category, but they don’t really explain the criteria for such ratings. If they were really paying attention, it would seem highly questionable for the Xoom to rate very highly because, if you consider what you can do with this gadget, it barely rates a xero — make that a zero.
You see, you can rate CPU and graphics performance, and whether the interface is snappy or not. You can certainly judge the features, such as onboard cameras, or, in the case of the original iPad, the lack thereof. You might also look at accessory ports, such as an SD card slot, or an HDMI port, neither of which Apple incorporates, even though they offer convenient and relatively low-cost adapters for both.
But none of this matters if there are no apps to run. Apple has over 65,000 specifically designed to run on an iPad. In contrast, the Motorola Xoom, which uses Android 3.0, or Honeycomb, has a few dozen.
As a result, the argument is over. Your usability options on a Xoom, or any other Android-based tablet, are extremely limited, unless you want to stick with regular smartphone apps, which look horrible when blown up to the full size of the display.
Other Android tablets, using older versions of the OS, are in worse shape. Those products are dead ends, because they weren’t designed with tablets in mind, and new apps will never be compatible. That’s their fault for not paying attention to Google. This, along with the growing and irritating fragmentation issue, are the reasons that Google is asserting greater control over the ways licensees can fiddle with the interface and bundled software. Indeed, they are also holding off on releasing the source code, with no definite date as to when it might be available.
Unfortunately, CR doesn’t grok any of this. They seem utterly unable to judge these products on the basis of real usability, which means having enough apps to actually afford a choice of what to do beyond email, Internet access, and access to your favorite social network. Indeed, one of the reasons that the Mac was denigrated early on was the myth that very few applications were available for that platform. In that case, it wasn’t true for the most part, except for certain vertical applications for business consumers that were only available on Windows.
When it comes to an Android 3.0 tablet, the situation is dreadful, utterly dreadful. Worse, app developers aren’t going to have much of an incentive to build apps for the Xoom and its ilk, simply because the iPad has swamped the marketplace. Developers who want to make money know which platform is best.
Understand that CR’s editors are entitled to their opinions, and they even entitled to publish highly flawed reviews that fail to address the needs of the end user. Rather, I’m concerned why the media, tech, mainstream and elsewhere, largely accepts CR’s word as gospel. The vast, vast majority of articles about their reviews are not critical. They never consider whether or not the magazine is using poorly conceived testing methods.
I can see where manufacturers might avoid public criticism, simply because it will sound like sour grapes. Indeed, when it comes to auto testing, CR has it right when handling tests revealed certain vehicles nearly tipping over or swerving out of control in extreme handling tests. While CR was once sued by an auto maker over this issue, more recently Toyota promptly responded to a reported handling flaw by rejiggering the onboard software so the electronic stability control would kick in faster. But you wonder why they never noticed during the product design phase.
When it comes to consumer electronics gear, Apple didn’t suffer in any noticeable way because the two flavors of the iPhone 4 weren’t recommended, based on that alleged “Death Grip.” Certainly, if there was an ounce of evidence that sales might be hurt, I’m quite sure Apple would have gotten ahold of CR in order to set them straight. Certainly what CR reports flies in the face of Apple’s tests, once posted online, and still mirrored all over the globe, not to mention blogs populated with antenna experts, and all those people who posted YouTube videos demonstrating that antenna sensitivity issues can be duplicated on many smartphones. I’m most troubled, however, that CR doesn’t seem to know about warning labels, and those cautions in user manuals about the unsavory impact of holding a mobile handset the wrong way.
Or maybe their test team doesn’t believe in reading manuals. While I realize consumers don’t either, a magazine that prides itself on thorough and balanced product testing ought to know better.
Print This Post
I don’t, not any more. And not just because of the iPhone. My experience has more than once been the flip of what they have reported—i.e. their top rated was my worst product, and their bottom or lower rated products were superior. I first discovered their suspect ratings back in the late 80s with automobiles. Both their research and the reader ratings are highly subjective. That should be no surprise to anyone who understands the highly subjective nature of humans. At least their “ease of use” criteria is starting to show some sense of understanding this, flawed as it may be. Not all things that are worth knowing or otherwise important are quantifiable.
Joe
I’ve NEVER had any faith in Consumer reports for anything more complicated than a toaster. They don’t have the knowledge and expertise; they can’t, without vastly increasing their staff or using outside reviewers. Someone who recognizes good creamed corn is not a tablet computer expert.
Case in point: I used to be a photographic retailer. I lived in fear of CR’s review of cameras; they seemed to have no relation to reality at all, but people would come storming in the door asking to buy the “top pick”. Once they rated a rather tatty consumer camera above a professional Nikon for reliability, when I knew quite well the reality was the exact opposite. Their “small” mistake was rating them based on the number of repairs per user, not per roll of film used. Completely ignoring that one group of users shot dozens of rolls per year, the other shot THOUSANDS.
It’s sort of like downrating a Macintosh because the Start Menu is almost impossible to find…
“Consumer Reports” should just merge with “The Onion”.
I have to say, giving CR a pass on their vehicle ratings doesn’t make sense to me. Why should I believe that they know how to test cars any better than they do computers and smartphones?
The Suzuki Samurai story is pretty ugly. It very much seems like CR decided to use Suzuki as a marketing tool to boost their visibility in the mainstream press and thereby increase their subscriptions. When you read that CR had to expend a lot of effort to get the Samurai to tip over and even change their testing methodology for the Samurai, it starts to sound a lot like their handling of the iPhone 4 testing.
Apple is a very high profile company. What better way to get press coverage than to trash a beloved Apple product upgrade?
This article and the accompanying comments are borne of pure technological superciliousness.
So Consumer Reports recommends a washer. You buy one but it fails. Only a silly person would immediately conclude that Consumer Reports’ recommendation was bogus. Honda makes the most reliable cars in the world, and Consumer Reports records this in their annual car issue, but if your Accord breaks down that just means that you were unlucky.
Consumer Reports test things like normal humans would, not how Apple fans would.
On a Mac you cannot resize a Finder window from anywhere but a tiny triangle at the bottom right.
Consumer Reports gets a lashing for pointing this out?!
This is yet more of the ‘us vs. them’ deal:
Micros**t is the enemy.
Google is the enemy.
Palm is the enemy.
Motorola is the enemy.
Michael Dell is the enemy.
HP is the enemy.
ANY company whose products compete or even overlap slightly with Apple’s is the enemy.
Anyone executive who is not Steve, Phil, Bertrand, Tim or Scott is the enemy.
Anybody that dares disagree with the aforementioned is the enemy.
Any analyst that does not put an ‘outperform’ rating on APPL is the enemy.
And now, Consumer Reports is the enemy.
@Neil, The issue is not, and never has been, whether Apple is the “enemy.” If Apple makes a bad product, they deserve the criticisms they receive. However, CR’s review process gives insufficient emphasis to usability factors. They rate the poor-selling Xoom as identical to iPad 1, despite its known OS bugs, and the total lack of a decent amount of tablet-based software to run. According to published reports, the public knows, even if CR doesn’t. Some 100,000 have been sold so far, and the iPad killer brigade is going back to the drawing boards.
Certainly, a favorable review is no guarantee of success either, but CR’s methodology overlooked the significance of the Xoom’s shortcomings, hence it was rated as better than it really is. If the Xoom had an Apple label on it, it would still deserve a ding.
Peace,
Gene
@Neil, “Only a silly person would immediately conclude that Consumer Reports’ recommendation was bogus.”
No sillier than the person who takes CR’s recommendations as gospel.
All their “data” is presented as objective and unbiased when in reality there is no such thing (much less, their data should be accepted as somehow complete). Even something as seemingly innocuous as reader reports on auto reliability. Hey, they are just reporting what happened! Or are they? That Honda, as you say, has the “best reliability record”. So. really, since I was smart enough to buy a Honda, my problem was actually an anomaly. So it really isn’t worth reporting. On top of that, buying a Honda is showing the world how smart and pragmatic I am to purchase one. To broadcast any problems I really have with the car only demonstrates how gullible I am. So best to keep quiet about all that. And besides, everyone KNOWS American cars are more poorly made than Hondas. So we are just reporting (or NOT reporting) the “facts”.
And never mind that cell phone manufacturers have been publishing for years how not to hold their cell phone, obviously only Apple is guilty of such an abomination since they are the only ones receiving all the attention. iPhone’s are the only ones we should test for such conditions (which aren’t really true anywhere in the world except CR’s lab. Not to mention all the additional clicks their site gets when they publish their blogs.)
This is less about “Us v. Them” and more about “Them” not being all they market themselves to be.
Who fights the power that fights the power? Who watches the watcher?
Joe
Excellent observations. CR does not publish “operational definitions” for most of their selection categories. Nor do they share their category weightings. Sometimes, terms are so unclear, that they defy comprehension. Their shock and awe campaigns such as iPhone antenna gate and others over the years seem to be thinly veiled attempts to assert and re-affirm their authority in the minds of the public. Lastly, CR has a tendency to recommend consumer products that are disposable over those that are more durable. Its not just favoring flashy features. There seems to be a strong bias against products that are more expensive than the norm. (Apple products do seem to be the exception)
There are tremendous opportunities here for people who are ethical to jump in and do credible product ratings.