Consumer Reports Can’t Be All Things to All People
July 23rd, 2010If you accept the conventional wisdom, Consumer Reports is the best product review publication in the U.S. The reason is that it’s run by a non-profit corporation that not only doesn’t accept advertising, but actually purchases the items it tests, most often from regular dealers. In other words, it’s incorruptible.
So how can you beat that?
Unfortunately, there’s also that old saying about being a jack of all trades and the master of none that applies very much to Consumer Reports.
Yes, they seem to do a credible job at auto reviews, and on such ordinary consumer products as dishwasher detergent, washing machines, and even digital cameras and flat screen TVs. But the situation becomes troublesome when they confront such sophisticated gear as personal computers and smartphones.
Even though they seem to have a more tech-savvy staff nowadays, their reviews remain useless when it comes to the promised intent, which is to guide you towards making the right purchasing decision.
Take the eternal Mac versus PC dilemma. You never see detailed descriptions of the differences between the Mac OS and Windows in the pages of CR. They may pay lip service to major upgrades, such as Snow Leopard and Windows 7. But when confronted with a choice between the two, which should you pick?
CR won’t tell you why one might be better than the other, or even, in their collective opinion, that there’s no practical difference. Of course, I’d dispute that heavily, as would most people who have had a fair amount of experience with the Mac and the PC, but the question isn’t even asked in their annual reader surveys.
You do see Apple getting high marks, because Macs contain premium hardware. But you might come away with the impression that the Mac is nothing more than a high-priced PC, so why even give it a second thought if you’re not prepared to pay extra for one?
To CR, it’s the same as choosing whether to buy a Toyota or a Lexus. In this case, it’s the same company, sometimes similar platforms, but it’s mostly about style, right?
CR does, however, have a knack for getting solid press. You’ll find their spokespeople on many TV shows, not to mention being quoted regularly in the mainstream press. But they are also given softball questions and never pressed to confront the serious lapses in their testing process.
That takes us to Antennagate, where CR became a major offender, along with the Web-based tabloid publication that is credited with (or blamed for) disclosing the notorious signal attenuation problems encountered on the iPhone 4. Now maybe the official offender, Gizmodo, has an ax to grind, having gotten themselves embroiled in that notorious missing or stolen iPhone prototype. It stands to reason that they are no longer included on Apple’s official invite list for media events, and good luck to them if they hope to get review products in the future.
Unfortunately, the Antennagate controversy quickly became a YouTube sensation, then got picked up by the major media, particularly after CR was able to duplicate the phenomenon in a specially-configured test that an independent antenna expert regarded as essentially junk science.
Although the iPhone 4 got the highest rating among all tested smartphones, by a small margin, CR won’t recommend it. It doesn’t matter that you can pretty much take any smartphone, as many people have done, and find an appropriate death grip to cause the signal to dip.
Apple has been busy exposing blatant offenders from rival companies on their site, as the executives from those smartphone makers continue to cry foul.
But you wonder how CR might have missed the obvious warnings in manuals and, on occasion, affixed to the smartphones themselves, which explained the potential for signal loss and perhaps lower battery life if you hold those gadgets the wrong way. You’d think a publication that prides itself on being thorough (not to mention fair) would actually read the manuals and make an effort to understand a product’s known limitations.
On the other hand, posting a blog about the iPhone generates far more attention, and hits of course, than if they identified a similar shortcoming with a gadget from HTC, Motorola, Nokia or Samsung. After all, we’re talking about Apple’s “Jesus phone” here, the one that’s magical, mystical and therefore cannot possibly obey the laws of physics.
What’s even more troubling is that CR refuses to admit the testing was not properly conducted. Rather, they imagine that Apple will have an unknown fix at some undefined moment in the future, and, while saying that the offer of free bumpers or third-party cases is a good first step, it’s not enough.
If CR wants to be fair and balanced, they should be demanding the same of every other smartphone maker; that is, free cases. That way, all the products will be on an equal footing, and not susceptible to signal variation under normal use and service.
But I’m not expecting that to happen anytime soon. As long as Apple continues to have difficulty meeting iPhone 4 demand, there’s little to be concerned about. But my respect for CR has gone down several more notches.
Print This Post
I quit subscribing to Consumer Reports long ago. They let the prejudices of their editorial staff override the findings of their technicians.
I doubt if many Europeans follow Consumer Reports and the iPhone 4 is selling like crazy overseas. As long as Apple keeps those free cases coming, CR magazine will have its best use as toilet paper.
I disagree with CR in many instances, thus do not see them as the holy grail. However, I do value their opinion and it’s well worth looking at what they have to say about a product before making a big purchase. But like any review, there is always a certain amount of bias regardless of how objective one tries to be. Part of knowing how to gauge a review is knowing how a reviewer rates things relative to the way you would personally rate the same things.
@arw, My problem with CR is that they seem oblivious to their shortcomings and aren’t apt to recognize when their product reviews fail to deliver useful advice. The iPhone 4 debacle is just a recent example.
Peace,
Gene
CR started to use “Yellow Journalism” a longtime ago!!! They should start printing it on Yellow Paper!!!
No human institution is “incorruptible” although it may avoid the appearance of favoritism. An institution may be honest even when it accepts loans or gifts of the equipment it reviews. One way of verifying an institution’s honestly is when it reviews harshly the flawed products of its advertisers. Consumers Reports does not place itself at risk, this way. Hence, a refusal to take advertisements may not be virtuous; it could be that CR is denying us vital information.
CR could be covertly biased in other ways. We must check the assumptions which CR does not talk about.
The Consumers Union, who operates Consumers Reports, has a consumerist Leftist bias. Its adversarial attitudes verge on paranoia regarding businesses. They have little good will toward them, because they act as though business is a corporatist conspiracy against the people.
Consumers Reports is in good odor among the Press because they share similar prejudices.
CR has utilitarian preconceptions which pretend that products are strictly interchangeable. This means that they dismiss subtleties which the consumers value. Moreover, they imply that such subtleties are irrelevant. This is why they must not value the differences between Mac OSX and Windows. The two OS’s have a surface similarity, after all.
CR’s comments about Antennagate are typical of its biases. The iPhone 4’s problems cannot be merely trivial, accidental, the product of a smear campaign, a peculiarity of its design or symptomatic of mobile phones in general.
Gene, you are more knowledgeable about Apple than CR is. This makes it easy for you to perceive CR’s biases. Being an honest person, you are well aware of your own biases, although you would likely regard them as expert opinions tested by long experience.
If you were an expert on ordinary consumer products such as dishwasher detergent, washing machines, digital cameras or flat screen TVs, then you could see CR’s biases there, too. Unfortunately, the world makes us into specialists. This gives us depth and the perception of subtleties, but it often denies us perspective or a wider view. Fortunately, the Internet allows such experts to voice their opinions freely in public. This means that CR, as a generalist institution, is becoming less influential.
@Louis Wheeler, This isn’t a left-wing or right-wing issue. It’s a factual one. Let’s not put politics into this. If anything, you could call Apple a left-wing company, because Jobs is a friend of the Clintons and Al Gore is on their board.
Or maybe that has nothing to do with it.
Peace,
Gene
@Gene Steinberg,
I agree that it is a factual issue, but one’s bias’ are fair game. Consumers Reports is in the opinion business; that is all they have to sell. How reasonable are those opinions? Where do they derive?
I was merely saying that CR is open to criticism. They should not be considered incorruptible. If CR had a Right Wing bias’, I would remark on that. The Consumers Union has a history which most people are unaware of. How is this off limits? Especially when a good case can be made that CR’s actions are predicable by looking at its past?
I am not opposed to introducing politics into technical situations where appropriate and a good case can be built to defend it. It should not be introduced lightly as a throwaway remark, though. Talking points do not constitute a case. You need evidence and logic to substantiate an opinion.
Steve Jobs, himself, is a Democrat and he has a mild leftist bias. I don’t hold that against him, because he has been an excellent influence on Apple. But, his company profile is mostly apolitical. I seriously doubt that Al Gore is working Board Member.
Even companies run by known conservatives, which are less common than you would think in big business, are often forced to placate political interest groups.
Most of the historical interest groups have had a leftist bias going back to FDR, but that is undergoing revision. The open access of the internet is lowering the information costs; many traditional information sources are undergoing great financial stress. The Mainstream Media, such as the New York Times, is losing viewership, money and influence. Why shouldn’t Consumers Reports be under similar threats?
Why shouldn’t CR use Apple’s prestige to stir up interest in themselves? I can’t prove this true, but it seems highly suspicious when CR refuses to recommend a mobile phone which they gave their highest marks to only two months before.
Gene,
You are spot on about Consumer Reports and their jack-of-all-trades problem. I have felt this same way for the past ten years, and have even used the same words to describe it.
I’m a bit of an “expert” in the insurance field and have watched with frustration and disdain as they try to evaluate insurance policies, and are consistently in error. Their knowledge of the subject is far too shallow. But more worrisome, it’s far too easy to see their bias and even their agenda.
I say all this as someone who still subscribes to the magazine after 20 to 30 years… for auto and appliance reviews, etc.
Thanks for the good work Gene. God bless.
CR is no friend of Apple’s. But I guess we can thank them for the free cases.
If the Consumer organisation in the US is anything like the New Zealand one, it’s run by idiots who haven’t got half a brain cell to share among them. You do NOT test the performance of a Macintosh against a load of Windows machines by running Windows on it using an emulation program (Boot Camp). After all, would anyone test a load of Windows machines against a Macintosh by running OS X on them using an emulation program? Of course not! So what’s sauce for Windows machines should also be sauce for Macs.
You are engaged in a number of misconceptions, Laraine.
Bootcamp is not an emulation program; it is a means of natively running Windows in a separate partition on Mac hardware. The OS’s do not run concurrently like with Parallels Desktop for the Mac or VMware Fusion.
Under BootCamp, Windows is completely in charge of the Apple hardware, It is much the same as when installing Linux on Wintel hardware. Running BootCamp has all the problems of Wintel, such as a need for Anti-vrus programs. The Mac OS has no need for anti-virus software at all, while it is a necessity on Windows.
Furthermore, the fastest tests of running Windows Seven has been on Mac Hardware. This makes your comments correct in a rather odd way, because Apple hardware is less problematic than Wintel hardware. Too often, Wintel manufacturers cut corners to save pennies and this sabotages performance.
There are no legitimate means of running Mac OSX on Wintel machines. Hackers have done this with mixed results and disappointing performance. This is often due to the substandard hardware on Wintel machines.
What we can test for is the speed of applications which exist on both OS’s. Too often, because CR has a Wintel bias it “accidentally” introduces problems which do not really exist. It introduces irrelevant considerations and expectations which do not apply to the Mac.
” the fastest tests of running Windows Seven has been on Mac.”
Not according to Consumer New Zealand, whose test results don’t make sense to me. They gave the Mac top marks for performance in one table, and then took it away in another table where (ostensibly) the only thing the Mac beat the Dell Alienware M17X Core i7 on was in “Communications”. For Memories, TV & movies, gaming, music, productivity and HDD they had the Dell down as best. When you look at the tables in juxtaposition, they clearly contradict each other. And it’s still downright dumb to compare a Mac with Windows boxes under anything but its own native software.
Yes, I know all about the virus problems. I’ve been using a Mac since May 1986. I wouldn’t want Windows on my Mac even if I could have it. I can’t because my Mac is a 6-year-old 1.8 G5. But my husband has an Intel Mac and he doesn’t want Windows either.
Anyway, unless members of Consumer New Zealand have only half a brain cell, they should soon see all the blatant flaws in Consumer’s so-called test.
Our priorities and preconceptions can often mess us up. Thus, we don’t compare equitably. Hence, I doubt CR’s veracity.
It is unclear if Consumers Reports even knows how to create a level playing field. Their parent organization, the Consumers Union, has had a long history of supporting far leftist causes and legislation. This leads to Consumers Reports positioning themselves ideologically, and often expressing an anti-business bias.
Since the Internet allows owners of products to express their views, I no longer believe that there is much need for consumer magazines. Especially, those who’s motives are questionable.
About nine months ago, I read about a speed test between Dell and Mac laptops (MacFeats, not CR.) Running applications on a Mac using Snow Leopard was about 10% faster than Dell laptops using identical processors and GPUs. The Mac was consistently faster at everything but games, in which the Dells were 20% faster. The Dells also had much poorer battery life.The reason was that Apple intentionally throttles back the GPU to conserve battery power, while Windows does not. It shows their priorities.
@Louis Wheeler, I know it’s hard, Louis, but you have to take your politics out of the equation. The reason Consumer Reports screws up tests of tech gear is because their methodology is flawed, not because their politics and yours aren’t in alignment.
The reasons Mac gaming performance is inferior are far more complicated than alleged underclocking of graphics chips. Drivers have traditionally not been well optimized for games, and most gaming companies deliver superior support for Windows. WIth the arrival of Valve for the Mac, that disparity may be changing, at least to some degree. In light of the success of games on the iOS, Apple is taking that market seriously at long last.
Peace,
Gene
And as I keep telling everybody, Gene, a computer is NOT a toy.
@Laraine, CR, actually, can be pretty accurate when it comes to reviewing mashed potatoes. They recently reviewed a product here in the U.S. that actually tastes as good or better than the home-brewed variety, and they called it “Excellent.” But I don’t give them near as much credibility when it comes to autos, where, other than safety considerations, they’re frequently off the mark. With tech gear, they haven’t a clue.
Peace,
Gene
People are complaining on Consumer NZ’s web site that their car test is 4 years out of date. If I wanted a new car (and pigs might fly, Laraine!) I would probably buy a magazine specifically devoted to testing new cars. But our car is second-hand: a Mitsubishi L20–what you would call a pick-up but we call a ute–that doesn’t even have electric windows, never mind a computer.
Can Consumers Reports be trusted? I think not. I gave what I thought were valid reasons for my distrust. Their tactics and positions, too often, align with the ideology of consumerism from which they originated.
This is not, so much, about CR’s politics as their honesty. If Consumers Reports were a holy roller group with a creationist bias, I would distrust them too. There are groups on both the Left and Right who don’t mind shading the truth for a propaganda advantage. I have serious doubts about Creationism and Darwinism. Neither proponent has built good enough a case to persuade me. Even if Natural Selection works, this does not automatically prove that God does not exist. When both sides of an argument are illogical, I must discard both.
I have met people on the left who are intellectually honest. I really do value them, because they are rare. With them, any disagreement eventually breaks on opposing world views. I believe in absolute, not relative, truth. That is, that there is truth and that it will affect me whether I believe in it or not. Sometimes, we only find the truth when it smacks up upside the head. Intellectual honesty kicks in only when we recognize that we have been in error.
Unfortunately, the Left remind me too much of a cult. They believe that when someone studies their arguments, then they must come away agreeing with the Left. That smacks too much of bigotry.
I am not judging Consumers Reports harshly. I am merely reporting the facts as I see them. I don’t shade the truth or favor anyone; I merely try to see the facts as they are. My politics would not keep me from declaring it so if I found they were truthful. I always speak up when I see logical errors from anyone on my side of an argument. Winning an argument fraudulently does not favor our cause.
—
I agree that gaming performance is complicated on the Mac. Gaming is not one of Apple’s core competencies. Microsoft and the Wintel manufacturers have made that a priority. Sometimes, this has led to security concerns. Short term thinking often leads to intractable problems.
I mainly agree with your position. I see great hopes for OpenCl, but that is still immature. Considering where Apple was back in 1998. I think Apple has done rather well; it can only address so many concerns at a time. Gaming, Big Business and Government niches are Microsoft fiefs. Apple can nibble away at them, but it is not in Apple’s interest to make a frontal assault.
My comment about speed on the Mac is valid. The results are verifiable at Bare Feats. This can be explained by the fact that Snow Leopard has much less bloat than Windows Seven. Hence, it would tend to run a little faster.
Remember “The Mac Is Not A Typewriter” by Robin Williams? Well, I think somebody needs to write a book called “A Computer Is Not A Toy”. 😉
Actually, I find the lack of quality in Consumer New Zealand’s testing these days rather sad. When I started work in 1960, at what was then the Department of Industries and Commerce (people called us Idiots and Comics) Consumer was part of the department and had been in existence for about a year. Maybe they tested more mundane things n those days; maybe I was just more naive. I was, after all, only 16.