Yet Another Reviewing the Reviewers
December 9th, 2009Please Note: We’re a little late today. Your long-suffering editor is recovering from surgery and things got, well, a little complicated Tuesday. But the situation is far more reasonable today.
There was a time when publications had copy editors and technical editors to make sure that the material they published was as accurate as possible. This was particularly true for tech publishing, where lots and lots of intricate detail must be as accurate as possible.
Well, it’s not as if those positions have actually been exorcised from the ranks of a publisher’s staff, but you have to wonder sometimes if they are actually earning their paychecks. Consider a certain consumer-oriented magazine that doesn’t accept ads, and won’t even allow a company to reprint a favorable review. They even claim to buy all the products they test, so you can be totally assured they maintain an extremely high level of accuracy.
But in lots of ways, Consumer Reports falls down on the job. Part of the reason is that the magazine is often simplifying (notice I avoid the phrase dumbing down) down complicated information for a general audience, but you can also see a bias that should not exist in a publication with such high and mighty pretensions.
Take CR’s reviews of Macs. Yes, Apple’s gear gets high ratings, but you wonder sometimes the basis on which they arrive at certain conclusions and certainly about the outright factual errors or omissions.
Just the other day, in fact, I was examining the CR Buying Guide 2010, their magnum opus paperback work that contains the essence of all of recent tests evalating over 1,500 products. You would certainly come to believe that they are going to take extra time to recheck their facts before distilling down longer feature articles to be summarized in their annual book.
However, it’s the little things that count.
So in their roundup of reviews covering computers, phones, peripherals and related products, you can see subtle examples of a political agenda. In the section on RAM, for example, they conclude: “Computers with 3GB [of RAM] can be slightly faster; we don’t think any more than that is beneficial.”
Anyone who has seen their Macs slow down because lots of apps are running, and how they seem to just sing and dance with a memory upgrade to 4GB and higher wouldn’t believe that claim for a moment. But when you realize that millions of older 32-bit PCs can’t address more than 3GB of physical RAM, even when 4GB is installed, you can see that CR must be behind the times. And, no, the article doesn’t recognize the existence of a 32-bit and 64-bit version of Windows, nor what 64-bit even signifies.
CR also lists the processors used on PCs, concluding that Apple strictly uses the Intel Core 2 Duo, while mentioning the wider variety used on PCs. They ignore the existence of the Mac Pro, which has, since its introduction in 2006, contained Intel Xeons. While they do mention the arrival of the Intel Core i7, they say nothing about how many processor cores it contains or why this might be important, nor do they mention that one of the early models to be equipped with that new chip is the iMac; the one, in fact, on which I’m typing this article.
The review of smartphones mentions the older version of the iPhone (the 2008 3G in fact), and continues to claim the audio quality of phone calls on this device is worse than average. Unfortunately, CR’s precise methodology is never discussed, so I can’t tell you where they get that impression, or whether the problem is more carrier-related in their locale than attributable to the product itself. In any case, having used a number of mobile handsets from AT&T, Sprint and Verizon Wireless, I can tell you that the iPhone sounds as good (or bad) as any of them.
Unfortunately, the technical lapses can’t be blamed strictly on a single consumer product review publication. Various and sundry tech magazines also demonstrate the lack of technical chops. I am particularly concerned about Mac|Life, the successor to MacAddict, because they clearly need to beef up their review process. I do hope the new editor, Paul Curthoys, whose credentials include a stint at a magazine covering the NeXT computer way back when, will set things right.
In the meantime, we have that recent review of Epson’s Artisan 810 all-in-one printer that neglected to evaluate its copying, faxing or scanning capability, and conveyed the impression that print quality was the only factor worth considering. The reviewer also forgot to mention that niggling driver bug, where Mac OS X’s Collate feature, to print documents by the last page first for easy sorting, is unfortunately unsupported.
In the January 2010 issue of Mac|Life, they carry a review of a wireless charging product, WildCharger Pad, and give it a pretty unfavorable evaluation due to the ill-fitting charger case, but the conclusion is that the product is “Solid,” the equivalent of three stars, I suppose. Now yes, it does work, but the review appears to describe a product that’s noticeably subpar; the rating seems awfully optimistic.
I could go on. I have little hope that CR will mend its ways. But I would hope new leadership at Mac|Life will provide a sorely needed dose of technical authority to a rather middling magazine.
Print This Post
good overall point here. but in fairness to CR, it is impossible to do a really thorough review of multiple competing complex computer products in just a few pages. a lot of important details and qualifications just can’t fit. contrast that with the much greater detail they give to auto reviews. this month they use 7 pages to review 69 cell and smart phones (the cover story), while devoting another 7 pages to review just six models of cars! of course they would point out the cars cost you 100x more than a phone, etc. etc. and so warrant the much greater scrutiny. but nonetheless smartphones are really as complicated as cars to understand as a consumer product buying decision.
their real weakness tho are the comparison/ranking charts. the several “test” factors they choose to evaluate for red/black dots are flat-out arbitrary. most make sense, but others are always omitted. on the smartphone list for example, variety/quality of available apps and integration with your computer programs and cloud services is omitted, and so is the breadth of accessory products available. all three are really important considerations. the iPhone was still the top ranked smartphone, but only barely (3 points above the HTC TouchPro2). if those three factors had been added, that differential would have been much bigger.
and worst of all, their scoring system is never detailed – how they weight the factors relative to each other. for smartphones, one might think ease of use (“navigation” in the CR list) would be the single most important factor. did CR do it that way? the reader can never know.
there is no anti-Apple bias at CR. its products consistently get top rankings anyway. but the way it dumbs down or narrows the discussion of computer products turns it into a Readers Digest equivalent instead of definitive information.
CR has the same issues with regards to car reviews. I read their reviews of entry-luxury cars a few months ago when shopping and they made no distinction that the BMW 328i and Mercedes C300 are sportier and more fun to drive than the Lexus ES350, but did single out the Lexus as being more quiet and comfortable. That the Lexus is more quiet and comfortable is precisely because it is less fun to drive, and vice versa. Like with phones or computers, they have their own arbitrary idea of what aspects of a car are most important to buyers in general, rather than to buyers in the specific category. Buyers of premium luxury cars likely care very little about fuel economy, while buyers of hybrids likely care very little about cornering ability or acceleration.
Where CR does very well, however, is in reliability information. Combined with JD Power for cars, you get a pretty good idea if a given model will be troublesome or reliable. CR tracked very accurately the deterioration between 2000 and 2004 of Mercedes Benz reliability, by model, and has since accurately tracked the company’s recovery to its former high standards. The same goes with computers, where they gloss over the differences in operating systems, architecture and the like with broad generalizations, such as thinking only in terms of 32-bit Windows when looking at benefits from RAM upgrades. As the previous poster correctly states, however, CR gets it right when looking at reliability and owner satisfaction.
That CR doesn’t aim its reviews of any product at enthusiasts in that category is just a reflection on their focus, and I feel does little to reduce their value. For example, I bought a vacuum cleaner last week and there are many significant differences. The one I bought only had a middling rating in CR, but owner satisfaction and reliability ratings were very high. Does CR discriminate on the number of attachments or the sucking power? They correctly state that much cheaper vacuums suck just as hard, and ding the model I bought for its high price compared to those cheaper models.
They are absolutely correct, my carpets would be just as clean with a cheaper model, just as I would get to work with a quieter ride in a Lexus and a lower cost laptop will open a web page or handle a word processing application just as well as a more expensive model. CR will say little about the very cool ball roller on the expensive vacuum that makes it very easy to maneuver. They don’t mention the amazing road feel in a BMW or the reassuring heft to the steering in a Mercedes Benz. They are silent as to jewel-like construction of a MacBook Pro or the stunning display of an iMac. Thats fine, buyers who want more in-depth analysis can find it in more enthusiast-driven reviews. I read the reviews on Edmunds to get a feel for whether a given car is worthy of a test drive, but only after checking on CR to see if its a model plagued by problems.
@Andrew, Well, CR gets that information from really general reader surveys. I think the surveys are highly flawed because the questions are not substantive.
Peace,
Gene
I wouldn’t be so quick to condemn on the RAM issue – although it’s impossible be certain from the minimal context. Nehalems have triple banked RAM, which means that accessing three 1GiB sticks will be faster than two 2 GiB sticks, which would force a fall back to double banking. So long as your computation fits in 3 GiB, that configuration will be faster. The next equivalent speed would bt 6 GiB.
@YDD, The concept is way too sophisticated for their purposes. I suspect they were thinking of 32-bit PCs, using older CPUs.
Peace,
Gene
I am right with you on your comments here, except that I think Mac|Life may be too far gone to save. It has all of the same problems as other print publications plus the magazine’s editorial approach has been made over a half dozen times and is now all but impossible to discern. Their podcast has a juvenile flavor that suggests one possible direction I suppose. The death throes of print are really painful to watch.
Gene,
The death of Macaddict and the birth of Maclife was a sad day. I had a subscription to Macaddict and looked forward to new issues. It was refreshing to see people who wore the “fanboy” tag with pride, made fun of the Windoze drones and provided pretty good in-depth details of hardware and software. They would even “dis” their sister publication “Maximum PC”.
New corporate ownership and an apparent policy of “political correctness” (which I thought only applied to any whiff of putdown towards non-white males and females of any age, sexual orientation, religion, etc not computers) killed the magazine I loved. Now if seems we have “Maclite”.
I no longer subscribe and give the web site the once over “lite-ly” it deserves.
CR’s surveys are flawed.
CR should be a “starting point” for savvy shoppers. In the past, they’ve been pretty good at allowing someone to eliminate the dreck not by calling trash, trash but by downgrading products for sometimes trivial niggles.
I’ll be the first to admit they’ve been less than kind to the Mac. They are a “consumer” guide. Consume means to buy. In their surveys they limit the age of the products they want to know about. They don’t want to hear about the 34 year old Kelvinator fridge that’s been running 24/7 without service, the 4.0L Jeep six-cylinder that with only spark plugs and oil changes is now at 300,000+ miles, or the Bondi Blue iMac still chugging along in the spare bedroom.
Last time I looked “durability” isn’t a CR measurement except in maybe tires and fairly new cars. Mac people buy new Macs because we want a new computer, not because we need a new computer.
I’ve been using computers since my wife bought a Compaq luggable with two 400Kb disk drives and I still can’t adequately convey the superiority of the Macintosh experience.
CR’s staffers don’t live with a product. Cars bought and sold every 18 months and use is rotated among testers. Televisions tested in a lab or viewing room. Toasters see maybe two dozen loafs of bread, not five years of 8 pieces of toast daily.
You and I know that living with a Mac is incredibly better than dealing with a Windows PC.
The CR survey is voluntary and drawn from their readership. It’s long, boring and covers soup to nuts. Because of the squeaky wheel law, many forms are filled out because someone wants to “brag or drag” on a product. But CR is now surveying a statistically significant population of Mac users and they are having to listen to the game show host utter “survey said…Mac!”
On the whole, that’s a good thing.
’til later,
Gordon