So as we speak — more or less — Consumer Reports is rerunning its MacBook Pro battery tests. These are the ones that resulted in a non-recommended rating. That’s the first time such a thing happened to an Apple product, so it had to be extremely important. Potentially, it could hurt sales, and it sure brought lots of attention to CR, which garnered major headlines as a result.

Thus it came as no surprise when Apple marketing VP Philip Schiller responded — a rarity when it comes to a negative review — and explained that the company was working with the publication to see what was up.

On Tuesday, both Apple and CR posted updates on the controversial test results. Those posts got coverage in the tech media, but not so much in the mainstream press that made a huge deal of the original report, flawed as it was. What do I mean by flawed? Well, it seems that CR’s battery test consists of downloading 10 sites from a default server using a notebook’s default browser. This process is repeated until the battery dies.

But here’s the curious part: At the same time, CR disables caching, supposedly to have consistent results among all equipment. But this comes at the expense of failing to match a user’s actual use pattern, which would normally be to download some sites repeatedly, meaning they’d be cached. To shut off caching in Safari, CR needed to switch on the Develop menu in Advanced preferences. So not only was the use pattern not consistent with reality, but they also managed to trigger a rare bug in Apple’s Develop menu. The end result was that logos would be reloaded inconsistently, thus hurting battery life.

While it is Apple’s bug to fix, it is triggered by a test routine that doesn’t reflect real world use. Regardless, Apple fixed it — it’s in the macOS Sierra 10.12.3 public betas that will be made available to all Mac users soon — so the MacBook Pro should pass CR’s battery test.

Otherwise, this bug would probably have have gone unfixed and had no discernible impact.

Now the MacBook Pro has been an erratic performer when it comes to battery life. While most tests from tech publications show it is respectable, usually within the range of Apple’s 10-hour claim, some users are complaining of subpar results; sometimes it’s half what Apple claims, although it’s possible the recent 10.12.2 update may have helped some. But the fix mainly addressed a graphics bug, impacting the switch between integrated and discrete graphics, which would only affect the 15-inch model.

Still, Apple’s claims are thought to be controversial. Are they somehow stacking the deck in claiming 10-hour battery life and 30 hours idle? Did they devise some wacky test scheme that users wouldn’t normally use?

If you look at the fine print on Apple’s specs page for the MacBook Pro (if you can actually read anything that small), you’ll see what they did to come up with their battery life figures:

Testing conducted by Apple in October 2016 using preproduction 2.0GHz dual-core Intel Core i5-based 13-inch MacBook Pro systems with a 256GB SSD and 8GB of RAM (wireless web test, iTunes movie playback test, and standby test). Testing conducted by Apple in October 2016 using preproduction 2.9GHz dual-core Intel Core i5-based 13-inch MacBook Pro systems with a 512GB SSD and 8GB of RAM (wireless web test and iTunes movie playback test) and preproduction 2.9GHz dual-core Intel Core i5-based 13-inch MacBook Pro systems with a 256GB SSD and 8GB of RAM (standby test). The wireless web test measures battery life by wirelessly browsing 25 popular websites with display brightness set to 12 clicks from bottom or 75%. The iTunes movie playback test measures battery life by playing back HD 1080p content with display brightness set to 12 clicks from bottom or 75%. The standby test measures battery life by allowing a system, connected to a wireless network and signed in to an iCloud account, to enter standby mode with Safari and Mail applications launched and all system settings left at default. Battery life varies by use and configuration.

All right, let’s take a quick look. Some will quibble over using a display brightness of 75%. CR uses 100%, but I’ve never seen the need to run my Mac notebooks that high. Indeed, my aging 2010 17-inch MacBook Pro is set the 75% range, and since the new models have brighter displays, I accept Apple’s setting as realistic, and CR’s choice as somewhat on the extreme side. To each his/her own.

It would, however, be nice to know which 25 sites Apple selected, but that many should offer a representative sample of what a typical user might access during the course of a day. The iTunes movie playback rating is based on, obviously, playing back an iTunes movie. What else could it be?

You may have other ideas as to what should be tested to measure battery life more accurately. Maybe devise a canned routine using email access, word processing, and web access. Regardless, what Apple is doing isn’t out of the range of what a normal user might do. Compared to the CR methodology, it makes much more sense since it doesn’t require switching on an obscure setting.

None of this actually explains what is hurting the battery life on some MacBook Pros. Maybe the upcoming macOS Sierra update will address such problems too, since I don’t think the Develop menu bug is having much impact outside of CR. Maybe Apple has more work to do. Meantime, they have posted suggestions on what to do to maximize battery life on your equipment.

This article was posted on Thursday, January 12th, 2017 at 12:00 AM and is filed under News and tagged with: 2010 17-inch MacBook Pro, Advanced preferences, Consumer Reports, CR, Develop menu, iCloud, Intel Core i5, Itunes, Macbook Pro, macOS Sierra 10.12.2, macOS Sierra 10.12.3, Mail, Philip Schiller, Safari.