Why most battery life tests are a useless indicator of real-world usage

Apple’s new MacBook Pros have received widespread criticism for making life difficult for professionals by removing useful features like an SD card slot or prioritizing thinness over increased performance. The criticism was bolstered when the new 2016 MacBook Pro was noted by Consumer Reports for exhibiting sporadic battery life in its tests. Consumer Reports is an 80-year-old, independent, nonprofit organization that tests everything from electronics, cars, health products, cookware and more, and its results are taken very seriously.

That’s why this rating was shocking to Apple as well as many Mac users around the world, because MacBooks in the past generally haven’t shown a huge discrepancy between Apple’s claims and tests in the real world. It was the first time Consumer Reports didn’t recommend an Apple MacBook Pro.

The Cupertino hardware maker was quick to work with the organization to determine if there were any issues with how the laptops were tested. It was eventually revealed that a bug was causing incorrect battery life figures. Apple released a fix, alongside Consumer Reports revealing updated test results, finally getting the publication’s recommendation.

But if you see the updated results, Consumer Reports says the 13-inch MacBook Pro with TouchBar runs an average of 15.75 hours, the 13-inch MacBook Pro without TouchBar 18.75 hours, and the 15-inch MacBook Pro 17. 25 hours. . These figures far exceed the 10 hours of battery life in “wireless browsing” announced by Apple on its official site, for all these models. What’s going on?

The answer lies in Consumer Reports’ standardized process for checking battery life on all the laptops it tests. Here is a quote from his blog post:

Here’s how our battery test works: we download a series of 10 web pages repeatedly, starting with the battery fully charged and ending when the laptop shuts down. Web pages are stored on a server in our lab and transmitted over a dedicated WiFi network. We conduct our battery tests using the computer’s native browser, Safari, in the case of MacBook Pro laptops.

By comparison, here’s the fine print mentioned at the bottom of Apple’s specs page for the new MacBook Pros:

“The Wireless Web Test measures battery life by wirelessly browsing 25 popular websites with display brightness set to 12 clicks from low, or 75%.”

This explains why Consumer Reports may have gotten such high readings – for one thing, in its blog post, Consumer Reports says the screen brightness is kept at 100 nits. Given that the new MacBook Pros have a maximum screen brightness of 500 nits, that means the brightness would have been set to roughly 20%, much lower than the 75% brightness that Apple used for its tests.

Then, the discrepancy could also be due to the sites that were chosen to be loaded, as different sites will consume the battery to varying degrees. Finally, in the Consumer Reports test, the pages were served by a nearby local server. Since Apple’s testing methodologies are more loosely defined, it’s likely that its tests fetch these pages from the Internet, not a local server, which might have a bit more of an impact on battery life.

But none of these numbers come close to what people experience in the real world. Why is that? That’s because in the real world, your mileage may will always vary.

Consumer Reports’ numbers don’t try to determine what the typical customer will experience in terms of battery life – it tries to create a fair test that can help compare battery life across a wide range of devices. laptops from different manufacturers. At Gadgets 360, we do something similar in our reviews – the only difference is that our reviews also include plenty of real-world testing and details on how we’ve used the devices and why.

Lab tests are there to make it fairer to compare numbers from different devices. But once you actually use the device, the actual numbers can be quite different from what a lab test produces, and the review process as well as consumers need to take that into account.

Here’s an example – battery life figures can vary wildly depending on the browser you’re using. For example, from personal and anecdotal experience, it is known that Chrome on macOS is not as well optimized as Safari. As in, you can visibly notice the difference in battery life when using either browser on a Mac. The same behavior occurs when a Surface Book is tested with Microsoft’s Edge browser, instead of Chrome.

In the real world, people have their own preferences as to which web browsers they prefer. Maybe Mac owners want to use Chrome because they want their browsing history to sync with Chrome on their smartphone. Maybe they are forced to use Chrome because some extensions are not available on Safari. Maybe they want to use Safari instead of Chrome because of features like Reader Mode. Or maybe a Windows 10 user prefers to use Microsoft Edge because of the annotation feature. Or maybe they use Firefox.

In the real world, people are probably using more than one browser, more than 25 browser tabs, and many more apps running background processes than any simulated test. No wonder the actual use of any computing device doesn’t live up to the lab test numbers. Some companies like Apple choose to create simulations closer to the real world, while others like LG use a 10-year-old battery test app to claim their laptop runs for 23 hours. But none of these numbers will give you a clear idea of ​​how long it will last for you. Because it all depends on what you are going to do with the computer.