Sleek, high-tech wristbands are extremely popular these days, promising to measure heart rate, steps taken during the day, sleep, calories burned and even stress.
And, increasingly, patients are heading to the doctor armed with reams of data gathered from their devices. "They're essentially asking us to digest the data and offer advice about how to avoid cardiovascular disease," says cardiologist Euan Ashley, associate professor of medicine at the Stanford University Medical Center and Stanford Hospital and Clinics in northern California. And, being somewhat near Silicon Valley, he says he gets a lot of tech-savvy patients bringing fitness-tracker data to appointments.
The problem, he says, is that he just didn't know how reliable that data was. So, he and colleagues decided to study seven of the most popular devices and compare their accuracy to the gold-standard tests that doctor's use.
They looked at two metrics: heart rate and calories burned. For heart rate, the fitness trackers were compared to findings from an electro-cardiogram, or EKG. It turned out the devices were "surprisingly accurate", says Ashley. "Most devices most of the time were 'off' by only about 5 percent."
However, when it came to measuring how many calories a person burned, the findings were way off, says Ashley, showing a degree of inaccuracy that ranged from 20 percent to 93 percent, meaning 93 percent of the time the worst-performing device was wrong. Researchers compared the findings of the wrist devices to a sophisticated system of calculating metabolism which measures oxygen and carbon dioxide in people's breath.
"This is a very well-designed and well-done study," says Dr. Tim Church, a professor of preventative medicine at Pennington Biomedical Research Center at Louisiana State University who was not involved in the study. Church routinely consults with companies about how to introduce wellness strategies into the workplace. Being wrong 93 percent of the time means the findings from the fitness tracker are more "fiction than fact," he says, which can actually undermine a healthy diet. "It's just human nature. People are checking these inaccurate counts and they think they've earned a muffin or earned some ice cream and they're sabotaging their weight-loss program."
Church points to a study last year which found participants in a weight-loss program who also wore fitness trackers actually lost less weight than participants who didn't wear the trackers. "It's an instance of no information is probably better than having bad information," he says.
The Stanford study was published in the Journal of Personalized Medicine. It was relatively small, with 29 men and 31 women. In addition to the primary results, there were some other interesting findings. In certain groups of people — for example, those with darker skin, higher BMIs and men — the error made by devices was actually greater than for Caucasian women with a more healthy weight.
"So, for those for whom it might matter the most, who are trying to lose weight, the error was actually greater," says Ashley, who doesn't know why this may be the case. He speculates that it could be that companies use a fairly narrow group of people for testing the equations they use to measure heart rate and calories burned.
The study didn't look at how well devices count steps or monitor sleep or stress. The take-home message, says Ashley, is to not rely on the devices to measure total calories burned. Instead focus on eating what we know is a healthy diet, which is low in sugar and high in fiber, and to "eat not until you're full but until you're no longer hungry."
And, of course, people should exercise, he says, adding, "we have no more important intervention than exercise for the prevention of any number of diseases."
Makers of two devices, Fitbit and PulseOn say they remain confident in the performance of the trackers both in measuring heart rate and calories burned. In a statement, PulseOn questioned the study's methodology, saying that the high errors for calorie measurements "suggest that the authors may not have properly set all the user parameters on the device."
Mark Gorelick, the chief science officer at the device-maker Mio Global, says, "we agree that more accurate calorie estimation is important for the industry as a whole, since most individuals are monitoring calorie deficits for weight loss." The other device makers did not immediately respond to requests for comment.
AUDIE CORNISH, HOST:
Companies that make fitness trackers advertise great things in a small package. A sleek wristband with a small digital screen can measure how many steps you take, how well you sleep, how many calories you burn, even how stressed out you are. But are fitness trackers accurate? A study out today tries to answer that question, as NPR's Patti Neighmond reports.
PATTI NEIGHMOND, BYLINE: It's happening more every day, patients coming to see cardiologist Euan Ashley armed with reams of data gathered from their fitness tracking device.
EUAN ASHLEY: They may be bringing in heart rate monitoring data or step count data or calorie estimation data. And they're saying, this is something that I want you to use in giving me lifestyle advice about how to avoid cardiovascular disease.
NEIGHMOND: The problem was Ashley just didn't know how reliable the data were. He works at Stanford Hospital and clinics in California. He and colleagues decided to study seven of the most popular devices and compare them to the gold standard doctors use. The first measure - heart rate. The fitness trackers were compared to findings from an electrocardiogram.
ASHLEY: We were pleasantly surprised to find that it was actually pretty accurate.
NEIGHMOND: Most devices most of the time were off by only about 5 percent. But when it came to measuring calories burned, the findings were way off when compared to a sophisticated system used in doctors' offices.
ASHLEY: There was a significant degree of inaccuracy with the calorie estimations. In fact, it ranged from 20 percent up to 90 percent.
NEIGHMOND: The worst-performing device was off by 93 percent, making its findings more like fiction than fact, says obesity and exercise researcher Dr. Tim Church.
TIM CHURCH: People are checking these inaccurate calorie counts and they think they've earned a muffin or they've earned some ice cream. And they're sabotaging their weight loss program.
NEIGHMOND: Church points to a study last year which found participants in a weight loss program who also wore fitness trackers actually lost less weight than participants who didn't wear the trackers.
CHURCH: It's an instance of no information's probably better than having bad information.
NEIGHMOND: The Stanford study published in the Journal of Personalized Medicine was relatively small, 29 men and 31 women. Participants were a diverse group, says Ashley, and there were some other interesting findings.
ASHLEY: In certain groups of people - for example, those with darker skin or males or those who are a little bit larger - the error actually was greater. And so for some of the people for whom it would matter the most, maybe those who are trying to lose weight, the error was actually greater.
NEIGHMOND: Ashley doesn't know why this is but speculates it could be that companies use a fairly narrow group of people when designing the equations they use to measure heart rate and calories burned. Makers of two devices, Fitbit and PulseOn, say they remain confident in the performance of the trackers in measuring both heart rate and calories burned. In a statement, PulseOn takes issue with the study's methodology. The other device makers didn't get back to us with a comment. Patti Neighmond, NPR News.
(SOUNDBITE OF THE LIMINANAS SONG, "TROUBLE IN MIND") Transcript provided by NPR, Copyright NPR.