New Study Reveals Holes in Wearable Device Scores

By Frederic Sabater Pastor,
CTS Expert Coach,
Associate Researcher at University of Perpignan Via Domitia

In recent years the market has seen an influx of new companies creating wearable devices to measure sleep, recovery, breathing, blood glucose, blood oxygen, and more. They are developing innovative products and algorithms, and doing some fantastic marketing. Athletes are eager for devices that promise to improve fitness, enhance recovery, or provide insights that help them perform better. But marketing moves faster than evidence, and once science catches up, the claims don’t always match the results.

Some information wearables provide is both accurate and useful for athletes and coaches. That appears to be the case with heart rate variability (HRV) or resting heart rate (RHR). Other types of data are useful in certain situations, like pulse oximetry. However, marketing claims tend to be ahead of the science that validates them, and strain and recovery scores can be one of such cases.

Evidence on Strain and Recovery Scores

A recently published paper studied the usefulness of the WHOOP strap. The researchers’ goal was to test whether WHOOP-derived measures (including HRV, RHR, Strain and Recovery) were associated with laboratory measurements of metabolism (resting metabolic rate and T3 Thyroid hormone). They also looked at whether WHOOP measures are “significantly associated with a validated, sports-specific questionnaire commonly used to capture sports-specific and general stress, as well as recovery and the balance between stress and recovery, that is, the RESTQ.” In other words, researchers looked at physiological and psychological measures that indicate an athlete’s stress and recovery states, and compared them against the information presented by WHOOP’s strap and app.

What the scientists did:

The authors recruited an NCAA Division 1 Swim team, which included male and female Olympic Trials qualifiers, All-Americans and national team members from different countries. Subjects wore a WHOOP strap during training, but researchers specifically targeted a 6-week “Overload” period, as classified by their coaches, which would lead to a peak just before the championship season.

The rationale for timing the study was that the training overload would lead to increased stress (in part related to Energy Deficiency induced by the increased training load). This energy deficiency would then result in lower resting metabolic rates and thyroid hormone levels, both of which are metabolic markers that have been associated with energy deficiency. The overall stress would also be reflected in the RESTQ.

It is important to recognize that the purpose of this type of research is to find less invasive and less expensive ways to gather valid information about athletes. The lab tests used to evaluate lower metabolic rates and thyroid hormone levels are both invasive and expensive. If the metrics from a WHOOP strap can be shown to correlate with lab measurements accurately and reliably, coaches could track athletes’ stress states more frequently or consistently.

What WHOOP measures:

Besides HRV and RHR, WHOOP generates a few of its own metrics that were used in this study. It is important to point out that these metrics are black boxes. We have no idea what algorithms wearable companies use to calculate those metrics, or even what data they consider for input variables in their equations. That’s not a knock against WHOOP, specifically. It’s true across the wearable sensor/device industry.

WHOOP says Strain is calculated based on how high heart rate was during a workout. Recovery is calculated based on a combination of metrics, including HRV, RHR, “sleep performance”, breathing rate, skin temperature and blood oxygen levels. Recovery is then presented as a percentage. Each of those variables has the potential for error when measured by an arm-mounted device. We don’t know how the information is combined to come up with a value to present to an athlete, nor how the algorithm corrects for potential measurement errors. Finally, WHOOP attempts to communicate exercise energy expenditure, which seems to be based on heart rate during exercise.

What researchers found:

These swimmers were serious. They completed three strength training sessions per week and swam almost every day. They averaged 18 hours and 20 minutes of total training per week, swimming 6700 yards per day. However, something that seemed a bit strange from the data was that they reported average exercise energy expenditures of 901 kcal/day for the males and 578 for the females. This is notable because the men tipped the scales at 83.8 kg on average (185 lbs) and averaged 2 h 34 min of training per day. That energy expenditure seems incongruous with the level of training and caliber of the subjects. It is likely that the energy expenditure (which was estimated from heart rate, which was also used to calculate Strain) was underestimated.

Besides the energy expenditure glitch, they found that HRV was significantly correlated with sport-specific stress and total stress, as well as with lower-than-expected resting metabolic rate (which would be considered metabolic suppression, linked to energy deficiency and possibly overreaching). This is not surprising, since WHOOP has been previously shown to be pretty good at measuring HRV, and HRV should definitely get worse (i.e., lower) with stress. However, Strain and Recovery showed no relationship with any of the metabolic or RESTQ-derived stress variables.

How do we know?

The table below reports r values, which is a measure of the linear trend between two variables. An r value of 1 means there is a strong positive correlation between the two variables, whereas -1 indicates a strong negative correlation. An r value of zero means there is no correlation between the two variables. In the table below, let’s use “total recovery” row as an example. Across HRV, RHR, Strain, and Recovery, the correlation with “total recovery” as measured by the RESTQ resulted in r values of -0.05, -0.18, -0.03, -0.01, respectively. In other words, virtually no correlation.

In regard to the table below, researchers reported, “We found a negative correlation between HRV and sport-specific stress (r  =  −0.462; p  =  0.026) and total stress (r  =  −0.459; p  =  0.028) in all participants. No other correlations between HRV, RHR, strain, or recovery and general stress, any recovery subscales (general, sport-specific, and total), or recovery–stress balance variables were found in all participants.” In research contexts, strong correlations should have an r value between 0.8 and 1.0 or between -0.8 and -1.0. The two negative correlations they found, at r  =  −0.462 and r  =  −0.459, would be classified as weak to moderate correlations.

Male vs. Female Data

When data were analyzed by sex, none of the WHOOP variables were related to metabolic or stress variables for the females. However, in males, they found significant relationships between stress and HRV or resting heart rate. The interesting twist was that they found significant correlations between Strain and stress variables derived from the RESTQ (general stress, sport-specific stress, total stress and stress balance). In other words, the RESTQ – a low-tech questionnaire – revealed the expected correlations when the wearable device did not.

Another interesting piece was that the correlations were in the opposite direction that I would have expected: higher Strain corresponded with lower stress levels. The authors speculated that perhaps athletes who felt less stressed were able to push harder during training, thereby generating higher strain scores. As an aside, a suppressed heart rate during exercise at a given intensity has been associated with overreaching, which could explain that, because Strain relies on heart rate data. But then, if that means we cannot take heart rate into account, it would mean that Strain would lose some of its ability to reflect how strained an athlete is during exercise.

Final thoughts

In their conclusion, the authors said that WHOOP-derived HRV may provide insights into metabolism and sport-specific stress among athletes. That’s a well-established conclusion; HRV is useful for endurance athletes. However, this study should also serve as a word of caution about metrics derived from black-box proprietary algorithms, such as Strain and Recovery score.

It is important to make a distinction between “measures” of things that can be measured (such as heart rate or HRV) and “estimates” of unknown parameters. Some of these unknowns are impossible to measure and assigned a specific number, such as “stress”, “readiness”, “recovery”, “body battery”, or “sleep quality”. At best, those will be educated guesses. That doesn’t mean they have no value, however. There may be a use for them in training (for example, looking at trends over time), but we caution athletes not to be over reliant on algorithm-based scores to obtain insights or make decisions about the training process.

Think of this as the classic “trust but verify” scenario. Pay attention to the scores over time, but always view them in context of measurable data and subjective perceptions of stress and recovery (which evidence continues to show are remarkably accurate). In many cases, our own subjective perception of stress may be at least as good as any wearable device.

About Coach Frederic

Frederic Sabater Pastor @fredericspast is a multitalented and multilingual (English 🇺🇸, Spanish 🇪🇸, Catalan ) Spanish coach based out of France. He has published multiple acclaimed articles in the International Journal of Sports Physiology and Performance in the past few years such as “VO2max and Velocity at VO2max Play a Role in Ultradistance Trail-Running Performance”. In addition to his success in academia, Frederic has had coaching success with athletes in a variety of disciplines including triathlons, track and road races, obstacle course events, and ultra/trail races.

References:

Kellmann, Michael, and K Wolfgang Kallus. Recovery-Stress Questionnaire for Athletes : User Manual. Champaign, Ill. ; Leeds, Human Kinetics, 2001.

Lundstrom, E. A., De Souza, M. J., Koltun, K. J., Strock, N. C. A., Canil, H. N., & Williams, N. I. (2023). Wearable technology metrics are associated with energy deficiency and psychological stress in elite swimmers. International Journal of Sports Science & Coaching, 0(0). https://doi.org/10.1177/17479541231206424

Miller, Dean J et al. “A Validation of Six Wearable Devices for Estimating Sleep, Heart Rate and Heart Rate Variability in Healthy Adults.” Sensors (Basel, Switzerland) vol. 22,16 6317. 22 Aug. 2022, doi:10.3390/s22166317

Strock, Nicole C A et al. “Indices of Resting Metabolic Rate Accurately Reflect Energy Deficiency in Exercising Women.” International journal of sport nutrition and exercise metabolism vol. 30,1 (2020): 14-24. doi:10.1123/ijsnem.2019-0199