‘You don’t apply statistical analysis to hearsay’: AER expert slams new Harvard analysis
A new paper published in the Journal of Adolescent Health reports analysis of data from the FDA’s adverse events database to conclude that dietary supplements sold for weight loss, muscle building, and energy presented higher risks for severe adverse events compared to vitamins.
The FDA Adverse Event Reporting System (FAERS) is a listing of adverse events (AE) reported to the Agency. There is no analysis of the data, nor does the FDA determine if the events were actually caused by the product(s) in question.
But that didn't stop the scientists from Harvard T.H. Chan School of Public Health and Harvard University Information Technology from concluding: "The dietary supplement industry continues to grow, efforts aiming at reducing access and consumption, implementing proactive enforcement of regulations, and providing clear warning at the point of purchase are paramount in preventing severe medical outcomes among children, adolescents, and young adults and consumers in general."
“It looks really high-tech but means nothing”
Dr Rick Kingston, President, Regulatory and Scientific Affairs, SafetyCall International and Clinical Professor of Pharmacy, University of Minnesota, was highly critical of the utility of the data when it was first released by FDA in 2016, and he slammed the new paper.
Dr Kingston, who is also affiliated with the National Center for Natural Product Research at the University of Mississippi, told us: “The requirements for reporting “alleged” dietary supplement adverse events to the FDA do not require any professional assessment of causation. Even for the data reported in FAERS there is no indication of FDA’s assessment of the potential associations with any individual reported events. So, they throw every allegation in the mix but don’t tell you which ones they believe might represent a signal.
“It’s a passive system of surveillance with lots of background noise and is specifically intended for signal detection of incidents that should be evaluated and confirmed through more robust investigation on a case by case basis. Taking the data and medical outcomes at face value is fraught with limitations.
“You don’t apply statistical analysis to hearsay,” he added. “It looks really high-tech but means nothing. It’s like trawling the chat rooms to calculate incidence rates for heart attacks.”
Media coverage
The paper is already attracting coverage from a range of media outlets. NBC News ran an article with the headline, Teenagers can be seriously harmed by dietary supplements, MedScape has an article with the headline, Some Dietary Supplements Tied to Severe Outcomes in Youth, while Cosmos ran with, Diet supplements should never be used by children.
Dr Daniel Fabricant, president and CEO of the Natural Products Association (NPA), said the industry must continue to educate researchers, journalists, and the public that this data does not show causality. “FAERS is a sentinel system that you use to detect signals,” he said. “It’s not for developing hypotheses with no evaluation of the cases.”
Dr Fabricant also highlighted that reporting of adverse events is always higher for people who are younger. “The natural reaction is to be cautious and take your child to hospital. This is then recorded in the AER system.”
“This paper is a completely inappropriate use of the AER data and it’s built on a false premise. It's nonsense.”
Study details
The Harvard-based scientists used a retrospective observational study design to assess adverse event reports between January 2004 and April 2015 in the FAERS. They found 977 single-supplement–related adverse event reports affecting individuals aged between 0 and 25. Of these, 166 reports involved hospitalization, 39 reports involved life-threatening events, and 22 reports involved death, they wrote.
The highest risk supplement categories were evaluated to be for muscle building, energy, and weight loss.
SafetyCall’s Dr Kingston commented: “What the researchers apparently do not understand is these are spontaneously reported incidents made primarily by consumers with no corroboration by medical professionals.
“As telling as the data fields they reported are the data fields they and other researchers do not have access to in the database including: Incident narrative; who reported event (consumer, friend, relative, health professional, other); original symptom descriptors, including symptom onset, acuity and duration; quantity of exposure; ingredients/profile; concomitant medical conditions, medicines, other supplements/OTCs, etc; and other supporting or confounding factors.
“Lastly, there is no denominator to assess the relative risk. If there is a hundred times as much product being sold for the categories they cite, the relative risk could be extremely low. And, these two categories could be equally safe in the big picture.”
"Irresponsible"
Commenting on the study, Steve Mister, president & CEO of the Council for Responsible Nutrition (CRN), stated: “This is an irresponsible study, and its authors seemingly started with their conclusion that dietary supplements are harmful and worked backwards, disregarding significant evidence along the way. The article completely disregards the incredibly wide margin of safety of dietary supplements, including sports nutrition and weight management products. The authors analyzed 977 Adverse Event Reports (AERs) reported over an 11 year period for the 0-25 age group. While all adverse event reports deserve scrutiny, when viewed in context of the size of the entire dietary supplement market, this number is miniscule and actually demonstrates that these products are incredibly safe overall.
"The authors of the article clearly do not understand the regulation of the dietary supplement marketplace, set in place by the Dietary Supplement Health and Education Act (DSHEA) in 1994, and misinterpret the law throughout the piece. The article ignores how DSHEA, along with several other Federal Drug & Cosmetic Act amendments have given FDA more authority over supplements and foods than ever before. In the 25 years since DSHEA, many additional safety measures have been added to the regulatory regime, not the least of these, the requirement to report serious adverse events.
"The vast majority of dietary supplements, including sports nutrition and products intended for weight management are safe. This article focuses on outlier products that do not come from responsible industry. CRN acknowledges that there is a problem with tainted, illegal products masquerading as dietary supplements, and urges consumers to choose products made by responsible companies who scrupulously adhere to the law. This includes good manufacturing practices, mandatory adverse event reporting, mandatory facility registration, foreign supplier verification and allergen labeling, to name a few.”
Source: Journal of Adolescent Health
Published online ahead of print, doi: 10.1016/j.jadohealth.2019.03.005
“Taking Stock of Dietary Supplements' Harmful Effects on Children, Adolescents, and Young Adults”
Authors: F. Or et al.
