Transparent lines of responsibility critical to support reliability of studies, experts say

In a recent high profile enforcement action on the part of the Federal Trade Commission, the agency strongly criticized a study on a green coffee bean extract supplied by Applied Food Sciences​.  The agency alleged that the study design was flawed and that the data, which was generated in India and later written up for publication by two researchers associated with the University of Scranton, had been tampered with by the original research team in India.

The FTC has cited a number of what it said are critical flaws in the study. It stated the study was too small, at 16 subjects, to provide convincing data. FTC also noted that the largest amount of weight loss occurred during the washout periods, and the largest weight loss recorded occurred in a group receiving a placebo. It also criticized a lack of clarity about the blinding of the study and whether participants exercised during the study.

“These flaws undermine the reliability of all of the study data,”​ the agency wrote.

Clear lines of responsibility​

What is uncommon is not to have those areas of responsibility clearly spelled out for readers of the study, said Douglas Kalman, PhD, director of nutrition at the contract research organization Miami Research Associates.  And it is more uncommon still to have some of the participating parties in the publication of the study not be involved in a common effort from the outset.

“It is not common practice in research for a clinical trial to be done by one party and yet another party who was not involved write it up as a study of their own,”​ Kalman told NutraIngredients-USA. “While it is not uncommon in scientific research to employ medical writers to aid in translating data into a journal article, it is odd to have a study conducted by contract in one country yet written up by someone in a different country who was not involved in the study set up and so forth. The oddity is that with no transparency, how was a system of checks and balances carried out to help ensure data integrity and that the protocol was carried out to full GCP standards in acceptable format?”​

Having an integrated research team made of up members from different organizations is standard operating procedure in the pharmaceutical realm, said Dr Jay Udani, CEO and medical director of the firm Medicus Research. Udani offered insight into how large scale studies are carried through, while at the same time emphasizing that he was not commenting on the AFS situation in particular. Large scale clinical trials, some of which have multiple sites, almost demand a division of labor, he said.

“The different functions of a clinical trial can be distributed among multiple organizations.  One might be involved in the performance of a study, another is responsible for oversight, and others could be responsible for data entry and statistical analysis.  Another organization could be responsible for medical writing,”​ he said.

But critical in that chain is the need for clear lines of responsibility and a clear and transparent oversight function.

“It is not out of the realm of normal practice for multiple organizations to be involved in a study, but you have to know what were the roles of each organization. What was the scope of work? ​Generally when you have a clinical trial set up and you have multiple roles those roles are assigned at the outset. Within our organization we have centralized all those roles and we perform all of those roles,”​ Udani said.

“There have been some cases in which organizations are asked to do all of those steps and may or may not have the resources to do so. Some smaller organizations may not have the full internal skill set to complete the whole process,” ​he said.

Claims based on science​

Duffy McKay, senior vice president of scientific and regulatory affairs at the Council of Responsible Nutrition, said companies have to be extra cautious when choosing what claims to make based on the available science.  An ingredient supplier may tout the scientific underpinning of their wares, but FTC will hold the brand holder of a product liable for the claims they make on their products​. This is especially true in the weight loss category, where claims mentioning number of pounds lost and specific time frames tend to get the agency’s attention.

“Companies, when they are marketing in the weight loss category, have to go the extra step to make sure their science is well vetted and their claims are in line with the science that they have. Claims of losing weight without diet or exercise—when you have those kind of claims on an ingredient you have to take a second look,”​ he said.

“What also is an issue is how marketers, companies and so forth take data from studies and more than liberally promote exaggerations and such. This hurts everyone,”​ added Kalman.

Checks and balances​

The AFS study was published in a peer reviewed journal called Diabetes, Metabolic Syndrome and Obesity: Targets and Therapy​​. In theory, the peer reviewing process should have picked up the issues cited by FTC. But the process doesn’t always work as designed, and that’s true of studies done in a number of fields, not just nutrition.

“There are checks and balances in the data monitoring phase. Then there is the journal’s responsibility and the peer reviewers. There can be a little bit of culpability there. But even with all the checks and balances we know that some bad studies can get through,”​ McKay said.