|
|
|
The ability to evaluate research studies can help you figure out which medical treatments are truly effective.
Knowing what treatments work for any medical condition is a challenge for health care providers and certainly for health care users. We are bombarded by study results in newspapers and on the radio, TV, and even via Facebook and Twitter. Searching the Internet provides even more information,much of it overwhelming. But by becoming an educated consumer and learning how studies are designed for research, you can gain insight into which treatments are worth trying.
How Is the Study Designed?
The first step in evaluating a study is to determine its design.How a study was conducted tells us how much we can trust the results. Some studies, although they may provide evidence, can't tell us how effective an intervention is. Here's an example: An investigator teaches biofeedback to people with tinnitus and they report improvement in their tinnitus symptoms.This study, a case series, can't tell us whether the observed improvement was due to biofeedback, or to an improvement that would have happened over time anyway. What's required is a comparison group, a group of people who did not receive the biofeedback treatment.
A case-control study is one type of study that appears to include a comparison group. An investigator may have identified a group of people with tinnitus and compared those who improved within the last year to those who did not, finding that more people in the improved group used herbal supplements.
But can we determine if the group that improved was different in some other important way from those who did not? Perhaps the group that improved had more severe tinnitus originally, or they tried additional supplemental therapy more often. So it can't be said that improvement was due solely to the herbal supplements because it may have been the result of one or more factors, some of which we can identify, some we cannot. (For more on herbal supplements and tinnitus, see "The Skinny on Supplements," page 26.)
Is Bias Limited?
To know what treatments work, we need to look at experimental studies. Experimental studies are planned experiments that compare the result of a test treatment in some study participants with that of a control treatment in other study participants (by using placebo or dummy pills,a different treatment, or no treatment at all), with administration of treatment and follow-up occurring concurrently. In an experimental study, the investigator controls the treatment given to a participant during the study. It is not chosen by the study participant.
Although an experimental study is the best type of study to evaluate the effectiveness of a treatment, we also need to examine how the study was done to make sure the results are valid. We need to determine the internal validity of the study did the investigators design and then conduct the study in such a way as to minimize all potential sources of bias, or any system at icerror that affects the size of the measured treatment effect? The way participants were included in the trial, the way the treatment was administered, and the way the effectiveness of the treatment was measured can all introduce bias.
How did the investigators decide which treatment to give to which participants? Was the treatment assigned to the participant based on some participant characteristic, or was it randomized?
Randomizing which treatment any one individual receives is the best way scientists have of ensuring that the participants in one group are as similar as possible to the participants in another group, and that the only difference between the two groups is the treatment received. This then allows the investigators to measure the true treatment effect.
Randomized controlled trials are experimental studies in which the treatment is randomly allocated to individuals or groups of individuals. They are considered the gold standard for testing therapies.
Randomization by itself is not enough, however. Suppose an investigator has a patient waiting to be randomized into a trial, but is able to look at the randomization list and sees that the next assignment is not the one that the investigator believes the patient should receive. The investigator may decide not to enroll that individual, or to enroll him or her at another time. It is important to conceal the randomization list from the person who is enrolling the participants, because bias is introduced when the investigator decides who to enroll into the trial or when to enroll them.
How Is Treatment Administered?
Once treatment is assigned, it is also important that the study participant, the person delivering the treatment, and the person measuring the effect of the treatment are unaware of the treatment assignment. This is called blinding or masking.Let us suppose we have a trial of ginkgo biloba and its effects on tinnitus. An individual assigned to ginkgo biloba will believe that he or she is "better"the placebo effect if that person knows he is getting the treatment.
Conversely,individuals knowing they are assigned to the control intervention will believe they are not better, even if an objective test shows they are. If the person delivering the ginkgo treatment knows a participant is in the control group,then he may give her extra attention, more advice, or may even suggest additional treatments. And if the person measuring the effect of the treatment knows a participant has been assigned ginkgo, he may erroneously attribute any side effects to the ginkgo treatment.
Knowledge of the assigned treatment by the participant, people delivering the treatment, or people evaluating the study results may influence the perception of improvement,how the intervention is administered, and how the study results are measured.The risk of study error is minimized when as few people as possible know who is assigned to which treatment.
Did the investigators design and conduct the study in such a way as to minimize all potential sources of bias?
Is the Study Analysis Complete?
In many trials, some participants do not complete the trial and results are not available. There are many reasons, and some are avoidable, such as side effects of the treatment, no improvement, or obtaining and taking the opposite treatment. Participants may simply fail to return for study visits,which is called loss to follow-up. If the reason the participant does not complete the trial is associated in some way with the study results, the estimate of the treatment's effect may be incorrect.
No matter why a participant does not complete the trial, or does not complete the treatment regimen, that person should be followed to the end of the trial as much as possible and included in the analysis. The investigators must be able to account for each person.
How the results were analyzed could also result in misleading or just plain incorrect results. It is important that all participants are included in the analysis and each is analyzed according to the original treatment assignment. This is called an intention-to-treat analysis.
Sometimes, the participants did not receive the assigned treatment. What if some of the participants assigned to control decided to try ginkgo anyway? Should they not be included in the ginkgo group? The answer is not hey should be included in the control group. The rule is "once randomized, always analyze." This includes every participant, regardless of whether a participant did not receive the assigned treatment, decided to drop out of the study, or found and took the opposite treatment.
Intention-to-treat analysis is important because the reason a participant dropped out or did not take the assigned treatment may affect the measured result for that person. So the study results may underestimate the size of the effect of the treatment, but it will be a true effect.
In summary, a "good quality" trial has these elements:It is an experimental study with a concurrent test and control group; it has randomized individuals to treatment groups; it has kept investigators enrolling participants unaware of the treatment assignment; it has kept participants,people delivering the treatment, and people measuring the treatment effects unaware of the assigned treatments; and it has an analysis of results that includes all participants and in the group to which they were originally assigned.
Roberta W. Scherer, Ph.D., is an Associate Scientist at the Johns Hopkins Bloomberg School of Public Health. She has had a 15-year association with the Cochrane Collaboration, now serving as the Associate Director of the U.S. Cochrane Center and the U.S. Satellite of the Cochrane Eyes and Vision Group.
The risk of study error is minimized when as few people as possible know who is assigned to which treatment.
ONE WAY TO EVALUATE STUDY QUALITY
Evaluating the quality of research studies can be difficult and time-consuming. One solution is to refer to work by a group called the Cochrane Collaboration. This international, not-for-profit organization aims to help people make good healthcare decisions by finding and synthesizing randomized trial results in rigorous,systematic reviews. As part of the process, expert reviewers evaluate the"quality" of trials. Each Cochrane review includes a plain-language summary for non-healthcare professionals.
Currenttinnitus interventions that have been reviewed in the Cochrane Library include ginkgo biloba, tinnitus retraining therapy, cognitive behavioral therapy, sound therapy (masking), and antidepressants. In process are reviews on acupuncture and anti convulsants. The review for ginkgo biloba, which included three trials that had a low risk ofbias, found no effect on tinnitus.
Summaries of Cochrane reviews are available for free at www.cochrane.org.
If you are interested in being a participant in a clinical trial,contact your local hospital or doctor, or consult the CHEER clinical research network, www.cheerresearch.org.
