It sounds pretty impressive to hear, "A new study suggests . . . ." We perk up our ears and open our minds. After all, "science" has discovered something new! Proteinaholic, a recently published book authored by Garth Davis, MD, with Howard Jacobson, PhD, highlights the difference between how news reporters and scientists react when they hear about new research. "Reporters want to know 'what's new,' while scientists are trained not to trust brand-new results that contradict established findings." The result of this difference is that "small, badly designed studies with anomalous findings are reported as medical breakthroughs, instead of outlier //www.drcarney.com/data that needs to be replicated with rigor before informing public discussion and policy." How can the average lay person decide if the "newest, shiniest" study being heralded in the media is actually useful or not?
In chapter eight, Dr. Davis offers seven points to help us evaluate a medical study. In order to whet your appetite and to encourage you to run out and purchase the book, we are reviewing those seven points for you here:
1. "Never believe anything [found] in a newspaper, magazine, blog, or TV or radio story." Instead, we can use that secondary source as a springboard for our own research. Many times, the actual study is available online for us to read. If, however, we can't access the original study ourselves, we can rely on scientists who we know we can trust to help us accurately interpret the information.
2. "Never trust a single source in isolation." We should see multiple studies and replicable //www.drcarney.com/data before we accept a new recommendation. "Don't fall for the pseudoexperts online who use single studies to prove points while ignoring the preponderance of evidence to the contrary . . . be suspicious of outlier //www.drcarney.com/data and demand replication of the findings in larger, well-designed studies."
3. "Consider the source." Ask what institution the authors are associated with and where they got funding for the study. Unfortunately, funding nearly always influences the outcome of a study. Pride, however, can be an even bigger influence than money. Sometimes when we examine the history of a particular scientist's publications, we discover a trend. The researcher has spent their career studying a single topic. "Once they establish 'guru' status, it's mighty hard for them to ever see evidence that disproves their beliefs." Dr. Davis reminds us that a researcher's history doesn't disqualify their research, but it should alert the reader to be aware of the potential for "pride bias" in the study.
4. "Consider the study design." The widely assumed "gold standard" of research is the randomized controlled clinical trial (RCCT). Randomized means that the study participants are randomly assigned to either experimental groups or control groups. Controlled refers to the inclusion of a group in the study which receives either no treatment or a variation of the main treatment. The control group ensures that the outcome of the study was in fact due to the specific treatment in question. Clinical simply means that the experiment was conducted in a clinical setting with medical professionals monitoring the patient progress and outcomes of the study. Trial means that the study was an experiment where something was done to a specific group of people; it was not merely an observational study.
An RCCT works very well for evaluating the effect of a single variable while controlling all other factors. However, real life isn't so easily controlled, and therefore, an RCCT doesn't work very well for studying the effects of diet. To see the effects of diet on diseases like heart attacks, cancer, and the onset of type 2 diabetes, subjects would have to be studied over months or years. Since that is impractical, RCCTs on diet usually last only a couple of months. Rather than looking for disease, researchers look for biomarkers which correlate with disease or which are predictive of disease, but these biomarkers don't equate with disease.
There is one other problem with RCCTs on diet, says Dr. Davis. "The more 'airtight' the experimental design, the less it resembles real life and the less applicable the results." Just because the test subjects were told to follow a certain diet doesn't mean that's what they went home and did. When reading the study, it's important to look at not only how well the study was designed but also at how well the subjects complied with the guidelines they were given. Their adherence -- or lack of it -- will determine the usefulness of the study's conclusions.
5. "Question the choices of the statistical analysis." Statistics can be useful because they can help us determine if a particular finding is a result of the intervention or is merely the result of random chance. They can help us know how much significance to attribute to a particular result.
Statistics can, however, be adjusted. The adjustments are often very useful because they help bring the results into better focus. For example, adjusting the results by excluding //www.drcarney.com/data from participants who did not strictly adhere to an assigned diet might be very useful. If researchers neglect to make that adjustment, the conclusions of the study might be misleading. When we examine a study, we should pay particular attention to adjustments made to account for "confounding variables." These types of adjustments often result in misleading conclusions. Dr. Davis gives an example from a Time magazine article. Time's bold cover that week said, "Eat Butter."
Most of the studies included in the meta-analyses cited by the Time magazine article on saturated fat committed this error by adjusting for serum cholesterol. The problem is, saturated fat causes heart disease in part by raising cholesterol. By removing people with high cholesterol from the trial, the researchers removed those people most susceptible to the heart-disease-causing effects of saturated fat. This is known as "overadjustment bias," and it's the easiest way to torture the //www.drcarney.com/data to get the results you want.
6. "Are they doing real science?" Scientists recognize that no study is perfect, and that even large, well-designed studies need to be replicated by others to to make sure that the outcomes didn't happen just by chance. Therefore, reputable scientists are transparent about their work, inviting others to examine their //www.drcarney.com/data and try to disprove their findings. Most scientific articles include a discussion section where the authors themselves discuss the shortcomings of their study and offer insights for improvement. Reading this section is crucial to understanding the significance of a study's results.
7. "Do the findings make sense in the real world?" When it comes to making recommendations for optimal dietary choices, it helps if the researcher is also a physician. According to Dr. Davis, nonphysician researchers sometimes "come up with hypotheses without appropriate context and reach conclusions without real-life application." Further, nonphysician researchers run the risk of not recognizing the real-life application of their findings.
As a physician, I applaud Dr. Davis' empowering his readers with these steps to evaluating medical research for themselves. I appreciate it when patients come into my office informed and ready to discuss how they can apply scientific research to optimize their health. In fact, many times I ask patients to engage in a little learning before they have an appointment with me. I have an education room stocked with DVDs in my office so that patients and prospective patients can see the science for themselves at no cost. I also keep books and DVDS available for purchase at my office so that the public has access to sound medical research on how diet can prevent, halt, and even reverse disease. I've made medical studies on diet and disease-prevention available for reading anytime at the "Show Me the Science" section here at DrCarney.com.
I enjoy helping educate my patients and answering their questions on how best to apply science to optimize their health. In my opinion, medicine is well-practiced when I can partner with my patients in their healthcare, each of us working to give the patient the very best outcome available to them. It all starts with understanding the science. The book Proteinaholic by Garth Davis, MD, with Howard Jacobson, PhD, is an excellent scientific resource not only for those who are confused by all the nutritional misinformation bandied about in our culture, but for anyone interested in what it takes to get healthy and stay that way. What we have reviewed is a very small summary of nine pages of this highly informative 330 page book solidly based on science. If you are interested to learn more, then: You know what to do!