A 12-Step Program to Tell Good Science from Bad

Sharon Camp

Sharon Camp is the President and CEO of the Guttmacher Institute.

The very complexity of scientific studies can make them their own worst enemy. Valuable research is too often communicated in technical language and rigid formats that make it difficult for non-experts to interpret and evaluate the findings. Worse, some groups deliberately use outdated, incomplete, misleading and outright false information to further an ideological or religious agenda. This creates an environment in which it is increasingly difficult for the public and legislators to distinguish scientifically sound studies from agenda-driven junk science.

It needn't be that way. Social science research, with its focus on human behaviors, relationships and social institutions, can be a rich source of material for journalists, policymakers and program administrators. Indeed, social science findings have their greatest impact when they are useful to—and used by—groups that channel research into practice to improve people's lives.

Sharon Camp is the President and CEO of the Guttmacher Institute.

The very complexity of scientific studies can make them their own worst enemy. Valuable research is too often communicated in technical language and rigid formats that make it difficult for non-experts to interpret and evaluate the findings. Worse, some groups deliberately use outdated, incomplete, misleading and outright false information to further an ideological or religious agenda. This creates an environment in which it is increasingly difficult for the public and legislators to distinguish scientifically sound studies from agenda-driven junk science.

It needn't be that way. Social science research, with its focus on human behaviors, relationships and social institutions, can be a rich source of material for journalists, policymakers and program administrators. Indeed, social science findings have their greatest impact when they are useful to—and used by—groups that channel research into practice to improve people's lives.

The questions below (drawn from Guttmacher's "Interpreting Research Studies") are intended to help demystify social science research for those who could make use of the findings but lack specialized training in research methods. It identifies the key questions to ask when evaluating a research report and explains why the answers matter.

Like This Story?

Your $10 tax-deductible contribution helps support our research, reporting, and analysis.

Donate Now

  1. What makes the study important?
    A study's importance or newsworthiness depends on how it contributes to what we already know. Does the study answer a previously unaddressed question? Does it address an old question in a new way or with surprising results? Reading through the abstract or executive summary with these questions in mind can help you evaluate the study's relevance even before you review the full publication.
  2. Do the findings make sense?
    Do study's key "findings" or "results" make sense, given what you already know about the subject? And are they rooted in the existing body of research? A scientific report should be properly referenced, with original sources for all factual statements and data from other research clearly cited.
  3. Who conducted the research and wrote the report?
    It is important to consider whether the study results could be influenced by a researcher's conflict of interest. Are the authors well regarded in the scientific community? What are their professional credentials? Could their work have been influenced by those who employed or funded them? Any potential conflict of interest should be identified up front. That said, good researchers committed to a political or social agenda can still conduct unbiased, trustworthy studies that can withstand independent evaluation, provided they follow practices designed to protect the quality and integrity of research.
  4. Who published the report?
    An article published in a peer-reviewed journal has been evaluated by experts in the field to help ensure that it meets high scientific standards. The prestige of the journal is one indication of a study's quality. While studies from sources other than journals (including reports that research institutions publish themselves) may also contain solid, useful information, if an external review process is not mentioned, you should be more cautious about accepting the study's conclusions.
  5. Did the researcher select an appropriate group for study?
    A social scientist's work is about people. In practical terms a study often focuses on a subset, or sample, of the larger population. This sample must be selected carefully to ensure that the study results are applicable to the relevant general population. Using a representative sample is the best way to ensure that findings can be generalized to all members of the target population. Other common approaches are acceptable and—with appropriate statistical adjustments for weighting—can produce valid and representative results. Sometimes, however, a researcher may have good reasons to select the target population in a different way. When they do, they should explain their reasons, and you should consider the extent to which their findings are applicable to other groups.
  6. If comparison groups are used, how similar are they?
    If a study compares two or more groups (to evaluate the effects of an intervention, for example), the results will be valid only if the groups are similar in all ways other than their exposure to the intervention being studied. Any preexisting differences between the groups could account for different outcomes. In the best study designs, participants are randomly assigned to the study groups. But when differences do exist between the groups, researchers can use statistical techniques to control for differences. Experience and common sense can help determine whether the differences among them are important for the study.
  7. What has changed since the information was collected?
    Ideally, the data used in a study will have been collected recently so that the information reflects the current situation. However, because national-level surveys can be quite expensive and time consuming, data may not become public for several years and special analyses may require additional time. For example, data from the large National Survey of Family Growth, which was conducted in 2002, became public only in late 2004, and analyses are still ongoing. It is important to consider how any changes that have occurred in the intervening period, such as new policies, could affect the outcomes today.
  8. Are the methods appropriate to the research purpose?
    Social science studies can rely on either qualitative or quantitative methods or a combination of the two. As a rule, quantitative techniques (collecting and analyzing measurements such as whether a person is currently using a contraceptive method, etc.) are best for answering questions such as "How much?" "How many?" "How often?" or "When?" Quantitative studies can also indicate important relationships, such as whether poor women are more likely than better-off women to have more children than they want. Qualitative research (recording and analyzing interactions with people through techniques such as in-depth interviews or focus groups) may be more useful in obtaining a better understanding of complex contextual, attitudinal or behavioral issues or documenting a process.
  9. Does the study establish causation?
    Often, the goal of a study is to determine the effect of something: for example, a program, medication or policy. However, it is usually difficult to isolate the effects of one discrete factor from all the other things going on in people's lives. Even if the study shows that a particular outcome occurred after a program got under way, it can be difficult to prove that this intervention caused the outcome. In general, studies can prove only that an outcome is "associated with" or "correlated with" (rather than "caused by") an intervention. Be alert to researchers who make claims about cause and effect that seem dubious or who ignore other possible explanations for their findings.
  10. Is the time frame long enough to identify an impact?
    Studies can either follow their subjects over time, checking in with them at various intervals (a longitudinal study), or take a "snapshot" of subjects at a single moment in time (a cross-sectional study). A cross-sectional study is good for comparing groups, while a series of cross-sectional studies conducted within the same general population (but selecting a different group of people each time) can also provide information on trends over time, as long as the groups sampled are truly comparable. Because a longitudinal study follows the same group of individuals over time, it can be better for examining the effects of a particular intervention, as long as it allows enough time for adequate follow-up and is able to retain a sufficient number of participants.
  11. Could the data be biased as a result of poor research design?
    The wording and order of questions in a poll or survey can affect the answers participants provide. In addition, a low response rate (say, fewer than 70% of those selected), suggests that the results may be biased because the people who participated are not representative of the target group as a whole. Studies of sexual and reproductive behavior face another hurdle. Participants do not always answer sensitive questions truthfully. For example, adolescent boys tend to overreport sexual activity, while adolescent girls tend to underreport it.
  12. Are the results statistically significant?
    When a quantitative study uses a sample, it is important to determine mathematically that there is little probability the result could have occurred by chance—that is, that a different sample could have produced other results. In the social sciences, a study finding is generally considered statistically significant if there is no more than a 5% probability that it could have occurred by chance (often expressed as a "p-value" of 0.05 or less). Statistical significance alone is not enough to prove cause and effect, but it lends credibility to an argument.

The answers to these 12 questions should help you evaluate and interpret reports of research findings. Of course, a study may be flawlessly designed, conducted without bias, appropriately analyzed and statistically significant, yet convey nothing important to you. But if the findings are something that you care about, and you believe that the research is sound, you are in a position to play a critical role in social science research—interpreting the findings and transmitting them to the wider world to have a greater impact.

"Interpreting Research Studies," on which this blog post is based, was written by Jennifer Nadeau and Sharon Camp and shaped by the valuable input of many Guttmacher colleagues and partners.

Roundups Sexual Health

This Week in Sex: The Sexually Transmitted Infections Edition

Martha Kempner

A new Zika case suggests the virus can be transmitted from an infected woman to a male partner. And, in other news, HPV-related cancers are on the rise, and an experimental chlamydia vaccine shows signs of promise.

This Week in Sex is a weekly summary of news and research related to sexual behavior, sexuality education, contraception, STIs, and more.

Zika May Have Been Sexually Transmitted From a Woman to Her Male Partner

A new case suggests that males may be infected with the Zika virus through unprotected sex with female partners. Researchers have known for a while that men can infect their partners through penetrative sexual intercourse, but this is the first suspected case of sexual transmission from a woman.

The case involves a New York City woman who is in her early 20s and traveled to a country with high rates of the mosquito-borne virus (her name and the specific country where she traveled have not been released). The woman, who experienced stomach cramps and a headache while waiting for her flight back to New York, reported one act of sexual intercourse without a condom the day she returned from her trip. The following day, her symptoms became worse and included fever, fatigue, a rash, and tingling in her hands and feet. Two days later, she visited her primary-care provider and tests confirmed she had the Zika virus.

Like This Story?

Your $10 tax-deductible contribution helps support our research, reporting, and analysis.

Donate Now

A few days after that (seven days after intercourse), her male partner, also in his 20s, began feeling similar symptoms. He had a rash, a fever, and also conjunctivitis (pink eye). He, too, was diagnosed with Zika. After meeting with him, public health officials in the New York City confirmed that he had not traveled out of the country nor had he been recently bit by a mosquito. This leaves sexual transmission from his partner as the most likely cause of his infection, though further tests are being done.

The Centers for Disease Control and Prevention (CDC)’s recommendations for preventing Zika have been based on the assumption that virus was spread from a male to a receptive partner. Therefore the recommendations had been that pregnant women whose male partners had traveled or lived in a place where Zika virus is spreading use condoms or abstain from sex during the pregnancy. For those couples for whom pregnancy is not an issue, the CDC recommended that men who had traveled to countries with Zika outbreaks and had symptoms of the virus, use condoms or abstain from sex for six months after their trip. It also suggested that men who traveled but don’t have symptoms use condoms for at least eight weeks.

Based on this case—the first to suggest female-to-male transmission—the CDC may extend these recommendations to couples in which a female traveled to a country with an outbreak.

More Signs of Gonorrhea’s Growing Antibiotic Resistance

Last week, the CDC released new data on gonorrhea and warned once again that the bacteria that causes this common sexually transmitted infection (STI) is becoming resistant to the antibiotics used to treat it.

There are about 350,000 cases of gonorrhea reported each year, but it is estimated that 800,000 cases really occur with many going undiagnosed and untreated. Once easily treatable with antibiotics, the bacteria Neisseria gonorrhoeae has steadily gained resistance to whole classes of antibiotics over the decades. By the 1980s, penicillin no longer worked to treat it, and in 2007 the CDC stopped recommending the use of fluoroquinolones. Now, cephalosporins are the only class of drugs that work. The recommended treatment involves a combination of ceftriaxone (an injectable cephalosporin) and azithromycin (an oral antibiotic).

Unfortunately, the data released last week—which comes from analysis of more than 5,000 samples of gonorrhea (called isolates) collected from STI clinics across the country—shows that the bacteria is developing resistance to these drugs as well. In fact, the percentage of gonorrhea isolates with decreased susceptibility to azithromycin increased more than 300 percent between 2013 and 2014 (from 0.6 percent to 2.5 percent).

Though no cases of treatment failure has been reported in the United States, this is a troubling sign of what may be coming. Dr. Gail Bolan, director of CDC’s Division of STD Prevention, said in a press release: “It is unclear how long the combination therapy of azithromycin and ceftriaxone will be effective if the increases in resistance persists. We need to push forward on multiple fronts to ensure we can continue offering successful treatment to those who need it.”

HPV-Related Cancers Up Despite Vaccine 

The CDC also released new data this month showing an increase in HPV-associated cancers between 2008 and 2012 compared with the previous five-year period. HPV or human papillomavirus is an extremely common sexually transmitted infection. In fact, HPV is so common that the CDC believes most sexually active adults will get it at some point in their lives. Many cases of HPV clear spontaneously with no medical intervention, but certain types of the virus cause cancer of the cervix, vulva, penis, anus, mouth, and neck.

The CDC’s new data suggests that an average of 38,793 HPV-associated cancers were diagnosed each year between 2008 and 2012. This is a 17 percent increase from about 33,000 each year between 2004 and 2008. This is a particularly unfortunate trend given that the newest available vaccine—Gardasil 9—can prevent the types of HPV most often linked to cancer. In fact, researchers estimated that the majority of cancers found in the recent data (about 28,000 each year) were caused by types of the virus that could be prevented by the vaccine.

Unfortunately, as Rewire has reported, the vaccine is often mired in controversy and far fewer young people have received it than get most other recommended vaccines. In 2014, only 40 percent of girls and 22 percent of boys ages 13 to 17 had received all three recommended doses of the vaccine. In comparison, nearly 80 percent of young people in this age group had received the vaccine that protects against meningitis.

In response to the newest data, Dr. Electra Paskett, co-director of the Cancer Control Research Program at the Ohio State University Comprehensive Cancer Center, told HealthDay:

In order to increase HPV vaccination rates, we must change the perception of the HPV vaccine from something that prevents a sexually transmitted disease to a vaccine that prevents cancer. Every parent should ask the question: If there was a vaccine I could give my child that would prevent them from developing six different cancers, would I give it to them? The answer would be a resounding yes—and we would have a dramatic decrease in HPV-related cancers across the globe.

Making Inroads Toward a Chlamydia Vaccine

An article published in the journal Vaccine shows that researchers have made progress with a new vaccine to prevent chlamydia. According to lead researcher David Bulir of the M. G. DeGroote Institute for Infectious Disease Research at Canada’s McMaster University, efforts to create a vaccine have been underway for decades, but this is the first formulation to show success.

In 2014, there were 1.4 million reported cases of chlamydia in the United States. While this bacterial infection can be easily treated with antibiotics, it often goes undiagnosed because many people show no symptoms. Untreated chlamydia can lead to pelvic inflammatory disease, which can leave scar tissue in the fallopian tubes or uterus and ultimately result in infertility.

The experimental vaccine was created by Canadian researchers who used pieces of the bacteria that causes chlamydia to form an antigen they called BD584. The hope was that the antigen could prompt the body’s immune system to fight the chlamydia bacteria if exposed to it.

Researchers gave BD584 to mice using a nasal spray, and then exposed them to chlamydia. The results were very promising. The mice who received the spray cleared the infection faster than the mice who did not. Moreover, the mice given the nasal spray were less likely to show symptoms of infection, such as bacterial shedding from the vagina or fluid blockages of the fallopian tubes.

There are many steps to go before this vaccine could become available. The researchers need to test it on other strains of the bacteria and in other animals before testing it in humans. And, of course, experience with the HPV vaccine shows that there’s work to be done to make sure people get vaccines that prevent STIs even after they’re invented. Nonetheless, a vaccine to prevent chlamydia would be a great victory in our ongoing fight against STIs and their health consequences, and we here at This Week in Sex are happy to end on a bit of a positive note.

Analysis Politics

The 2016 Republican Platform Is Riddled With Conservative Abortion Myths

Ally Boguhn

Anti-choice activists and leaders have embraced the Republican platform, which relies on a series of falsehoods about reproductive health care.

Republicans voted to ratify their 2016 platform this week, codifying what many deem one of the most extreme platforms ever accepted by the party.

“Platforms are traditionally written by and for the party faithful and largely ignored by everyone else,” wrote the New York Times‘ editorial board Monday. “But this year, the Republicans are putting out an agenda that demands notice.”

“It is as though, rather than trying to reconcile Mr. Trump’s heretical views with conservative orthodoxy, the writers of the platform simply opted to go with the most extreme version of every position,” it continued. “Tailored to Mr. Trump’s impulsive bluster, this document lays bare just how much the G.O.P. is driven by a regressive, extremist inner core.”

Tucked away in the 66-page document accepted by Republicans as their official guide to “the Party’s principles and policies” are countless resolutions that seem to back up the Times‘ assertion that the platform is “the most extreme” ever put forth by the party, including: rolling back marriage equalitydeclaring pornography a “public health crisis”; and codifying the Hyde Amendment to permanently block federal funding for abortion.

Like This Story?

Your $10 tax-deductible contribution helps support our research, reporting, and analysis.

Donate Now

Anti-choice activists and leaders have embraced the platform, which the Susan B. Anthony List deemed the “Most Pro-life Platform Ever” in a press release upon the GOP’s Monday vote at the convention. “The Republican platform has always been strong when it comes to protecting unborn children, their mothers, and the conscience rights of pro-life Americans,” said the organization’s president, Marjorie Dannenfelser, in a statement. “The platform ratified today takes that stand from good to great.”  

Operation Rescue, an organization known for its radical tactics and links to violence, similarly declared the platform a “victory,” noting its inclusion of so-called personhood language, which could ban abortion and many forms of contraception. “We are celebrating today on the streets of Cleveland. We got everything we have asked for in the party platform,” said Troy Newman, president of Operation Rescue, in a statement posted to the group’s website.

But what stands out most in the Republicans’ document is the series of falsehoods and myths relied upon to push their conservative agenda. Here are just a few of the most egregious pieces of misinformation about abortion to be found within the pages of the 2016 platform:

Myth #1: Planned Parenthood Profits From Fetal Tissue Donations

Featured in multiple sections of the Republican platform is the tired and repeatedly debunked claim that Planned Parenthood profits from fetal tissue donations. In the subsection on “protecting human life,” the platform says:

We oppose the use of public funds to perform or promote abortion or to fund organizations, like Planned Parenthood, so long as they provide or refer for elective abortions or sell fetal body parts rather than provide healthcare. We urge all states and Congress to make it a crime to acquire, transfer, or sell fetal tissues from elective abortions for research, and we call on Congress to enact a ban on any sale of fetal body parts. In the meantime, we call on Congress to ban the practice of misleading women on so-called fetal harvesting consent forms, a fact revealed by a 2015 investigation. We will not fund or subsidize healthcare that includes abortion coverage.

Later in the document, under a section titled “Preserving Medicare and Medicaid,” the platform again asserts that abortion providers are selling “the body parts of aborted children”—presumably again referring to the controversy surrounding Planned Parenthood:

We respect the states’ authority and flexibility to exclude abortion providers from federal programs such as Medicaid and other healthcare and family planning programs so long as they continue to perform or refer for elective abortions or sell the body parts of aborted children.

The platform appears to reference the widely discredited videos produced by anti-choice organization Center for Medical Progress (CMP) as part of its smear campaign against Planned Parenthood. The videos were deceptively edited, as Rewire has extensively reported. CMP’s leader David Daleiden is currently under federal indictment for tampering with government documents in connection with obtaining the footage. Republicans have nonetheless steadfastly clung to the group’s claims in an effort to block access to reproductive health care.

Since CMP began releasing its videos last year, 13 state and three congressional inquiries into allegations based on the videos have turned up no evidence of wrongdoing on behalf of Planned Parenthood.

Dawn Laguens, executive vice president of Planned Parenthood Action Fund—which has endorsed Hillary Clinton—called the Republicans’ inclusion of CMP’s allegation in their platform “despicable” in a statement to the Huffington Post. “This isn’t just an attack on Planned Parenthood health centers,” said Laguens. “It’s an attack on the millions of patients who rely on Planned Parenthood each year for basic health care. It’s an attack on the brave doctors and nurses who have been facing down violent rhetoric and threats just to provide people with cancer screenings, birth control, and well-woman exams.”

Myth #2: The Supreme Court Struck Down “Commonsense” Laws About “Basic Health and Safety” in Whole Woman’s Health v. Hellerstedt

In the section focusing on the party’s opposition to abortion, the GOP’s platform also reaffirms their commitment to targeted regulation of abortion providers (TRAP) laws. According to the platform:

We salute the many states that now protect women and girls through laws requiring informed consent, parental consent, waiting periods, and clinic regulation. We condemn the Supreme Court’s activist decision in Whole Woman’s Health v. Hellerstedt striking down commonsense Texas laws providing for basic health and safety standards in abortion clinics.

The idea that TRAP laws, such as those struck down by the recent Supreme Court decision in Whole Woman’s Health, are solely for protecting women and keeping them safe is just as common among conservatives as it is false. However, as Rewire explained when Paul Ryan agreed with a nearly identical claim last week about Texas’ clinic regulations, “the provisions of the law in question were not about keeping anybody safe”:

As Justice Stephen Breyer noted in the opinion declaring them unconstitutional, “When directly asked at oral argument whether Texas knew of a single instance in which the new requirement would have helped even one woman obtain better treatment, Texas admitted that there was no evidence in the record of such a case.”

All the provisions actually did, according to Breyer on behalf of the Court majority, was put “a substantial obstacle in the path of women seeking a previability abortion,” and “constitute an undue burden on abortion access.”

Myth #3: 20-Week Abortion Bans Are Justified By “Current Medical Research” Suggesting That Is When a Fetus Can Feel Pain

The platform went on to point to Republicans’ Pain-Capable Unborn Child Protection Act, a piece of anti-choice legislation already passed in several states that, if approved in Congress, would create a federal ban on abortion after 20 weeks based on junk science claiming fetuses can feel pain at that point in pregnancy:

Over a dozen states have passed Pain-Capable Unborn Child Protection Acts prohibiting abortion after twenty weeks, the point at which current medical research shows that unborn babies can feel excruciating pain during abortions, and we call on Congress to enact the federal version.

Major medical groups and experts, however, agree that a fetus has not developed to the point where it can feel pain until the third trimester. According to a 2013 letter from the American Congress of Obstetricians and Gynecologists, “A rigorous 2005 scientific review of evidence published in the Journal of the American Medical Association (JAMA) concluded that fetal perception of pain is unlikely before the third trimester,” which begins around the 28th week of pregnancy. A 2010 review of the scientific evidence on the issue conducted by the British Royal College of Obstetricians and Gynaecologists similarly found “that the fetus cannot experience pain in any sense prior” to 24 weeks’ gestation.

Doctors who testify otherwise often have a history of anti-choice activism. For example, a letter read aloud during a debate over West Virginia’s ultimately failed 20-week abortion ban was drafted by Dr. Byron Calhoun, who was caught lying about the number of abortion-related complications he saw in Charleston.

Myth #4: Abortion “Endangers the Health and Well-being of Women”

In an apparent effort to criticize the Affordable Care Act for promoting “the notion of abortion as healthcare,” the platform baselessly claimed that abortion “endangers the health and well-being” of those who receive care:

Through Obamacare, the current Administration has promoted the notion of abortion as healthcare. We, however, affirm the dignity of women by protecting the sanctity of human life. Numerous studies have shown that abortion endangers the health and well-being of women, and we stand firmly against it.

Scientific evidence overwhelmingly supports the conclusion that abortion is safe. Research shows that a first-trimester abortion carries less than 0.05 percent risk of major complications, according to the Guttmacher Institute, and “pose[s] virtually no long-term risk of problems such as infertility, ectopic pregnancy, spontaneous abortion (miscarriage) or birth defect, and little or no risk of preterm or low-birth-weight deliveries.”

There is similarly no evidence to back up the GOP’s claim that abortion endangers the well-being of women. A 2008 study from the American Psychological Association’s Task Force on Mental Health and Abortion, an expansive analysis on current research regarding the issue, found that while those who have an abortion may experience a variety of feelings, “no evidence sufficient to support the claim that an observed association between abortion history and mental health was caused by the abortion per se, as opposed to other factors.”

As is the case for many of the anti-abortion myths perpetuated within the platform, many of the so-called experts who claim there is a link between abortion and mental illness are discredited anti-choice activists.

Myth #5: Mifepristone, a Drug Used for Medical Abortions, Is “Dangerous”

Both anti-choice activists and conservative Republicans have been vocal opponents of the Food and Drug Administration (FDA’s) March update to the regulations for mifepristone, a drug also known as Mifeprex and RU-486 that is used in medication abortions. However, in this year’s platform, the GOP goes a step further to claim that both the drug and its general approval by the FDA are “dangerous”:

We believe the FDA’s approval of Mifeprex, a dangerous abortifacient formerly known as RU-486, threatens women’s health, as does the agency’s endorsement of over-the-counter sales of powerful contraceptives without a physician’s recommendation. We support cutting federal and state funding for entities that endanger women’s health by performing abortions in a manner inconsistent with federal or state law.

Studies, however, have overwhelmingly found mifepristone to be safe. In fact, the Association of Reproductive Health Professionals says mifepristone “is safer than acetaminophen,” aspirin, and Viagra. When the FDA conducted a 2011 post-market study of those who have used the drug since it was approved by the agency, they found that more than 1.5 million women in the U.S. had used it to end a pregnancy, only 2,200 of whom had experienced an “adverse event” after.

The platform also appears to reference the FDA’s approval of making emergency contraception such as Plan B available over the counter, claiming that it too is a threat to women’s health. However, studies show that emergency contraception is safe and effective at preventing pregnancy. According to the World Health Organization, side effects are “uncommon and generally mild.”