A Primer on Conducting a Systematic Review of Evidence

The AAP has finally agreed to conduct a systematic review of the evidence for “gender-affirming care.” Here’s how they can ensure it’s done right.

Image Credit: Sezeryadigar on iStock

Reality’s Last Stand is a reader-supported publication. Please consider becoming a paying subscriber or making a one-time or recurring donation to show your support.

This article was originally published on the Society for Evidence-Based Gender Medicine’s website on August 6, 2023.

Last week, the American Academy of Pediatrics (AAP) announced that they would update their treatment recommendations for gender dysphoric youth based on the results of a systematic review of evidence the AAP plans to commission. We welcome the AAP’s decision, as this is an important step on the path to creating evidence-based treatment recommendations for managing youth gender dysphoria in the United States and Canada. However, we are concerned by the AAP’s choice to reaffirm its current policy for treating minors with puberty blockers, hormones and surgery pending the yet-to-be commissioned systematic review, and by the lengthy timeline for any updates to the policy this approach implies.

Systematic reviews of evidence, when performed correctly (the steps are outlined below), involve a rigorous process that typically takes a year or more to complete. Once a systematic review of evidence has been completed, updating treatment guidelines using a credible process takes several more months to a year. This suggests that an update won’t be available for 1-2 years or more. In the meantime, tens of thousands of youth will continue to be treated according to AAP’s non-evidence-based recommendations, exposing them to potentially non-beneficial or even harmful interventions.

According to the principles of evidence-based decision-making, when making treatment recommendations, one has to leverage the best available evidence, which is typically found in systematic reviews of evidence. Several systematic reviews have recently been completed by European public health authorities; three of these reviews are available in English. These independent systematic reviews of evidence found the practice of youth gender transition to either not be clearly beneficial, or net-harmful. It is incumbent on the AAP to leverage these systematic evidence reviews to urgently issue an interim update to its 2018 policy. In parallel with the the interim update, the AAP can initiate and complete its own rigorous systematic evidence review.

In addition to the unreasonably lengthy timeline for the urgently-needed updates implied by the AAP’s announcement, we find several statements in the AAP press release concerning. One such statement is the assertion that “policy authors and AAP leadership are confident the principles presented in the original policy, Ensuring Comprehensive Care and Support for Transgender and Gender-Diverse Children and Adolescents, remain in the best interest of children.” Not only is it inappropriate to start a research process with a pre-ordained conclusion, but this conclusion also contradicts the findings of the current systematic reviews of evidence. 

Another concerning statement is that “the decision to authorize a systematic review reflects the board’s concerns about restrictions to access to health care with bans on gender-affirming care in more than 20 states.” Conducting a systematic review of evidence to influence a political process, rather than to pursue knowledge about the outcomes of a medical intervention, distorts the purpose of evidence-based medicine and creates the appearance of a compromised process.

We call on the AAP leadership to issue an interim update to its 2018 policy based on the systematic reviews already conducted by public health authorities in Europe. For the process to be credible, the AAP should include a range of stakeholders with diverse views on the topic, from those who support gender transition of minors, to those who have voiced concerns about the practice. Once the immediate gap in the AAP’s current guidelines is addressed via an interim update, the AAP can embark on its own independent comprehensive systematic review of evidence, after which it can issue a final policy update.

The steps below comprise a quality systematic review of evidence.

Select the review team

The team should consist of experts in evidence appraisal and experts in the subject matter, including experts in mental health, hormone treatments, and surgery. The experts should disclose all potential conflicts of interest, including intellectual, professional, and financial conflicts. The Users’ Guide to Medical Literature, which is the key publication that outlines the principles of evidence-based medicine, provides this helpful overview of the importance of minimizing conflicts of interest:

Since it is unlikely that all conflicts of interest can be eliminated, they should be managed by including stakeholders with diverse and opposing views. SEGM would be happy to participate should the AAP seek partners who hold a position that differs from its own, while sharing the AAP’s stated commitment to evidence-based medicine.

Define key questions

A systematic review should answer specific clinical questions, which are typically written in a form of PICO: “population,” “intervention,” “comparator,” and “outcome.” Each of these will require extensive discussion.

For example, the population of adolescents with post-pubertal gender dysphoria may need to be analyzed separately from those with early childhood-onset of gender dysphoria; youth with neurocognitive conditions may need to be considered separately from youth without such comorbidities.

Further, primary outcomes of interest must be carefully selected. Originally, youth transitions were justified based on the expectation that undergoing medical interventions during adolescence, as compared to mature adulthood, would improve long-term mental health, overall functioning, and reduce suicides. However, more recently, some proponents of gender-affirming interventions have asserted that the goal is treatment is patient satisfaction or achievement of one’s embodiment goals.

While patient satisfaction is important in healthcare, given the irreversible nature of gender-affirming interventions, SEGM recommends selecting outcomes that enable assessment of the original premise of gender-transition of youth. 

Finally, it is important for systematic reviews to look at both the potential benefits and harms, as opposed to looking only at the potential benefits of an intervention. The potential harms include infertility/sterility, adverse effects of puberty blockers and cross-sex hormones on bone and cardiovascular health, and cognitive function.

Establish study eligibility criteria

Systematic reviews should specify eligibility criteria for inclusion of studies into the review. The criteria should consider patient age in the study (e.g., whether to include studies of adults) and the types of publications to (e.g., pre-prints vs only peer-reviewed studies). However, the most critical decision is which types of study designs will be included in a systematic review. Since no randomized controlled trials exist in this area of medicine, the review team will need to determine which lower quality study types to include or exclude.

Conduct literature search and select eligible studies

Unlike the “narrative review” used by the AAP previously, which allows for cherry-picking studies, a systematic review must follow a pre-specified search strategy. The search for evidence must be comprehensive, including searches of general and specific electronic databases, as well as other sources such as trials registries and “gray literature.” The search strategy must be documented such that it can be reproduced by other researchers.

Following the established eligibility criteria, at least two independent reviewers should check each reference title and abstract, and then full text for eligibility. Studies should be included even when they report only one of the outcomes of interest and meet all other eligibility criteria.

Perform data abstraction

Once the eligible studies are identified, pairs of independent reviewers should abstract all relevant data from each study. This is done using standardized and piloted forms. The exact same data are abstracted from all studies. Most of the information to be extracted is contained in the Methods and Results sections, and their supplementary material. Only very rarely do systematic reviews abstract any data from the Discussion section and systematic reviews should never rely on information presented in the Conclusions section.

Conduct risk of bias assessment

The assessment of studies for trustworthiness is a key step that differentiates systematic reviews from other types of “evidence reviews.” Pairs of independent reviewers assess the risk of bias of each study and each outcome in the study, using appropriate tools that comprehensively include all factors, based on theoretical considerations or empirical evidence, that could bias results. Examples of such tools include the Cochrane-endorsed RoB 2.0 tool for randomized trials and the ROBINS-I tool for non-randomized studies. This critical step helps to determine the credibility of each individual study result.

Perform data synthesis

Results across studies should be synthesized in a reproducible manner, following outlined methods. At this stage, the reviewers start working at an outcome level; that is, they perform a synthesis per outcome. Many times, reviewers also conduct meta-analyses, but in other cases reviewers perform a narrative synthesis done at each outcome level. This is the stage at which the AAP should also consider additional analyses, such as analyzing the results separately for youth with childhood vs. adolescent-onset gender dysphoria.

Conduct certainty / quality of evidence assessment

A systematic review culminates with the assessment of the certainty of the evidence, which is performed at each outcome level across all studies. It is not enough to know whether a certain intervention helps or harm and the estimate of the effect—it is critical to know how much one can trust this information. The assessment of the quality/certainty is done by formally considering the potential limitations of the evidence. This is done through the assessment of classes of evidence (CoE) using Grading of Recommendations Assessment, Development and Evaluation (GRADE).

Report and interpret results

Reviewers must report their findings for each outcome consistently. Specifically, for each outcome the reviewers need to report the number of studies, patients, the effects estimates (relative and absolute), and the certainty of the evidence. Interpretation of the outcomes should be done using standardized methods that consider both the effect and its certainty.

For more information for how to conduct and report a quality systematic review, see the Cochrane Handbook for Systematic Reviews of Interventions.

Reality’s Last Stand is 100% reader-supported. If you enjoyed this article, please consider upgrading to a paid subscription or making a recurring or one-time donation below. Your support is greatly appreciated.