Health and wellness are driven by evidence. Private, academic, governmental, and non-profit organizations need the most current evidence to make decisions, develop health claims, and drive policy. With an overwhelming amount of research published year after year, many opinion leaders rely on reviews to condense and organize the state of the evidence. We specialize in using literature review to help these stakeholders know that the evidence stands on their side. While we used to regularly conduct narrative reviews for these clients, we’ve left it behind in favor of structured review – here’s why.
click here to meet with our team about the best review for you
Narrative review intends to summarize the state of the research regarding a particular topic by telling the story of the evidence. However, as we have written (and read) dozens of reviews, we have become increasingly concerned about the methods – or, more precisely, lack thereof – that inform narrative review. There are consistent issues surrounding appropriate search depth, comparability between studies, and reducing opportunities for bias that are a genuine concern to anyone who truly wants to understand the evidence. These concerns have become a persistent topic of conversation, both internally and with clients. Our confidence in narrative review has eroded.
We fully believe that narrative reviews lack the structure needed to make objective claims about the state of the literature and are therefore fundamentally biased. Therefore, we have made a conscious decision to stop writing narrative reviews out of a commitment to provide the best quality product for our clients: unbiased, evidence-driven reviews geared toward the results-oriented needs of business.
Narrative reviews do not serve us, our clients, or the scientific community – so we stopped writing them. Here’s why we think you should too:
1. Narrative reviews are not replicable or verifiable.
Narrative reviews typically do not describe their methods for selecting and reviewing literature. Therefore, verifying or replicating their results is impossible. This puts narrative review at odds with scientific evidence in a fundamental way. The scientific method enables replication such that anyone can conduct the same work and achieve roughly the same result. However, since methods are not reported in narrative reviews, it is impossible to replicate the search.
Structured review provides the framework needed to improve replicability.
Similar to systematic review, structured reviews report search and inclusion criteria. This allows anyone reading the review to replicate the search, making the end result verifiable. This search transparency is crucial to reducing bias in research because it forces the reviewer to consider how they will report the search. It also allows readers to evaluate the search and inclusion criteria themselves. This allows them to identify biases when they do occur and assess the rest of the report accordingly.
2. Narrative review is prone to selection bias.
Narrative reviews are not required to include search details in the final publication. Therefore, readers have no evidence that the search methods were thorough and unbiased. Cherry-picking the publications that fit the narrative damages the review by introducing substantial bias that reinforce specific hypotheses instead of exploring the evidence landscape as it currently exists.
This is particularly true in research areas experiencing rapid growth. Authors are likely to review articles that rise to the top of search results or are included in journals they recognize. This can make even the most well-intentioned author prone to bias.
Cherry-picking can also happen by excluding publications from journals that the author does not deem quality without evaluating the publication itself. This can result in erroneous exclusions and bias the review. Of course, it is important to consider publication quality as part of the work. Without a rubric for deciding when studies are in- or out-of-scope, selection bias is hard to avoid.
Structured review leverages systematic methods to avoid selection bias.
The most effective method to avoid cherry-picking is to develop clear inclusion criteria and apply them rigorously (like the methods outlined in PRISMA flowcharts). Structured review does just this, applying the same criteria to every result of the search. This gives both readers and reviewers confidence that publications were not included or excluded without a justifiable cause.
Like systematic reviews, structured reviews allow for flexibility beyond using search engines. Some methods that we use include:
Reference scraping
Also known as backwards citation searching, reference scraping is used to identify studies for inclusion from the reference list of a given publication. This technique can help reviewers discover additional publications that meet inclusion criteria that may not have arisen in the original search. In addition to using search engines, we commonly scrape references from useful publications because it not only results in new inclusions, but also helps us trace the line of inquiry and understand the context of the research more completely.
Citation searching
Forward citation searches use similar methods as reference scraping, but move forward in time. This method identifies subsequent articles that cite a given publication, and provide an opportunity to understand the impact of the original article as well as any follow-up research that builds on the original conclusions.
Meta-review
When scraping references from reviews or meta-analyses, we call it “meta-review”. This is a slight twist on reference scraping. Typically, we start with reviews identified by our search and use them to find other publications of interest. We particularly like this method because it allows us to identify a key opinion leader and follow their thought process to a particular conclusion. Once we have done that, we are able to re-evaluate the primary literature to verify that we agree with their conclusion. This method is particularly useful when we are exploring a controversial hypothesis or looking to understand the true strength of a claim.
Tracing key opinion leaders
We are often able to identify a few key opinion leaders in our original searches. Typically, key opinion leaders have an extensive publication history, are frequently cited in others’ work, and are influential among their peers. Once we identify these key opinion leaders, we can trace their publication history to identify new possible inclusions. We can also use this publication history to understand the line of inquiry and contextualize our included research, including potential sources of bias, conflicts of interest, etc. Understanding an author’s career trajectory can often clarify the purpose behind their research.
3. Narrative reviews do not allow direct comparison between studies.
Narrative review uses written paragraphs to describe the results of the included research (hence the name). They do not conduct any pooled analyses using the data from the studies summarized. This prevents pooled analysis and therefore prevents true objectivity. Instead, narrative review functions as a pooled resource of the dominant opinions at the time of publication. This can be an adequate way to get a high-level understanding of a body of evidence. However, it does not provide any assurance that the dominant opinions are correct as it does not fully explore the alternative hypothesis.
Structured review makes it easy to put studies into context.
Structured review owes its name to the creation of a data structure behind the review. These reviews are accompanied by a database that standardizes and contextualizes the data from all included studies. This disciplined approach allows the reviewer to break down their hypotheses and research questions into their component parts.
This database allows the reviewer to ask important questions about the literature, such as:
- What study design(s) are more common? Is the evidence mostly cohort studies? Mostly randomized trials? How does this impact the conclusions?
- How many studies show positive relationships between the variable of interest and the outcome of interest? Negative? Null? How does this impact confidence in the literature?
- What is the geographical distribution of the research? Are we seeing a lot of research come out of certain countries?
- Who are the key opinion leaders in this field? Who is publishing the most research? Are their conclusions in line with the rest of the literature?
Structured review takes the quantitative approach of systematic review and breaks it down further to improve replicability. For example, if the goal of a systematic review was to identify how many double-blind, randomized, placebo-controlled trials were conducted, structured review would break this question down into three variables: blinding, randomization, and the use of a placebo. This makes data extraction simpler, as the reviewers only have to answer a series of basic yes/no questions, rather than categorize entire study designs. This simplification improves our confidence in the responses and enables accurate extraction with distributed teams while providing a more complete look at the body of research.
Bonus: structured review allows for powerful visualizations
The database produces an evidence landscape that lends itself to visualization (like the publication trends graph shown below). We love the ability to add visual elements to our reviews because they improve knowledge translation and communication with all the stakeholders in the review. Ultimately, this can make the evidence landscape accessible to more people. With visualization, it becomes easy to share the evidence with stakeholders that are not familiar with the entire landscape. This includes other business units in the organization or empowering consumers directly via social media. Structured review illuminates the evidence so that anyone can understand the take-home message.
Ultimately, structured review provides stronger evidence that is easier to understand, verify, and replicate.
While we believe there is a place for narrative review, we know better options exist for building a body of evidence. We rely on structured review because you rely on us to bring you strong evidence that can be used across your business and beyond.
How do you think structured review can help you? When you write your next review, what approach will you choose?
Learn more about how Traverse Science uses visualization
to make understanding the literature fast and easy