Skip to main content

Systematic Reviews: Systematic Review Process

A guide to finding resources for a systematic review in the health and medical sciences

Steps in Developing a Systematic Review

  • Define the question - a clearly defined question will ensure that your research produces relevant results.
  • Define the protocol, which includes the inclusion/exclusion and eligibility criteria. The protocol defines the process for selecting studies and reduces the risk of bias.
  • Develop the search strategy.
  • Identify any recent or ongoing systematic reviews.
  • Search relevant sources to identify the evidence.
  • Appraise and select suitable studies.
  • Synthesise the data.
  • Document the search process.

Defining the Question using PICO

The PICO structure can be used to help you put together a search strategy and formulate the question:

      Participants, Patient or Population       

      Intervention(s) (therapy, treatment, etc.)

      Comparison (other intervention or treatment, no treatment, etc. It's not always necessary to have a comparison group)

      Outcome(s)

In some cases the review question may also include the Study Design (PICOS). This is outlined in the Centre for Reviews and Dissemination, University of York, guide “Systematic reviews: CRD's guidance for undertaking reviews in health care”:

“The review question can be framed in terms of the population, intervention(s), comparator(s) and outcomes of the studies that will be included in the review. These elements of the review question, together with study design, will then be refined in order to determine the specific inclusion criteria that will be used when selecting studies for the review.”

"Not every review question will specify type of study design to be included". See Levels of Evidence page for hierarchy of study design.

Finding the Evidence

It is important when searching for evidence that search terms are referred back to your original PICO question. The process of finding evidence follows these steps: 

1. Identify terms to fit your PICO question. These keywords will be used in searching databases. Check thesaurus terms in the relevant databases to identify other relevant keywords or subject terms to include in your search. Be aware of differences in American and English spelling and terminology. Thesaurus terms may also vary between databases.

2. Find systematic reviews - it's always necessary to check whether a systematic review answering your question has already been conducted or is under way. Published reviews also provide a starting point for identifying the studies.

3. Find journal articles - search for published primary studies in databases such as MEDLINE, CINAHL, PsycINFO. Citation searching in Scopus or Web of Science, allows you to follow a research trail forwards, backwards or to related research.

4. Search the Grey Literature,  such as conference proceedings, theses, reports and unpublished literature.

5. Hand searching involves examining manually key journals, conference proceedings and other relevant publications. Hand searching is to overcome deficiencies in indexing or database coverage. The citation databases, Web of Science and SCOPUS are useful for identifying key journals and authors, as well as tracking research and citation searching.

6. Appraisal and selection of studies. Structured appraisal helps to select the highest quality evidence available and minimise bias.  

7. Synthesis of study results. Data from each individual study needs to be collated, combined and summarised. Quantitative systematic reviews use formal statisitical techniques such as meta-analysis to perform this step.

"As well as drawing results together, synthesis should consider the strength of evidence, explore whether any observed effects are consistent across studies, and investigate possible reasons for any inconsistencies" (Centre for Reviews and Dissemination, 2009, section 1.3.5). Systematic reviews: CRD's guidance for undertaking reviews in health care  

8. Report on all steps of the systematic review process and present results. See PRISMA for further information. Links to reporting guidelines for systematic reviews (PRISMA) and other study types are available on the Equator Network website.

Documenting the Search Process

Document the search process:

The search process needs to be documented in enough detail to ensure that it can be reported correctly in the review and reproduced for verification.

For each database search record:

  • Database searched
  • Database provider (e.g. EBSCO)
  • Search strategy - keywords used and how these were combined in the search
  • Years searched
  • Date search was run
  • Any filters used
  • Number of studies identified

EndNote software can be used to record full bibliographical details for each citation and additional notes relating to the selection and evaluation of that  source.

Report search results:

There are a number of places where searches can be reported. These include the appendix, the review abstract, the methods section or the results section.

 

Protocol

Why is it important to have a plan?

A major cause of bias in a systematic review is answering a different question to that being originally asked. This is why it is important to develop a review plan or protocol.

The benefits of having a protocol before the beginning of a review:

  • relate to the validity and merit of a research process that reduces risk of bias
  • promotes a systematic rather than ad hoc approach to the review process
  • facilitates communication with others and promotes consistency between review team members
  • support the reliability and usefulness of reviews to health professionals
     

The protocol should include:

  • Review question/objective
  • Inclusion/exclusion criteria (scope including types of studies, participants, interventions)
  • Search strategy
  • Methodology
  • Declaration of interests

How to make your protocol visible

Registering your protocol helps avoid duplication of work. You do not want anyone else to do the exact same review you are doing.

A good place to register a health review is PROSPERO. Once you register your review will:

  • be available open access through the PROSPERO database.
     
  • have a unique registration number. This number can be cited in publications and reports to provide the link between your planned and completed review. This is recommended by PRISMA (2009) and many publishers.

You can publish your protocol. For example, in the Systematic Reviews journal:

Protocols for Cochrane and Joanna Briggs Institute reviews are published on their websites:

Many other journals will allow you to publish your protocol. You should also look for relevant journals within your discipline area.

Loading

PICo, SPICE or SPIDER for qualitative studies

PICo

P:    Population or Problem of interest)
I
:     Interest (The phenomena of Interest relates to a defined event, activity, experience or process)
Co
:  Context (Context is the setting or distinct characteristics. Note: Context not comparator)
 

Two other mnemonics may also be used to create protocols for both qualitative and quantitative studies - SPICE and SPIDER

SPICE: Within social sciences research, SPICE may be more appropriate for formulating research questions:

S:  Setting
P:  Perspective
I:   Intervention
C:  Comparison
E:  Evaluation

 

SPIDER can be used for both qualitative and quantitative studies:

S:   Sample
Pl:  Phenomenon of Interest
D:   Design
E:   Evaluation
R:   Research Type

 

References:

Booth, A. (2006). Clear and present questions: Formulating questions for evidence based practice. Library Hi Tech, 24(3), 355-368. doi:10.1108/07378830610692127

Cooke, A., Smith, D., & Booth, A. (2012). Beyond PICO: The SPIDER tool for qualitative evidence synthesis. Qualitative Health Research, 22(10), 1435-1443. doi:10.1177/1049732312452938

Methley, A., Campbell, S., Chew-Graham, C., McNally, R., & Cheraghi-Sohi, S. (2014). PICO, PICOS and SPIDER: A comparison study of specificity and sensitivity in three search tools for qualitative systematic reviews. BMC Health Services Research, 14(1), 579. doi:10.1186/s12913-014-0579-0