Ronny Gunnarsson. Writing a systematic review [in Science Network TV]. Available at: http://science-network.tv/writing-a-systematic-review/. Accessed January 16, 2018.
Before you initiate in a project you may want to know what others have done. Is the wheel already invented? Or perhaps, what kind of problems seemed to be related to constructing a wheel? Can I avoid these problems? You start by defining your focus of interest, your topic. Before commencing on doing a literature review you may need to spend some time in defining your topic. This can be compared to the layers of an onion where you have a core focus and layers of related topics around. Another approach is to describe all your interests as different circles that partially overlap. How would you describe the area where most (or all) circles meet?
Table of Contents
- 1 Different types of literature reviews
- 2 Heads and tails
- 3 Steps in a systematic review:
- 4 Content of your manuscript and author instructions
- 5 Finding a journal to submit your manuscript to
- 6 More practical tips
- 7 Useful links
- 8 References
Different types of literature reviews
There are different types of literature reviews such as narrative reviews, scoping reviews and systematic reviews, each with a different level of ambition. The next step for you is to decide what type of review you aim for. These types can be described as an hierarchy with five levels (partly adapted from Dijkers , Erwin and Higgins ):
|A) Narrative review||Literature is not collected systematically using a specific search algorithm. The amount of data extracted from publications are limited. They can’t be reproduced since a detailed description of the search strategy and information extracted from each publication is not published.|
|B) Evidence mapping||As above but adds more specific focus or questions to be answered. The search is also more systematic using a defined combination of search terms. Findings are briefly summarized and described in tables.|
|C) Scoping review||As above but also add a description of relevant evidence.|
|D-1) Systematic review without a meta-analysis
D-3) Qualitative metasynthesis
D-4) Meta-narrative reviews (=Mixed studies review = Integrative review)
D-5) Realist syntheses
|D-1) As above but the search strategy is usually a bit more focused or narrowed enabling a more extensive in depth search. A systematic review should also add an evaluation of methodological quality and finally make a qualitative synthesis of best evidence and subsequent recommendations based upon this.
D-2) A meta-analysis is most often a systematic review that also adds a quantitative synthesis (recalculation) of evidence. It will deliver the highest possible evidence for the right topic were included publications are fairly homogeneous. A meta-analysis is not always suitable and can sometimes be a worse option than a systematic review without a meta-analysis. Furthermore, a meta-analysis is sometimes done on handpicked publications not collected through a systematic search. Hence, most meta-analyses, but not all of them, are also systematic reviews
D-3) Qualitative metasynthesis compares a number of publications where empiric-holistic (qualitative) methods were used to summarize their findings .
D-4) Meta-narrative reviews aim to do a systematic review of studies using either an empiric-holistic (qualitative), empiric-atomistic (quantitative) or mixed methods approach with an ability to embrace studies with very different design and concepts . These reviews are well structured detailing search strategy and data extraction methods following the RAMESES publication standards for meta-narrative reviews . Since they provide enough information to be reproducible they can be considered to be a type of systematic review. Meta-narrative reviews, sometimes referred to as mixed studies review , are specially suited to describe concepts evolving, perhaps in parallel tracks in separate paradigms.
D-5) Realist evaluations answers the question what works, for whom, in what circumstances, and how in a broad range of interventions . To compile multiple realist evaluations you do a realist synthesis . This follows the RAMESES publication standards stating that enough information about search strategy and information retrieved should be described so the review can be reproduced. Realist synthesis can be considered as a special case of meta-narrative reviews (see above) focused on evaluating what works.
|E-1) Systematic review of systematic reviews (=Umbrella Review )
E-2) Quantitative meta-synthesis (=meta-meta-analysis)
|E-1) If several systematic reviews already exist in your topic what should you do? The easiest option is to do nothing and say we already know enough. However, if previous reviews came up with very different conclusions it means we don’t know enough. If that is the case should you do a review of systematic reviews or just a new systematic review addressing any potential shortcomings of previous reviews. The latter choice is likely to be both better and more demanding.
The increasing number of systematic reviews of the same or overlapping topics creating a confusion. Doing a new systematic review (level D above) would be the best option if you suspect there are new publications since the last review. This table lists five levels (A-E) where B is more ambitious than A and so on. However, I advocate that level E is not more ambitious or “better” than level D .
E-2) Doing a systematic review of reviews combining results from several meta-analysis into one new conclusion is labelled meta-meta-analysis or quantitative meta-synthesis . Quantitative meta-synthesis (E2) is a special case of systematic review of systematic reviews (E1).
Demarcation between different types of reviews
There might be varying opinions on the definitions above. It seems that there is less varying opinions about what constitutes a systematic review and more varying opinions about what constitutes a scoping review .
A narrative review or evidence mapping is the minimum type of work you need to do before starting any project. Today an increasing proportion of peer reviewed scientific journals would not accept manuscripts presenting a narrative review or a scoping review. Most peer-reviewed scientific journals would expect a systematic review following PRISMA or RAMESES guidelines. Hence, one important question is if you aim to publish your review in a peer-reviewed scientific journal. In a PhD project you would normally expect a systematic review with or without a meta-analysis.
A confusion exists around “narrative reviews” and “meta-narrative reviews” since both contain the word “narrative”. Narrative reviews are not systematic and not reproducible. Meta-narrative reviews contain a detailed description of the search strategy, data extracted and they can therefore be reproduced. Meta-narrative reviews can therefore be considered to be a kind of systematic review but with a different focus than the traditional systematic reviews.
The rest of this page will only talk about systematic reviews (including meta-analysis) and not about doing a systematic review of systematic reviews or quantitative meta-synthesis.
Meta analysis – is it always better?
A special situation is to sum results of several studies and calculate a new result as in a meta analysis. A meta analysis can be made when you do a review of randomized controlled trials (RCT) or some observational studies (see table below). However, a meta analysis can sometimes (less common) also be done including only a limited number of studies. Hence, most meta analyses are systematic reviews but not all systematic reviews are meta analyses. The main reasons for doing a meta-analysis:
- Pooling data to show a clearer picture of what the evidence says. This is usually illustrated with a forrest plot.
- In the situation where many publications have similar outcome variables (homogeneity in outcome) but very different result. A meta-analysis can explore possible confounding factors to why they come up with different result.
- Pooling data can tell us if we know enough or if more studies are needed.
A systematic literature review with a meta analysis is not always better than a systematic literature review without a meta analysis. A meta analysis is usually the best way forward if most of the retrieved publications has a reasonable homogeneity in choice of outcome variables (effect sizes). However, it is very common that retrieved publications are very heterogeneous and in that case it might be better to do a review without meta-analysis rather than excluding most publications and including a few in a meta-analysis. A meta-analysis has the ability to pool data and provide a new result that the single studies could not. Please have a look at this introduction to meta-analysis (prepared by the Cochrane Consumers and Communication Group, La Trobe University and generously support by Cochrane Australia. Written by Jack Nunn and Sophie Hill.):
Heads and tails
A review of what previous researchers have done should preferably be made systematically and presented in a way that it can be reproduced by others. PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) is a good example of a systematic approach. A systematic literature review builds upon previous publications. This situation can be described as a coin with two sides (“heads or tails”). One side of the coin are advice given to authors of a single original article to ensure highest possible quality (“heads”). The other side of the coin, used in systematic reviews, are different ways to evaluate previously published articles (“tails”). Both sides of the coin (“heads and tails”) contains similar opinions concerning good quality. The table below elucidates the two different sides of the coin.
|⇐ “HEADS” ⇒||⇐ “TAILS” ⇒|
|Type of original study||Groups compared||Guide to write original article||Aim for meta analysis||Guide to write a systematic review: Follow PRISMA or CAMARADES where it applies and also guidelines found below (where they apply).||Examples of systematic literature reviews|
|Randomized Controlled Trials||Yes||CONSORT||Yes||Previously different scales and scoring were used for assessment of methodological quality of RCTs. Example of scoring systems used were Jadad , the Delphi list , Maastricht-Amsterdam List (MAL) and PEDRO . However, many more scales exist . These scorings were used to separate studies of high or low quality by introducing a cut off in the scale. This may then be used to establish levels of evidence for a specific topic. However, for randomized controlled trials we now recommend that you follow the Cochrane Collaboration tool for assessing risk of bias graphically without directly calculating a score .
For meta-analysis please read more about doing a meta-analysis on the Cochrane collaboration website.
For randomized controlled trials involving animals please look at ARRIVE , SYRCLE’s risk of bias tool for animal studies or CAMARADES .
|Comparison of Aspirin, Warfarin, and New Anticoagulants for Stroke Prevention . (example of a meta-analysis)|
|Randomized Controlled Trials||Yes||CONSORT||No||Treatments in whiplash-associated disorders . (Example of establishing best-evidence synthesis).
Dendritic cells prolong the survival of renal allografts . (Example of review of lab based research)
|Observational studies (Case-Control / cohort studies)||Yes||STROBE||Yes||The Newcastle-Ottawa Scale (NOS) can often be used as a basis. Quite often you may need to modify or add a few elements tailored to your topic. Use MOOSE if you pool data and make a meta-analysis.||Growth factor and coronary heart disease
Risk Factors for Conversion of Laparoscopic Cholecystectomy .
|Observational studies (Case-Control / cohort studies)||Yes||STROBE||No||The Newcastle-Ottawa Scale (NOS) can often be used as a basis. Quite often you may need to modify or add a few elements tailored to your topic.||Risk factors for conversion of laparoscopic cholecystectomy . (example of how to define methodological quality assessment – Table 3 – and the subsequent outcome – Figure 2)|
|Observational studies||No||STROBE||Yes||There are a few scales that sometimes may fit such as QUACS . However, you often need to construct a scale relevant for the topic you study. Look at examples to the right first. Avoid using a quality sum score and instead present quality as a figure (The reference is an example for RCTs but can be adapted to other types of studies).||Depression in chronic kidney disease . (example of presenting risk of bias as a figure – figure 2)|
|Observational studies||No||STROBE||No||There are a few scales that sometimes may fit such as QUACS or MMAT . However, you often need to construct a scale relevant for the topic you study. Look at examples to the right first. Avoid using a quality sum score and instead present quality as a figure (The reference is an example for RCTs but can be adapted to other types of studies).
For anatomical studies consider the AQUA tool .
|Patency after Balloon Angioplasty in Hemodialysis Fistulas. (example of quality assessment – Table 2)
Postoperative Adverse Events & the WHO Surgical Safety Checklist (example of quality assessment – Figure 2-3)
Orthorexia nervosa . (example of an integrative review – focusing on how publications relate to different epistemological paradigms)
Staff Acceptance of Tele-ICU Coverage . (example of an integrative review – including studies with a qualitative approach).
Staff Women and HIV (example of an integrative review)
|Evaluation of diagnostic tests||No||STARD||Yes||QUADAS or a modification of it .||(Coming)|
|Evaluation of diagnostic tests||No||STARD||No||QUADAS or a modification of it .||(Coming)|
Steps in a systematic review:
Describe the preliminary scope of your review and put it down in writing. Drawing overlapping circles where one circle is one topic may be useful. Try to describe your scope as a union between two or more circles.
- Search in relevant databases for published systematic reviews relevant to your scope. Carefully document what you find.
- Go to PROSPERO – An international prospective register of systematic reviews for human research (or CAMARADES for animal research) and search for ongoing, not yet published, reviews covering your scope.
- If you found any published or ongoing reviews that seems identical to your scope then you may have to change scope. Revise the scope of your review and document in writing the revised scope of your review.
- Test different combinations of search terms to find a relevant combination resulting in hits relevant to your scope. A good combination will yield approximately 50% of hits being relevant (it is difficult to find search term giving a higher yield of relevant publications). You also have to consider the number of hits from your search and your available resources. One or two persons doing a review may embrace a thousand titles reading approximately 500 abstracts, retrieve up to 100 full text publications and after reviewing them include 10-70 full text publications in the assessment. These numbers can be increased if you have more resources (more staff) available. You need to rethink if you get much more hits than you can manage with available resources. If that is the case revise the scope of your review again and test a new search strategy. This is like a dance between testing search terms and fine tuning the scope of the review.
- Carefully document the final scope of your review.
- Define a final relevant set of search terms. Carefully document the final version of set of search terms.
- Register your systematic review in a register of systematic reviews. This is mandatory for reviews of randomized controlled trials and recommended for all other reviews. Go to PROSPERO – An international prospective register of systematic reviews for human research (or CAMARADES for animal studies). Examples of registrations in PROSPERO of systematic reviews that are not meta analyses:
- Do the search using your final set of search terms. You must do the search in at least two publication databases and the set of search terms will be very similar (though not identical) in each database. Carefully document the hits in each separate database ignoring that many publications are duplicates. (All this needs to be written in the PRISMA flow chart).
- Import the hits from each database into a reference managing software (EndNote or similar) and let the reference managing software merge the hits to one single list of publications, eliminating all duplicates. Carefully document the number after removing duplicates.
- Scan trough all titles and read selected abstracts. Decide which titles seems relevant. Carefully document this step.
- Read abstract for all articles where the title seems relevant. Carefully document this procedure.
- Read full text articles where the abstract seemed relevant. Carefully document this procedure.
- Decide which articles are relevant and retrieve them in full text. In the previous steps carefully document why some articles were not chosen.
- When you read the full text you will find a few articles that are actually not relevant for your review. Carefully document why some full text articles are omitted.
- Study reference lists within the remaining set of relevant articles to find a few more articles. Include them in the final list of relevant articles. Carefully document this step
- Decide how methodological quality (risk of bias) assessment of each single publication in the final list should be done (see table above). It was previously common for randomized controlled trials to score them according to an accepted scoring system. However, it is now more common to present this as a figure rather than calculating a score for each publication. Please note that “level of evidence” is used to summarize the evidence from multiple publications focusing on evaluating some kind of intervention. “Level of evidence” is not suitable to assess single publications evaluating an intervention. Furthermore, publications not evaluating an intervention need to be assessed using different approaches than the ones used for randomised controlled trials (or similar publications evaluating an intervention). Some advice might be found from examples in the table above but you may have to construct your own adaptation of how to assess the methodological quality of single publications.
- Decide how methodological quality should influence your conclusions. Should studies with poor quality be ignored partly or completely? Please write this down and ensure it is expressed in your final publication.
- Decide how findings of publications should be extracted and presented. Can you do a meta-analysis? It is usually better to refrain from doing a meta-analysis if the included publications are heterogeneous.
- Summarize your findings:
a) Methodological quality assessment of single publications, preferably give details in a table or figure while summarizing the main issues in text.
b) Actual findings, preferably summarised in sub-themes or topics rather than describing every publication (the latter should be done in one or several tables where the reader can look up details). If your review focus on an intervention this would be the place to state the level of evidence for each finding.
- Write the article following guidelines from PRISMA, CAMARADES or other relevant guidelines. You will find good advice on how to write a systematic review in Systematic Reviews: CRD’s guidance for undertaking systematic reviews in health care. (Start by looking at Box 1.11 on page 78). You may also find the explanation of the PRISMA checklist useful . Also see advice below about how to write this up.
Systematic reviews can be a shortcut to grasp the outcome of many studies. However, remember that the outcome of a systematic review depends on a number of decisions made during the process. Read the example of low back pain.
A manuscript for submission should not have a nice graphical layout. This is something the journal adds on after the manuscript is accepted. They have their own methods to address this. Any graphical stuff added before just makes their job more difficult. This means that most journals don’t want you to put tables into the text. Some journals want you to put a placeholder (a kind of notation) describing where you would prefer a table. They still want you to have a table either as a separate page later in the manuscript or, as a completely separate file. The desired layout of your manuscript is for most journals:
- Title page
- Abstract / Summary
- Introduction / Background
- References / Bibliography
All scientific journals have author instructions. Study them carefully and follow them. Some journals may want you to submit without a title page. Some journals want the tables and figures as the last part of the manuscript. Other journals want you to upload tables and figures as separate files. Some journals may want the acknowledgement in another place. All text should be written with 1.5 or 2 in line spacing. Most journals prefer a font size of 12. The maximum allowed length of a manuscript varies depending on journal. Item 4-8 above should be headings in your manuscript. Item 3 (Introduction) is also a heading in some journals while in others the background text starts directly without a heading.
Check in author instructions how long (how many words) your manuscript can be. A few journals have no word limit. However, most journals have and for a literature review it would usually be somewhere between 2,500-4,000 words. The shorter the more likely it is read. It is difficult and very time consuming to shorten a manuscript expressing yourself more succinct. However, if successful the reader will experience a well digested interesting text that reads more easily. For every sentence you need to ask yourself – is this sentence really necessary? Can I refer to the original publication instead of describing all details? Once you have gone through all sentences you need to start over and ask for every word – is this word really necessary? Have a look at some of the examples given in the table above.
Contains the title, list all authors and states all author’s affiliation. I also states who is corresponding author. Identify the manuscript as a systematic review (and meta-analysis if relevant) in the title. The title page is a separate page. Some journals prefer it to be the first page of the document. Other journals want it as a separate file.
Abstract / Summary
Look in the PRISMA checklist and the author instructions for the journal to figure out what subheadings you need in the abstract. The abstract is usually also a separate page coming after the title page but before the introduction. Hence, the introduction starts on a new page.
Paragraphs in Introduction-Background
Your introduction-background should consist of a few paragraphs that works like a red cord. Each paragraph prepares the way for the next paragraph:
- Begin with describing the topic and why it is an important topic. You may want to mention prevalence or costs associated with this topic. You are not supposed to write a new textbook here so this paragraph must not be too long.
- The rationale for a project is always to solve a problem. In this paragraph describe the unresolved problem associated with the topic. The problem may be (a few examples):
-We may have different treatment options for disease X. It is unclear what the effect of these treatment options are.
-It is unclear how patients with a higher risk should be identified.
-The correlation between different risk factors and an unwanted outcome is unclear.
(1 and 2 may be combined into one paragraph)
- Once you have defined the topic and the associated problem then you need to state what others have done to solve the problem. In the case of a systematic literature review you need to describe all previously published literature reviews, including reviews that are not systematic and that did not follow the PRISMA guidelines. Sometimes you (and your advisors) may find that to your knowledge there are no previously published literature reviews. If that is the case state state this rather than just not addressing previous reviews. (If there are no reviews then this paragraph will only be one sentence and if that is the case I recommend that you combine this sentence with 1+2 above).
- In the previous paragraphs you have described the topic, the remaining problem and previous systematic reviews to shed light on the problem. If previous literature reviews sheed enough light on the problem (solved it completely) then there would not be a need for a new systematic literature review. Thus, you need to describe, in light of previous published reviews, why a new review is needed. If there are no previous reviews then you still need to state why we need a literature review. This is the rationale for your project.
- State the aim. An example might be “Accordingly, we conducted a meta-analysis combining publically available data with the aim of …” or “We conducted a systematic review and meta-analysis of studies of… ” or “The aim of this review was to systematically examine studies which have… “. (This aim can often be combined with 4 above into one paragraph).
Paragraphs in Methods
Please read the checklist in PRISMA guidelines under the section “Methods”. This gives good advice on the paragraphs you expect to find here. When you describe the search strategy you have the choice of describing in text or refer to a table describing your key words used.
Paragraphs in Results
Please read the checklist in PRISMA guidelines under the section “Results”. This gives good advice on the paragraphs you expect to find here. When you refer to “Study selection” you should refer to the PRISMA flow chart as a figure. You can download this figure as a Word file from the PRISMA homepage so you don’t have to make that figure yourself, just change text and figures. Several of the other sections may also be suitable to present in tables (look at the examples given above). Remember that the text should not simply repeat facts from tables. Repeating details from each publication in the text makes it difficult to read and grasp. The text should guide the reader and point out the main trends referring to tables and references where relevant.
Paragraphs in Discussions
- Begin with a short summary of your main findings (results). Don’t describe all findings here, just the most important ones. This summary should be very short (1-3 sentences).
- In the following sections summarize your most important findings, without repeating details from the results, and discuss its relevance for different readers. Why do some previous publications end up in opposite conclusions? What do we really know when the existing literature is compiled? What are the consequences for doctors, policy makers, etc.
- Describe strength and weaknesses with this review.
- What is your final conclusions around your findings. Also try to clarify for the reader what your review adds compared to previous existing reviews. What was your unique contribution. Finally, try to identify the need for any future studies that you think should be done?
The most common pitfall in writing a systematic literature review is to simply repeat the results in the discussion without any deeper conclusions. A good discussion should not repeat results but rather explain peculiar results and provide possible explanations to why some findings in the literature are contradicting. It should be enjoyable to read and end in understandable take home messages. Writing a good discussion is probably the most challenging part of writing a good systematic literature review and this is often the part that ultimately decides if it is going to be published and read.
The decision to put information in a table or purely in the text (usually under the heading Results) depends on the amount of information. Small amounts of information should be described solely in text. The more information you can arrange into a table the better. However, there is an upper limit where tables tends to be too big and difficult to grasp. Always carefully consider if all information is necessary. The problem of information overload tends to be more common than the problem of providing too little information (although the latter exists as well). Also consider the number of tables you want to have. Few journals would accept more than five tables. In most manuscripts you can often do well with 1-3 tables. I recommend that you have a look at tables in the examples given previously (in the large table). A few things to consider:
- Start each table on a new page. On a Windows computer you create a page break with Ctrl+Enter.
- Tables should normally only have a few horizontal lines and no vertical lines. Cells should not be filled with colors.
- Every table should have a numbering and a title following the numbering (see examples given above).
- Most tables need some kind of footnotes to clarify some details.
The first figure is usually the PRISMA flow chart. You can download this figure as a Word file from the PRISMA homepage so you don’t have to make that figure yourself. Just change text and figures and insert this page in your Word manuscript. Ensure that all figures given in the flowchart sums up. This is usually your only figure (if you are not doing a meta-analysis). Each figure should start on a new page and figures should all be placed after tables.
Thank staff and participants for their contribution. Do not thank authors (they get their fame by being authors). When it comes to staff be specific on what they did that you want to thank them for. Describe any funding and potential conflict of interests.
References / Bibliography
There are two main systems for presenting references; Harvard and Vancouver. Each journal tends to have its own variation of any of these. This means that the reference list and the annotations in the text are specific to a journal. Follow author instructions regarding reference style carefully. A good software for managing references will save you a lot of time. The importance of these software cannot be overestimated.
Finding a journal to submit your manuscript to
The best way to make your findings easily accessible is to submit them to a scientific journal. This journal has routines to ensure your publication is indexed in the right publication and citation databases. There are different pathways to find the best journal for submission:
- Write your manuscript as you want first (following guidelines given further down on this page). Once you are finished have a look at your own reference list in your manuscript. You cite journals in your reference list because they have published other studies relevant to your study. Hence, they are likely to be interested in your research. Have a look at the journals found in your own reference list. Which journals do you cite most often. What is their impact factor? Decide which one you want to try first. Make minor adjustments to your manuscript to tailor it to this journal.
- Find the journals in your topic with the highest impact factor before your manuscript is written and tailor your writing according to these author instructions.
I would recommend option A in most situations. This will give you a short list of 3-5 possible journals. If it gets rejected by the first then you already have an idea of where to send it next time. Before finally deciding check that the journal you aim to submit to…
- …is indexed in at least one of the main citation databases such as Thomson Reuter’s Web of Science or Elsevier’s Scopus (follow these links and see if the journal you plan to submit to is indexed there).
- …is not a journal suspected to only aim for your money (as a publication fee). It might be worth having a look at Wikipedia on Predatory open access publishing.
More practical tips
The webpage Writing a scientific publication has valuable practical tips. I would specially point out the sections:
- A few tips when using Word or similar software
- Academic writing
- Using a translator or language editor
- After submission – Rejection
- After submission – Checking a proof and answering questions if accepted
- Plagiarism – a gateway to hell?
- Read more on the homepage for PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses). The PRISMA Statement consists of a 27-item checklist and a four-phase flow diagram. Have a look at the checklist and flow diagram before you start. Please also have a look at the examples given above.
- AMSTAR describing good quality of a systematic literature review.
- System for the Unified Management, Assessment and Review of Information (SUMARI), the Joanna Briggs Institutes premier software for the systematic review of literature.
- Review Manager (RevMan) is the software used for preparing and maintaining Cochrane Reviews.
- Critical Appraisal Tools – University of South Australia
- Centre for Evidence-Based Medicine – Toronto
- Appraisal Tools at the Critical Appraisal Skills Programme (CASP)
- Software for Publication Bias by Michael Borenstein
Ronny Gunnarsson. Writing a systematic review [in Science Network TV]. Available at: http://science-network.tv/writing-a-systematic-review/. Accessed January 16, 2018.