methodology review of literature definition

What is a Literature Review? How to Write It (with Examples)

A literature review is a critical analysis and synthesis of existing research on a particular topic. It provides an overview of the current state of knowledge, identifies gaps, and highlights key findings in the literature. 1 The purpose of a literature review is to situate your own research within the context of existing scholarship, demonstrating your understanding of the topic and showing how your work contributes to the ongoing conversation in the field. Learning how to write a literature review is a critical tool for successful research. Your ability to summarize and synthesize prior research pertaining to a certain topic demonstrates your grasp on the topic of study, and assists in the learning process. 

Table of Contents

What is the purpose of literature review , a. habitat loss and species extinction: , b. range shifts and phenological changes: , c. ocean acidification and coral reefs: , d. adaptive strategies and conservation efforts: .

• Choose a Topic and Define the Research Question: 
• Decide on the Scope of Your Review: 
• Select Databases for Searches: 
• Conduct Searches and Keep Track: 
• Review the Literature: 
• Organize and Write Your Literature Review: 
• How to write a literature review faster with Paperpal? 

Frequently asked questions 

What is a literature review .

A well-conducted literature review demonstrates the researcher’s familiarity with the existing literature, establishes the context for their own research, and contributes to scholarly conversations on the topic. One of the purposes of a literature review is also to help researchers avoid duplicating previous work and ensure that their research is informed by and builds upon the existing body of knowledge.

A literature review serves several important purposes within academic and research contexts. Here are some key objectives and functions of a literature review: 2  

1. Contextualizing the Research Problem: The literature review provides a background and context for the research problem under investigation. It helps to situate the study within the existing body of knowledge. 

2. Identifying Gaps in Knowledge: By identifying gaps, contradictions, or areas requiring further research, the researcher can shape the research question and justify the significance of the study. This is crucial for ensuring that the new research contributes something novel to the field.

Find academic papers related to your research topic faster. Try Research on Paperpal

3. Understanding Theoretical and Conceptual Frameworks: Literature reviews help researchers gain an understanding of the theoretical and conceptual frameworks used in previous studies. This aids in the development of a theoretical framework for the current research. 

4. Providing Methodological Insights: Another purpose of literature reviews is that it allows researchers to learn about the methodologies employed in previous studies. This can help in choosing appropriate research methods for the current study and avoiding pitfalls that others may have encountered. 

5. Establishing Credibility: A well-conducted literature review demonstrates the researcher’s familiarity with existing scholarship, establishing their credibility and expertise in the field. It also helps in building a solid foundation for the new research. 

6. Informing Hypotheses or Research Questions: The literature review guides the formulation of hypotheses or research questions by highlighting relevant findings and areas of uncertainty in existing literature. 

Literature review example 

Let’s delve deeper with a literature review example: Let’s say your literature review is about the impact of climate change on biodiversity. You might format your literature review into sections such as the effects of climate change on habitat loss and species extinction, phenological changes, and marine biodiversity. Each section would then summarize and analyze relevant studies in those areas, highlighting key findings and identifying gaps in the research. The review would conclude by emphasizing the need for further research on specific aspects of the relationship between climate change and biodiversity. The following literature review template provides a glimpse into the recommended literature review structure and content, demonstrating how research findings are organized around specific themes within a broader topic. 

Literature Review on Climate Change Impacts on Biodiversity:  

Climate change is a global phenomenon with far-reaching consequences, including significant impacts on biodiversity. This literature review synthesizes key findings from various studies: 

Climate change-induced alterations in temperature and precipitation patterns contribute to habitat loss, affecting numerous species (Thomas et al., 2004). The review discusses how these changes increase the risk of extinction, particularly for species with specific habitat requirements. 

Observations of range shifts and changes in the timing of biological events (phenology) are documented in response to changing climatic conditions (Parmesan & Yohe, 2003). These shifts affect ecosystems and may lead to mismatches between species and their resources. 

The review explores the impact of climate change on marine biodiversity, emphasizing ocean acidification’s threat to coral reefs (Hoegh-Guldberg et al., 2007). Changes in pH levels negatively affect coral calcification, disrupting the delicate balance of marine ecosystems. 

Recognizing the urgency of the situation, the literature review discusses various adaptive strategies adopted by species and conservation efforts aimed at mitigating the impacts of climate change on biodiversity (Hannah et al., 2007). It emphasizes the importance of interdisciplinary approaches for effective conservation planning. 

Strengthen your literature review with factual insights. Try Research on Paperpal for free!

How to write a good literature review 

Writing a literature review involves summarizing and synthesizing existing research on a particular topic. A good literature review format should include the following elements. 

Introduction: The introduction sets the stage for your literature review, providing context and introducing the main focus of your review. 

• Opening Statement: Begin with a general statement about the broader topic and its significance in the field. 
• Scope and Purpose: Clearly define the scope of your literature review. Explain the specific research question or objective you aim to address. 
• Organizational Framework: Briefly outline the structure of your literature review, indicating how you will categorize and discuss the existing research. 
• Significance of the Study: Highlight why your literature review is important and how it contributes to the understanding of the chosen topic. 
• Thesis Statement: Conclude the introduction with a concise thesis statement that outlines the main argument or perspective you will develop in the body of the literature review. 

Body: The body of the literature review is where you provide a comprehensive analysis of existing literature, grouping studies based on themes, methodologies, or other relevant criteria. 

• Organize by Theme or Concept: Group studies that share common themes, concepts, or methodologies. Discuss each theme or concept in detail, summarizing key findings and identifying gaps or areas of disagreement. 
• Critical Analysis: Evaluate the strengths and weaknesses of each study. Discuss the methodologies used, the quality of evidence, and the overall contribution of each work to the understanding of the topic. 
• Synthesis of Findings: Synthesize the information from different studies to highlight trends, patterns, or areas of consensus in the literature. 
• Identification of Gaps: Discuss any gaps or limitations in the existing research and explain how your review contributes to filling these gaps. 
• Transition between Sections: Provide smooth transitions between different themes or concepts to maintain the flow of your literature review. 

Write and Cite as yo u go with Paperpal Research. Start now for free!

Conclusion: The conclusion of your literature review should summarize the main findings, highlight the contributions of the review, and suggest avenues for future research. 

• Summary of Key Findings: Recap the main findings from the literature and restate how they contribute to your research question or objective. 
• Contributions to the Field: Discuss the overall contribution of your literature review to the existing knowledge in the field. 
• Implications and Applications: Explore the practical implications of the findings and suggest how they might impact future research or practice. 
• Recommendations for Future Research: Identify areas that require further investigation and propose potential directions for future research in the field. 
• Final Thoughts: Conclude with a final reflection on the importance of your literature review and its relevance to the broader academic community. 

Conducting a literature review 

Conducting a literature review is an essential step in research that involves reviewing and analyzing existing literature on a specific topic. It’s important to know how to do a literature review effectively, so here are the steps to follow: 1  

Choose a Topic and Define the Research Question:  

• Select a topic that is relevant to your field of study. 
• Clearly define your research question or objective. Determine what specific aspect of the topic do you want to explore? 

Decide on the Scope of Your Review:  

• Determine the timeframe for your literature review. Are you focusing on recent developments, or do you want a historical overview? 
• Consider the geographical scope. Is your review global, or are you focusing on a specific region? 
• Define the inclusion and exclusion criteria. What types of sources will you include? Are there specific types of studies or publications you will exclude? 

Select Databases for Searches:  

• Identify relevant databases for your field. Examples include PubMed, IEEE Xplore, Scopus, Web of Science, and Google Scholar. 
• Consider searching in library catalogs, institutional repositories, and specialized databases related to your topic. 

Conduct Searches and Keep Track:  

• Develop a systematic search strategy using keywords, Boolean operators (AND, OR, NOT), and other search techniques. 
• Record and document your search strategy for transparency and replicability. 
• Keep track of the articles, including publication details, abstracts, and links. Use citation management tools like EndNote, Zotero, or Mendeley to organize your references. 

Review the Literature:  

• Evaluate the relevance and quality of each source. Consider the methodology, sample size, and results of studies. 
• Organize the literature by themes or key concepts. Identify patterns, trends, and gaps in the existing research. 
• Summarize key findings and arguments from each source. Compare and contrast different perspectives. 
• Identify areas where there is a consensus in the literature and where there are conflicting opinions. 
• Provide critical analysis and synthesis of the literature. What are the strengths and weaknesses of existing research? 

Organize and Write Your Literature Review:  

• Literature review outline should be based on themes, chronological order, or methodological approaches. 
• Write a clear and coherent narrative that synthesizes the information gathered. 
• Use proper citations for each source and ensure consistency in your citation style (APA, MLA, Chicago, etc.). 
• Conclude your literature review by summarizing key findings, identifying gaps, and suggesting areas for future research. 

Whether you’re exploring a new research field or finding new angles to develop an existing topic, sifting through hundreds of papers can take more time than you have to spare. But what if you could find science-backed insights with verified citations in seconds? That’s the power of Paperpal’s new Research feature!  

How to write a literature review faster with Paperpal?  

Paperpal, an AI writing assistant, integrates powerful academic search capabilities within its writing platform. With the Research | Cite feature, you get 100% factual insights, with citations backed by 250M+ verified research articles, directly within your writing interface. It also allows you auto-cite references in 10,000+ styles and save relevant references in your Citation Library. By eliminating the need to switch tabs to find answers to all your research questions, Paperpal saves time and helps you stay focused on your writing.   

Here’s how to use the Research feature:  

• Ask a question: Get started with a new document on paperpal.com. Click on the “Research | Cite” feature and type your question in plain English. Paperpal will scour over 250 million research articles, including conference papers and preprints, to provide you with accurate insights and citations. 

• Review and Save: Paperpal summarizes the information, while citing sources and listing relevant reads. You can quickly scan the results to identify relevant references and save these directly to your built-in citations library for later access. 
• Cite with Confidence: Paperpal makes it easy to incorporate relevant citations and references in 10,000+ styles into your writing, ensuring your arguments are well-supported by credible sources. This translates to a polished, well-researched literature review. 

The literature review sample and detailed advice on writing and conducting a review will help you produce a well-structured report. But remember that a good literature review is an ongoing process, and it may be necessary to revisit and update it as your research progresses. By combining effortless research with an easy citation process, Paperpal Research streamlines the literature review process and empowers you to write faster and with more confidence. Try Paperpal Research now and see for yourself.  

A literature review is a critical and comprehensive analysis of existing literature (published and unpublished works) on a specific topic or research question and provides a synthesis of the current state of knowledge in a particular field. A well-conducted literature review is crucial for researchers to build upon existing knowledge, avoid duplication of efforts, and contribute to the advancement of their field. It also helps researchers situate their work within a broader context and facilitates the development of a sound theoretical and conceptual framework for their studies.

Literature review is a crucial component of research writing, providing a solid background for a research paper’s investigation. The aim is to keep professionals up to date by providing an understanding of ongoing developments within a specific field, including research methods, and experimental techniques used in that field, and present that knowledge in the form of a written report. Also, the depth and breadth of the literature review emphasizes the credibility of the scholar in his or her field.  

Before writing a literature review, it’s essential to undertake several preparatory steps to ensure that your review is well-researched, organized, and focused. This includes choosing a topic of general interest to you and doing exploratory research on that topic, writing an annotated bibliography, and noting major points, especially those that relate to the position you have taken on the topic. 

Literature reviews and academic research papers are essential components of scholarly work but serve different purposes within the academic realm. 3 A literature review aims to provide a foundation for understanding the current state of research on a particular topic, identify gaps or controversies, and lay the groundwork for future research. Therefore, it draws heavily from existing academic sources, including books, journal articles, and other scholarly publications. In contrast, an academic research paper aims to present new knowledge, contribute to the academic discourse, and advance the understanding of a specific research question. Therefore, it involves a mix of existing literature (in the introduction and literature review sections) and original data or findings obtained through research methods. 

Literature reviews are essential components of academic and research papers, and various strategies can be employed to conduct them effectively. If you want to know how to write a literature review for a research paper, here are four common approaches that are often used by researchers.  Chronological Review: This strategy involves organizing the literature based on the chronological order of publication. It helps to trace the development of a topic over time, showing how ideas, theories, and research have evolved.  Thematic Review: Thematic reviews focus on identifying and analyzing themes or topics that cut across different studies. Instead of organizing the literature chronologically, it is grouped by key themes or concepts, allowing for a comprehensive exploration of various aspects of the topic.  Methodological Review: This strategy involves organizing the literature based on the research methods employed in different studies. It helps to highlight the strengths and weaknesses of various methodologies and allows the reader to evaluate the reliability and validity of the research findings.  Theoretical Review: A theoretical review examines the literature based on the theoretical frameworks used in different studies. This approach helps to identify the key theories that have been applied to the topic and assess their contributions to the understanding of the subject.  It’s important to note that these strategies are not mutually exclusive, and a literature review may combine elements of more than one approach. The choice of strategy depends on the research question, the nature of the literature available, and the goals of the review. Additionally, other strategies, such as integrative reviews or systematic reviews, may be employed depending on the specific requirements of the research.

The literature review format can vary depending on the specific publication guidelines. However, there are some common elements and structures that are often followed. Here is a general guideline for the format of a literature review:  Introduction:   Provide an overview of the topic.  Define the scope and purpose of the literature review.  State the research question or objective.  Body:   Organize the literature by themes, concepts, or chronology.  Critically analyze and evaluate each source.  Discuss the strengths and weaknesses of the studies.  Highlight any methodological limitations or biases.  Identify patterns, connections, or contradictions in the existing research.  Conclusion:   Summarize the key points discussed in the literature review.  Highlight the research gap.  Address the research question or objective stated in the introduction.  Highlight the contributions of the review and suggest directions for future research.

Both annotated bibliographies and literature reviews involve the examination of scholarly sources. While annotated bibliographies focus on individual sources with brief annotations, literature reviews provide a more in-depth, integrated, and comprehensive analysis of existing literature on a specific topic. The key differences are as follows: 

References 

• Denney, A. S., & Tewksbury, R. (2013). How to write a literature review.  Journal of criminal justice education ,  24 (2), 218-234. 
• Pan, M. L. (2016).  Preparing literature reviews: Qualitative and quantitative approaches . Taylor & Francis. 
• Cantero, C. (2019). How to write a literature review.  San José State University Writing Center . 

Paperpal is a comprehensive AI writing toolkit that helps students and researchers achieve 2x the writing in half the time. It leverages 22+ years of STM experience and insights from millions of research articles to provide in-depth academic writing, language editing, and submission readiness support to help you write better, faster.  

Get accurate academic translations, rewriting support, grammar checks, vocabulary suggestions, and generative AI assistance that delivers human precision at machine speed. Try for free or upgrade to Paperpal Prime starting at US$19 a month to access premium features, including consistency, plagiarism, and 30+ submission readiness checks to help you succeed.  

Experience the future of academic writing – Sign up to Paperpal and start writing for free!  

Related Reads:

• Empirical Research: A Comprehensive Guide for Academics 
• How to Write a Scientific Paper in 10 Steps 
• How Long Should a Chapter Be?
• How to Use Paperpal to Generate Emails & Cover Letters?

6 Tips for Post-Doc Researchers to Take Their Career to the Next Level

Self-plagiarism in research: what it is and how to avoid it, you may also like, academic integrity vs academic dishonesty: types & examples, dissertation printing and binding | types & comparison , what is a dissertation preface definition and examples , the ai revolution: authors’ role in upholding academic..., the future of academia: how ai tools are..., how to write a research proposal: (with examples..., how to write your research paper in apa..., how to choose a dissertation topic, how to write a phd research proposal, how to write an academic paragraph (step-by-step guide).

Libraries | Research Guides

Literature reviews, what is a literature review, learning more about how to do a literature review.

• Planning the Review
• The Research Question
• Choosing Where to Search
• Organizing the Review
• Writing the Review

A literature review is a review and synthesis of existing research on a topic or research question. A literature review is meant to analyze the scholarly literature, make connections across writings and identify strengths, weaknesses, trends, and missing conversations. A literature review should address different aspects of a topic as it relates to your research question. A literature review goes beyond a description or summary of the literature you have read. 

• Sage Research Methods Core This link opens in a new window SAGE Research Methods supports research at all levels by providing material to guide users through every step of the research process. SAGE Research Methods is the ultimate methods library with more than 1000 books, reference works, journal articles, and instructional videos by world-leading academics from across the social sciences, including the largest collection of qualitative methods books available online from any scholarly publisher. – Publisher

• Next: Planning the Review >>
• Last Updated: Jul 8, 2024 11:22 AM
• URL: https://libguides.northwestern.edu/literaturereviews

• University of Texas Libraries

Literature Reviews

• What is a literature review?
• Steps in the Literature Review Process
• Define your research question
• Determine inclusion and exclusion criteria
• Choose databases and search
• Review Results
• Synthesize Results
• Analyze Results
• Librarian Support
• Artificial Intelligence (AI) Tools

What is a Literature Review?

A literature or narrative review is a comprehensive review and analysis of the published literature on a specific topic or research question. The literature that is reviewed contains: books, articles, academic articles, conference proceedings, association papers, and dissertations. It contains the most pertinent studies and points to important past and current research and practices. It provides background and context, and shows how your research will contribute to the field. 

A literature review should: 

• Provide a comprehensive and updated review of the literature;
• Explain why this review has taken place;
• Articulate a position or hypothesis;
• Acknowledge and account for conflicting and corroborating points of view

From  S age Research Methods

Purpose of a Literature Review

A literature review can be written as an introduction to a study to:

• Demonstrate how a study fills a gap in research
• Compare a study with other research that's been done

Or it can be a separate work (a research article on its own) which:

• Organizes or describes a topic
• Describes variables within a particular issue/problem

Limitations of a Literature Review

Some of the limitations of a literature review are:

• It's a snapshot in time. Unlike other reviews, this one has beginning, a middle and an end. There may be future developments that could make your work less relevant.
• It may be too focused. Some niche studies may miss the bigger picture.
• It can be difficult to be comprehensive. There is no way to make sure all the literature on a topic was considered.
• It is easy to be biased if you stick to top tier journals. There may be other places where people are publishing exemplary research. Look to open access publications and conferences to reflect a more inclusive collection. Also, make sure to include opposing views (and not just supporting evidence).

Source: Grant, Maria J., and Andrew Booth. “A Typology of Reviews: An Analysis of 14 Review Types and Associated Methodologies.” Health Information & Libraries Journal, vol. 26, no. 2, June 2009, pp. 91–108. Wiley Online Library, doi:10.1111/j.1471-1842.2009.00848.x.

Librarian Assistance

For help, please contact the librarian for your subject area.  We have a guide to library specialists by subject .

• Last Updated: Aug 26, 2024 5:59 AM
• URL: https://guides.lib.utexas.edu/literaturereviews

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, automatically generate references for free.

• Knowledge Base
• Dissertation
• What is a Literature Review? | Guide, Template, & Examples

What is a Literature Review? | Guide, Template, & Examples

Published on 22 February 2022 by Shona McCombes . Revised on 7 June 2022.

What is a literature review? A literature review is a survey of scholarly sources on a specific topic. It provides an overview of current knowledge, allowing you to identify relevant theories, methods, and gaps in the existing research.

There are five key steps to writing a literature review:

• Search for relevant literature
• Evaluate sources
• Identify themes, debates and gaps
• Outline the structure
• Write your literature review

A good literature review doesn’t just summarise sources – it analyses, synthesises, and critically evaluates to give a clear picture of the state of knowledge on the subject.

Instantly correct all language mistakes in your text

Be assured that you'll submit flawless writing. Upload your document to correct all your mistakes.

Table of contents

Why write a literature review, examples of literature reviews, step 1: search for relevant literature, step 2: evaluate and select sources, step 3: identify themes, debates and gaps, step 4: outline your literature review’s structure, step 5: write your literature review, frequently asked questions about literature reviews, introduction.

• Quick Run-through
• Step 1 & 2

When you write a dissertation or thesis, you will have to conduct a literature review to situate your research within existing knowledge. The literature review gives you a chance to:

• Demonstrate your familiarity with the topic and scholarly context
• Develop a theoretical framework and methodology for your research
• Position yourself in relation to other researchers and theorists
• Show how your dissertation addresses a gap or contributes to a debate

You might also have to write a literature review as a stand-alone assignment. In this case, the purpose is to evaluate the current state of research and demonstrate your knowledge of scholarly debates around a topic.

The content will look slightly different in each case, but the process of conducting a literature review follows the same steps. We’ve written a step-by-step guide that you can follow below.

The only proofreading tool specialized in correcting academic writing

The academic proofreading tool has been trained on 1000s of academic texts and by native English editors. Making it the most accurate and reliable proofreading tool for students.

Correct my document today

Writing literature reviews can be quite challenging! A good starting point could be to look at some examples, depending on what kind of literature review you’d like to write.

• Example literature review #1: “Why Do People Migrate? A Review of the Theoretical Literature” ( Theoretical literature review about the development of economic migration theory from the 1950s to today.)
• Example literature review #2: “Literature review as a research methodology: An overview and guidelines” ( Methodological literature review about interdisciplinary knowledge acquisition and production.)
• Example literature review #3: “The Use of Technology in English Language Learning: A Literature Review” ( Thematic literature review about the effects of technology on language acquisition.)
• Example literature review #4: “Learners’ Listening Comprehension Difficulties in English Language Learning: A Literature Review” ( Chronological literature review about how the concept of listening skills has changed over time.)

You can also check out our templates with literature review examples and sample outlines at the links below.

Download Word doc Download Google doc

Before you begin searching for literature, you need a clearly defined topic .

If you are writing the literature review section of a dissertation or research paper, you will search for literature related to your research objectives and questions .

If you are writing a literature review as a stand-alone assignment, you will have to choose a focus and develop a central question to direct your search. Unlike a dissertation research question, this question has to be answerable without collecting original data. You should be able to answer it based only on a review of existing publications.

Make a list of keywords

Start by creating a list of keywords related to your research topic. Include each of the key concepts or variables you’re interested in, and list any synonyms and related terms. You can add to this list if you discover new keywords in the process of your literature search.

• Social media, Facebook, Instagram, Twitter, Snapchat, TikTok
• Body image, self-perception, self-esteem, mental health
• Generation Z, teenagers, adolescents, youth

Search for relevant sources

Use your keywords to begin searching for sources. Some databases to search for journals and articles include:

• Your university’s library catalogue
• Google Scholar
• Project Muse (humanities and social sciences)
• Medline (life sciences and biomedicine)
• EconLit (economics)
• Inspec (physics, engineering and computer science)

You can use boolean operators to help narrow down your search:

Read the abstract to find out whether an article is relevant to your question. When you find a useful book or article, you can check the bibliography to find other relevant sources.

To identify the most important publications on your topic, take note of recurring citations. If the same authors, books or articles keep appearing in your reading, make sure to seek them out.

You probably won’t be able to read absolutely everything that has been written on the topic – you’ll have to evaluate which sources are most relevant to your questions.

For each publication, ask yourself:

• What question or problem is the author addressing?
• What are the key concepts and how are they defined?
• What are the key theories, models and methods? Does the research use established frameworks or take an innovative approach?
• What are the results and conclusions of the study?
• How does the publication relate to other literature in the field? Does it confirm, add to, or challenge established knowledge?
• How does the publication contribute to your understanding of the topic? What are its key insights and arguments?
• What are the strengths and weaknesses of the research?

Make sure the sources you use are credible, and make sure you read any landmark studies and major theories in your field of research.

You can find out how many times an article has been cited on Google Scholar – a high citation count means the article has been influential in the field, and should certainly be included in your literature review.

The scope of your review will depend on your topic and discipline: in the sciences you usually only review recent literature, but in the humanities you might take a long historical perspective (for example, to trace how a concept has changed in meaning over time).

Remember that you can use our template to summarise and evaluate sources you’re thinking about using!

Take notes and cite your sources

As you read, you should also begin the writing process. Take notes that you can later incorporate into the text of your literature review.

It’s important to keep track of your sources with references to avoid plagiarism . It can be helpful to make an annotated bibliography, where you compile full reference information and write a paragraph of summary and analysis for each source. This helps you remember what you read and saves time later in the process.

You can use our free APA Reference Generator for quick, correct, consistent citations.

Prevent plagiarism, run a free check.

To begin organising your literature review’s argument and structure, you need to understand the connections and relationships between the sources you’ve read. Based on your reading and notes, you can look for:

• Trends and patterns (in theory, method or results): do certain approaches become more or less popular over time?
• Themes: what questions or concepts recur across the literature?
• Debates, conflicts and contradictions: where do sources disagree?
• Pivotal publications: are there any influential theories or studies that changed the direction of the field?
• Gaps: what is missing from the literature? Are there weaknesses that need to be addressed?

This step will help you work out the structure of your literature review and (if applicable) show how your own research will contribute to existing knowledge.

• Most research has focused on young women.
• There is an increasing interest in the visual aspects of social media.
• But there is still a lack of robust research on highly-visual platforms like Instagram and Snapchat – this is a gap that you could address in your own research.

There are various approaches to organising the body of a literature review. You should have a rough idea of your strategy before you start writing.

Depending on the length of your literature review, you can combine several of these strategies (for example, your overall structure might be thematic, but each theme is discussed chronologically).

Chronological

The simplest approach is to trace the development of the topic over time. However, if you choose this strategy, be careful to avoid simply listing and summarising sources in order.

Try to analyse patterns, turning points and key debates that have shaped the direction of the field. Give your interpretation of how and why certain developments occurred.

If you have found some recurring central themes, you can organise your literature review into subsections that address different aspects of the topic.

For example, if you are reviewing literature about inequalities in migrant health outcomes, key themes might include healthcare policy, language barriers, cultural attitudes, legal status, and economic access.

Methodological

If you draw your sources from different disciplines or fields that use a variety of research methods , you might want to compare the results and conclusions that emerge from different approaches. For example:

• Look at what results have emerged in qualitative versus quantitative research
• Discuss how the topic has been approached by empirical versus theoretical scholarship
• Divide the literature into sociological, historical, and cultural sources

Theoretical

A literature review is often the foundation for a theoretical framework . You can use it to discuss various theories, models, and definitions of key concepts.

You might argue for the relevance of a specific theoretical approach, or combine various theoretical concepts to create a framework for your research.

Like any other academic text, your literature review should have an introduction , a main body, and a conclusion . What you include in each depends on the objective of your literature review.

The introduction should clearly establish the focus and purpose of the literature review.

If you are writing the literature review as part of your dissertation or thesis, reiterate your central problem or research question and give a brief summary of the scholarly context. You can emphasise the timeliness of the topic (“many recent studies have focused on the problem of x”) or highlight a gap in the literature (“while there has been much research on x, few researchers have taken y into consideration”).

Depending on the length of your literature review, you might want to divide the body into subsections. You can use a subheading for each theme, time period, or methodological approach.

As you write, make sure to follow these tips:

• Summarise and synthesise: give an overview of the main points of each source and combine them into a coherent whole.
• Analyse and interpret: don’t just paraphrase other researchers – add your own interpretations, discussing the significance of findings in relation to the literature as a whole.
• Critically evaluate: mention the strengths and weaknesses of your sources.
• Write in well-structured paragraphs: use transitions and topic sentences to draw connections, comparisons and contrasts.

In the conclusion, you should summarise the key findings you have taken from the literature and emphasise their significance.

If the literature review is part of your dissertation or thesis, reiterate how your research addresses gaps and contributes new knowledge, or discuss how you have drawn on existing theories and methods to build a framework for your research. This can lead directly into your methodology section.

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a dissertation , thesis, research paper , or proposal .

There are several reasons to conduct a literature review at the beginning of a research project:

• To familiarise yourself with the current state of knowledge on your topic
• To ensure that you’re not just repeating what others have already done
• To identify gaps in knowledge and unresolved problems that your research can address
• To develop your theoretical framework and methodology
• To provide an overview of the key findings and debates on the topic

Writing the literature review shows your reader how your work relates to existing research and what new insights it will contribute.

The literature review usually comes near the beginning of your  dissertation . After the introduction , it grounds your research in a scholarly field and leads directly to your theoretical framework or methodology .

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the ‘Cite this Scribbr article’ button to automatically add the citation to our free Reference Generator.

McCombes, S. (2022, June 07). What is a Literature Review? | Guide, Template, & Examples. Scribbr. Retrieved 9 September 2024, from https://www.scribbr.co.uk/thesis-dissertation/literature-review/

Is this article helpful?

Shona McCombes

Other students also liked, how to write a dissertation proposal | a step-by-step guide, what is a theoretical framework | a step-by-step guide, what is a research methodology | steps & tips.

Which review is that? A guide to review types

• Which review is that?
• Review Comparison Chart
• Decision Tool
• Critical Review
• Integrative Review
• Narrative Review
• State of the Art Review
• Narrative Summary
• Systematic Review
• Meta-analysis
• Comparative Effectiveness Review
• Diagnostic Systematic Review
• Network Meta-analysis
• Prognostic Review
• Psychometric Review
• Review of Economic Evaluations
• Systematic Review of Epidemiology Studies
• Living Systematic Reviews
• Umbrella Review
• Review of Reviews
• Rapid Review
• Rapid Evidence Assessment
• Rapid Realist Review
• Qualitative Evidence Synthesis
• Qualitative Interpretive Meta-synthesis
• Qualitative Meta-synthesis
• Qualitative Research Synthesis
• Framework Synthesis - Best-fit Framework Synthesis
• Meta-aggregation
• Meta-ethnography
• Meta-interpretation
• Meta-narrative Review
• Meta-summary
• Thematic Synthesis
• Mixed Methods Synthesis
• Narrative Synthesis
• Bayesian Meta-analysis
• EPPI-Centre Review
• Critical Interpretive Synthesis
• Realist Synthesis - Realist Review
• Scoping Review
• Mapping Review
• Systematised Review
• Concept Synthesis
• Expert Opinion - Policy Review
• Technology Assessment Review

Methodological Review

• Systematic Search and Review

A methodological review is a type of systematic secondary research (i.e., research synthesis) which focuses on summarising the state-of-the-art methodological practices of research in a substantive field or topic" (Chong et al, 2021).

Methodological reviews "can be performed to examine any methodological issues relating to the design, conduct and review of research studies and also evidence syntheses". Munn et al, 2018)

Further Reading/Resources

Clarke, M., Oxman, A. D., Paulsen, E., Higgins, J. P. T., & Green, S. (2011). Appendix A: Guide to the contents of a Cochrane Methodology protocol and review. Cochrane Handbook for systematic reviews of interventions . Full Text PDF

Aguinis, H., Ramani, R. S., & Alabduljader, N. (2023). Best-Practice Recommendations for Producers, Evaluators, and Users of Methodological Literature Reviews. Organizational Research Methods, 26(1), 46-76. https://doi.org/10.1177/1094428120943281 Full Text

Jha, C. K., & Kolekar, M. H. (2021). Electrocardiogram data compression techniques for cardiac healthcare systems: A methodological review. IRBM . Full Text

References Munn, Z., Stern, C., Aromataris, E., Lockwood, C., & Jordan, Z. (2018). What kind of systematic review should I conduct? A proposed typology and guidance for systematic reviewers in the medical and health sciences. BMC medical research methodology , 18 (1), 1-9. Full Text Chong, S. W., & Reinders, H. (2021). A methodological review of qualitative research syntheses in CALL: The state-of-the-art. System , 103 , 102646. Full Text

• << Previous: Technology Assessment Review
• Next: Systematic Search and Review >>
• Last Updated: Aug 19, 2024 1:08 PM
• URL: https://unimelb.libguides.com/whichreview

Research Methods and Design

• Action Research
• Case Study Design

Literature Review

• Quantitative Research Methods
• Qualitative Research Methods
• Mixed Methods Study
• Indigenous Research and Ethics This link opens in a new window
• Identifying Empirical Research Articles This link opens in a new window
• Research Ethics and Quality
• Data Literacy
• Get Help with Writing Assignments

A literature review is a discussion of the literature (aka. the "research" or "scholarship") surrounding a certain topic. A good literature review doesn't simply summarize the existing material, but provides thoughtful synthesis and analysis. The purpose of a literature review is to orient your own work within an existing body of knowledge. A literature review may be written as a standalone piece or be included in a larger body of work.

You can read more about literature reviews, what they entail, and how to write one, using the resources below. 

Am I the only one struggling to write a literature review?

Dr. Zina O'Leary explains the misconceptions and struggles students often have with writing a literature review. She also provides step-by-step guidance on writing a persuasive literature review.

An Introduction to Literature Reviews

Dr. Eric Jensen, Professor of Sociology at the University of Warwick, and Dr. Charles Laurie, Director of Research at Verisk Maplecroft, explain how to write a literature review, and why researchers need to do so. Literature reviews can be stand-alone research or part of a larger project. They communicate the state of academic knowledge on a given topic, specifically detailing what is still unknown.

This is the first video in a whole series about literature reviews. You can find the rest of the series in our SAGE database, Research Methods:

Videos covering research methods and statistics

Identify Themes and Gaps in Literature (with real examples) | Scribbr

Finding connections between sources is key to organizing the arguments and structure of a good literature review. In this video, you'll learn how to identify themes, debates, and gaps between sources, using examples from real papers.

4 Tips for Writing a Literature Review's Intro, Body, and Conclusion | Scribbr

While each review will be unique in its structure--based on both the existing body of both literature and the overall goals of your own paper, dissertation, or research--this video from Scribbr does a good job simplifying the goals of writing a literature review for those who are new to the process. In this video, you’ll learn what to include in each section, as well as 4 tips for the main body illustrated with an example.

• Literature Review This chapter in SAGE's Encyclopedia of Research Design describes the types of literature reviews and scientific standards for conducting literature reviews.
• UNC Writing Center: Literature Reviews This handout from the Writing Center at UNC will explain what literature reviews are and offer insights into the form and construction of literature reviews in the humanities, social sciences, and sciences.
• Purdue OWL: Writing a Literature Review The overview of literature reviews comes from Purdue's Online Writing Lab. It explains the basic why, what, and how of writing a literature review.

Organizational Tools for Literature Reviews

One of the most daunting aspects of writing a literature review is organizing your research. There are a variety of strategies that you can use to help you in this task. We've highlighted just a few ways writers keep track of all that information! You can use a combination of these tools or come up with your own organizational process. The key is choosing something that works with your own learning style.

Citation Managers

Citation managers are great tools, in general, for organizing research, but can be especially helpful when writing a literature review. You can keep all of your research in one place, take notes, and organize your materials into different folders or categories. Read more about citations managers here:

• Manage Citations & Sources

Concept Mapping

Some writers use concept mapping (sometimes called flow or bubble charts or "mind maps") to help them visualize the ways in which the research they found connects.

There is no right or wrong way to make a concept map. There are a variety of online tools that can help you create a concept map or you can simply put pen to paper. To read more about concept mapping, take a look at the following help guides:

• Using Concept Maps From Williams College's guide, Literature Review: A Self-guided Tutorial

Synthesis Matrix

A synthesis matrix is is a chart you can use to help you organize your research into thematic categories. By organizing your research into a matrix, like the examples below, can help you visualize the ways in which your sources connect. 

• Walden University Writing Center: Literature Review Matrix Find a variety of literature review matrix examples and templates from Walden University.
• Writing A Literature Review and Using a Synthesis Matrix An example synthesis matrix created by NC State University Writing and Speaking Tutorial Service Tutors. If you would like a copy of this synthesis matrix in a different format, like a Word document, please ask a librarian. CC-BY-SA 3.0
• << Previous: Case Study Design
• Next: Quantitative Research Methods >>
• Last Updated: May 7, 2024 9:51 AM

CityU Home - CityU Catalog

• USC Libraries
• Research Guides

Organizing Your Social Sciences Research Paper

• 5. The Literature Review
• Purpose of Guide
• Design Flaws to Avoid
• Independent and Dependent Variables
• Glossary of Research Terms
• Reading Research Effectively
• Narrowing a Topic Idea
• Broadening a Topic Idea
• Extending the Timeliness of a Topic Idea
• Academic Writing Style
• Applying Critical Thinking
• Choosing a Title
• Making an Outline
• Paragraph Development
• Research Process Video Series
• Executive Summary
• The C.A.R.S. Model
• Background Information
• The Research Problem/Question
• Theoretical Framework
• Citation Tracking
• Content Alert Services
• Evaluating Sources
• Primary Sources
• Secondary Sources
• Tiertiary Sources
• Scholarly vs. Popular Publications
• Qualitative Methods
• Quantitative Methods
• Insiderness
• Using Non-Textual Elements
• Limitations of the Study
• Common Grammar Mistakes
• Writing Concisely
• Avoiding Plagiarism
• Footnotes or Endnotes?
• Further Readings
• Generative AI and Writing
• USC Libraries Tutorials and Other Guides
• Bibliography

A literature review surveys prior research published in books, scholarly articles, and any other sources relevant to a particular issue, area of research, or theory, and by so doing, provides a description, summary, and critical evaluation of these works in relation to the research problem being investigated. Literature reviews are designed to provide an overview of sources you have used in researching a particular topic and to demonstrate to your readers how your research fits within existing scholarship about the topic.

Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper . Fourth edition. Thousand Oaks, CA: SAGE, 2014.

Importance of a Good Literature Review

A literature review may consist of simply a summary of key sources, but in the social sciences, a literature review usually has an organizational pattern and combines both summary and synthesis, often within specific conceptual categories . A summary is a recap of the important information of the source, but a synthesis is a re-organization, or a reshuffling, of that information in a way that informs how you are planning to investigate a research problem. The analytical features of a literature review might:

• Give a new interpretation of old material or combine new with old interpretations,
• Trace the intellectual progression of the field, including major debates,
• Depending on the situation, evaluate the sources and advise the reader on the most pertinent or relevant research, or
• Usually in the conclusion of a literature review, identify where gaps exist in how a problem has been researched to date.

Given this, the purpose of a literature review is to:

• Place each work in the context of its contribution to understanding the research problem being studied.
• Describe the relationship of each work to the others under consideration.
• Identify new ways to interpret prior research.
• Reveal any gaps that exist in the literature.
• Resolve conflicts amongst seemingly contradictory previous studies.
• Identify areas of prior scholarship to prevent duplication of effort.
• Point the way in fulfilling a need for additional research.
• Locate your own research within the context of existing literature [very important].

Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper. 2nd ed. Thousand Oaks, CA: Sage, 2005; Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1998; Jesson, Jill. Doing Your Literature Review: Traditional and Systematic Techniques . Los Angeles, CA: SAGE, 2011; Knopf, Jeffrey W. "Doing a Literature Review." PS: Political Science and Politics 39 (January 2006): 127-132; Ridley, Diana. The Literature Review: A Step-by-Step Guide for Students . 2nd ed. Los Angeles, CA: SAGE, 2012.

Types of Literature Reviews

It is important to think of knowledge in a given field as consisting of three layers. First, there are the primary studies that researchers conduct and publish. Second are the reviews of those studies that summarize and offer new interpretations built from and often extending beyond the primary studies. Third, there are the perceptions, conclusions, opinion, and interpretations that are shared informally among scholars that become part of the body of epistemological traditions within the field.

In composing a literature review, it is important to note that it is often this third layer of knowledge that is cited as "true" even though it often has only a loose relationship to the primary studies and secondary literature reviews. Given this, while literature reviews are designed to provide an overview and synthesis of pertinent sources you have explored, there are a number of approaches you could adopt depending upon the type of analysis underpinning your study.

Argumentative Review This form examines literature selectively in order to support or refute an argument, deeply embedded assumption, or philosophical problem already established in the literature. The purpose is to develop a body of literature that establishes a contrarian viewpoint. Given the value-laden nature of some social science research [e.g., educational reform; immigration control], argumentative approaches to analyzing the literature can be a legitimate and important form of discourse. However, note that they can also introduce problems of bias when they are used to make summary claims of the sort found in systematic reviews [see below].

Integrative Review Considered a form of research that reviews, critiques, and synthesizes representative literature on a topic in an integrated way such that new frameworks and perspectives on the topic are generated. The body of literature includes all studies that address related or identical hypotheses or research problems. A well-done integrative review meets the same standards as primary research in regard to clarity, rigor, and replication. This is the most common form of review in the social sciences.

Historical Review Few things rest in isolation from historical precedent. Historical literature reviews focus on examining research throughout a period of time, often starting with the first time an issue, concept, theory, phenomena emerged in the literature, then tracing its evolution within the scholarship of a discipline. The purpose is to place research in a historical context to show familiarity with state-of-the-art developments and to identify the likely directions for future research.

Methodological Review A review does not always focus on what someone said [findings], but how they came about saying what they say [method of analysis]. Reviewing methods of analysis provides a framework of understanding at different levels [i.e. those of theory, substantive fields, research approaches, and data collection and analysis techniques], how researchers draw upon a wide variety of knowledge ranging from the conceptual level to practical documents for use in fieldwork in the areas of ontological and epistemological consideration, quantitative and qualitative integration, sampling, interviewing, data collection, and data analysis. This approach helps highlight ethical issues which you should be aware of and consider as you go through your own study.

Systematic Review This form consists of an overview of existing evidence pertinent to a clearly formulated research question, which uses pre-specified and standardized methods to identify and critically appraise relevant research, and to collect, report, and analyze data from the studies that are included in the review. The goal is to deliberately document, critically evaluate, and summarize scientifically all of the research about a clearly defined research problem . Typically it focuses on a very specific empirical question, often posed in a cause-and-effect form, such as "To what extent does A contribute to B?" This type of literature review is primarily applied to examining prior research studies in clinical medicine and allied health fields, but it is increasingly being used in the social sciences.

Theoretical Review The purpose of this form is to examine the corpus of theory that has accumulated in regard to an issue, concept, theory, phenomena. The theoretical literature review helps to establish what theories already exist, the relationships between them, to what degree the existing theories have been investigated, and to develop new hypotheses to be tested. Often this form is used to help establish a lack of appropriate theories or reveal that current theories are inadequate for explaining new or emerging research problems. The unit of analysis can focus on a theoretical concept or a whole theory or framework.

NOTE: Most often the literature review will incorporate some combination of types. For example, a review that examines literature supporting or refuting an argument, assumption, or philosophical problem related to the research problem will also need to include writing supported by sources that establish the history of these arguments in the literature.

Baumeister, Roy F. and Mark R. Leary. "Writing Narrative Literature Reviews."  Review of General Psychology 1 (September 1997): 311-320; Mark R. Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper . 2nd ed. Thousand Oaks, CA: Sage, 2005; Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1998; Kennedy, Mary M. "Defining a Literature." Educational Researcher 36 (April 2007): 139-147; Petticrew, Mark and Helen Roberts. Systematic Reviews in the Social Sciences: A Practical Guide . Malden, MA: Blackwell Publishers, 2006; Torracro, Richard. "Writing Integrative Literature Reviews: Guidelines and Examples." Human Resource Development Review 4 (September 2005): 356-367; Rocco, Tonette S. and Maria S. Plakhotnik. "Literature Reviews, Conceptual Frameworks, and Theoretical Frameworks: Terms, Functions, and Distinctions." Human Ressource Development Review 8 (March 2008): 120-130; Sutton, Anthea. Systematic Approaches to a Successful Literature Review . Los Angeles, CA: Sage Publications, 2016.

Structure and Writing Style

I.  Thinking About Your Literature Review

The structure of a literature review should include the following in support of understanding the research problem :

• An overview of the subject, issue, or theory under consideration, along with the objectives of the literature review,
• Division of works under review into themes or categories [e.g. works that support a particular position, those against, and those offering alternative approaches entirely],
• An explanation of how each work is similar to and how it varies from the others,
• Conclusions as to which pieces are best considered in their argument, are most convincing of their opinions, and make the greatest contribution to the understanding and development of their area of research.

The critical evaluation of each work should consider :

• Provenance -- what are the author's credentials? Are the author's arguments supported by evidence [e.g. primary historical material, case studies, narratives, statistics, recent scientific findings]?
• Methodology -- were the techniques used to identify, gather, and analyze the data appropriate to addressing the research problem? Was the sample size appropriate? Were the results effectively interpreted and reported?
• Objectivity -- is the author's perspective even-handed or prejudicial? Is contrary data considered or is certain pertinent information ignored to prove the author's point?
• Persuasiveness -- which of the author's theses are most convincing or least convincing?
• Validity -- are the author's arguments and conclusions convincing? Does the work ultimately contribute in any significant way to an understanding of the subject?

II.  Development of the Literature Review

Four Basic Stages of Writing 1.  Problem formulation -- which topic or field is being examined and what are its component issues? 2.  Literature search -- finding materials relevant to the subject being explored. 3.  Data evaluation -- determining which literature makes a significant contribution to the understanding of the topic. 4.  Analysis and interpretation -- discussing the findings and conclusions of pertinent literature.

Consider the following issues before writing the literature review: Clarify If your assignment is not specific about what form your literature review should take, seek clarification from your professor by asking these questions: 1.  Roughly how many sources would be appropriate to include? 2.  What types of sources should I review (books, journal articles, websites; scholarly versus popular sources)? 3.  Should I summarize, synthesize, or critique sources by discussing a common theme or issue? 4.  Should I evaluate the sources in any way beyond evaluating how they relate to understanding the research problem? 5.  Should I provide subheadings and other background information, such as definitions and/or a history? Find Models Use the exercise of reviewing the literature to examine how authors in your discipline or area of interest have composed their literature review sections. Read them to get a sense of the types of themes you might want to look for in your own research or to identify ways to organize your final review. The bibliography or reference section of sources you've already read, such as required readings in the course syllabus, are also excellent entry points into your own research. Narrow the Topic The narrower your topic, the easier it will be to limit the number of sources you need to read in order to obtain a good survey of relevant resources. Your professor will probably not expect you to read everything that's available about the topic, but you'll make the act of reviewing easier if you first limit scope of the research problem. A good strategy is to begin by searching the USC Libraries Catalog for recent books about the topic and review the table of contents for chapters that focuses on specific issues. You can also review the indexes of books to find references to specific issues that can serve as the focus of your research. For example, a book surveying the history of the Israeli-Palestinian conflict may include a chapter on the role Egypt has played in mediating the conflict, or look in the index for the pages where Egypt is mentioned in the text. Consider Whether Your Sources are Current Some disciplines require that you use information that is as current as possible. This is particularly true in disciplines in medicine and the sciences where research conducted becomes obsolete very quickly as new discoveries are made. However, when writing a review in the social sciences, a survey of the history of the literature may be required. In other words, a complete understanding the research problem requires you to deliberately examine how knowledge and perspectives have changed over time. Sort through other current bibliographies or literature reviews in the field to get a sense of what your discipline expects. You can also use this method to explore what is considered by scholars to be a "hot topic" and what is not.

III.  Ways to Organize Your Literature Review

Chronology of Events If your review follows the chronological method, you could write about the materials according to when they were published. This approach should only be followed if a clear path of research building on previous research can be identified and that these trends follow a clear chronological order of development. For example, a literature review that focuses on continuing research about the emergence of German economic power after the fall of the Soviet Union. By Publication Order your sources by publication chronology, then, only if the order demonstrates a more important trend. For instance, you could order a review of literature on environmental studies of brown fields if the progression revealed, for example, a change in the soil collection practices of the researchers who wrote and/or conducted the studies. Thematic [“conceptual categories”] A thematic literature review is the most common approach to summarizing prior research in the social and behavioral sciences. Thematic reviews are organized around a topic or issue, rather than the progression of time, although the progression of time may still be incorporated into a thematic review. For example, a review of the Internet’s impact on American presidential politics could focus on the development of online political satire. While the study focuses on one topic, the Internet’s impact on American presidential politics, it would still be organized chronologically reflecting technological developments in media. The difference in this example between a "chronological" and a "thematic" approach is what is emphasized the most: themes related to the role of the Internet in presidential politics. Note that more authentic thematic reviews tend to break away from chronological order. A review organized in this manner would shift between time periods within each section according to the point being made. Methodological A methodological approach focuses on the methods utilized by the researcher. For the Internet in American presidential politics project, one methodological approach would be to look at cultural differences between the portrayal of American presidents on American, British, and French websites. Or the review might focus on the fundraising impact of the Internet on a particular political party. A methodological scope will influence either the types of documents in the review or the way in which these documents are discussed.

Other Sections of Your Literature Review Once you've decided on the organizational method for your literature review, the sections you need to include in the paper should be easy to figure out because they arise from your organizational strategy. In other words, a chronological review would have subsections for each vital time period; a thematic review would have subtopics based upon factors that relate to the theme or issue. However, sometimes you may need to add additional sections that are necessary for your study, but do not fit in the organizational strategy of the body. What other sections you include in the body is up to you. However, only include what is necessary for the reader to locate your study within the larger scholarship about the research problem.

Here are examples of other sections, usually in the form of a single paragraph, you may need to include depending on the type of review you write:

• Current Situation : Information necessary to understand the current topic or focus of the literature review.
• Sources Used : Describes the methods and resources [e.g., databases] you used to identify the literature you reviewed.
• History : The chronological progression of the field, the research literature, or an idea that is necessary to understand the literature review, if the body of the literature review is not already a chronology.
• Selection Methods : Criteria you used to select (and perhaps exclude) sources in your literature review. For instance, you might explain that your review includes only peer-reviewed [i.e., scholarly] sources.
• Standards : Description of the way in which you present your information.
• Questions for Further Research : What questions about the field has the review sparked? How will you further your research as a result of the review?

IV.  Writing Your Literature Review

Once you've settled on how to organize your literature review, you're ready to write each section. When writing your review, keep in mind these issues.

Use Evidence A literature review section is, in this sense, just like any other academic research paper. Your interpretation of the available sources must be backed up with evidence [citations] that demonstrates that what you are saying is valid. Be Selective Select only the most important points in each source to highlight in the review. The type of information you choose to mention should relate directly to the research problem, whether it is thematic, methodological, or chronological. Related items that provide additional information, but that are not key to understanding the research problem, can be included in a list of further readings . Use Quotes Sparingly Some short quotes are appropriate if you want to emphasize a point, or if what an author stated cannot be easily paraphrased. Sometimes you may need to quote certain terminology that was coined by the author, is not common knowledge, or taken directly from the study. Do not use extensive quotes as a substitute for using your own words in reviewing the literature. Summarize and Synthesize Remember to summarize and synthesize your sources within each thematic paragraph as well as throughout the review. Recapitulate important features of a research study, but then synthesize it by rephrasing the study's significance and relating it to your own work and the work of others. Keep Your Own Voice While the literature review presents others' ideas, your voice [the writer's] should remain front and center. For example, weave references to other sources into what you are writing but maintain your own voice by starting and ending the paragraph with your own ideas and wording. Use Caution When Paraphrasing When paraphrasing a source that is not your own, be sure to represent the author's information or opinions accurately and in your own words. Even when paraphrasing an author’s work, you still must provide a citation to that work.

V.  Common Mistakes to Avoid

These are the most common mistakes made in reviewing social science research literature.

• Sources in your literature review do not clearly relate to the research problem;
• You do not take sufficient time to define and identify the most relevant sources to use in the literature review related to the research problem;
• Relies exclusively on secondary analytical sources rather than including relevant primary research studies or data;
• Uncritically accepts another researcher's findings and interpretations as valid, rather than examining critically all aspects of the research design and analysis;
• Does not describe the search procedures that were used in identifying the literature to review;
• Reports isolated statistical results rather than synthesizing them in chi-squared or meta-analytic methods; and,
• Only includes research that validates assumptions and does not consider contrary findings and alternative interpretations found in the literature.

Cook, Kathleen E. and Elise Murowchick. “Do Literature Review Skills Transfer from One Course to Another?” Psychology Learning and Teaching 13 (March 2014): 3-11; Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper . 2nd ed. Thousand Oaks, CA: Sage, 2005; Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1998; Jesson, Jill. Doing Your Literature Review: Traditional and Systematic Techniques . London: SAGE, 2011; Literature Review Handout. Online Writing Center. Liberty University; Literature Reviews. The Writing Center. University of North Carolina; Onwuegbuzie, Anthony J. and Rebecca Frels. Seven Steps to a Comprehensive Literature Review: A Multimodal and Cultural Approach . Los Angeles, CA: SAGE, 2016; Ridley, Diana. The Literature Review: A Step-by-Step Guide for Students . 2nd ed. Los Angeles, CA: SAGE, 2012; Randolph, Justus J. “A Guide to Writing the Dissertation Literature Review." Practical Assessment, Research, and Evaluation. vol. 14, June 2009; Sutton, Anthea. Systematic Approaches to a Successful Literature Review . Los Angeles, CA: Sage Publications, 2016; Taylor, Dena. The Literature Review: A Few Tips On Conducting It. University College Writing Centre. University of Toronto; Writing a Literature Review. Academic Skills Centre. University of Canberra.

Writing Tip

Break Out of Your Disciplinary Box!

Thinking interdisciplinarily about a research problem can be a rewarding exercise in applying new ideas, theories, or concepts to an old problem. For example, what might cultural anthropologists say about the continuing conflict in the Middle East? In what ways might geographers view the need for better distribution of social service agencies in large cities than how social workers might study the issue? You don’t want to substitute a thorough review of core research literature in your discipline for studies conducted in other fields of study. However, particularly in the social sciences, thinking about research problems from multiple vectors is a key strategy for finding new solutions to a problem or gaining a new perspective. Consult with a librarian about identifying research databases in other disciplines; almost every field of study has at least one comprehensive database devoted to indexing its research literature.

Frodeman, Robert. The Oxford Handbook of Interdisciplinarity . New York: Oxford University Press, 2010.

Another Writing Tip

Don't Just Review for Content!

While conducting a review of the literature, maximize the time you devote to writing this part of your paper by thinking broadly about what you should be looking for and evaluating. Review not just what scholars are saying, but how are they saying it. Some questions to ask:

• How are they organizing their ideas?
• What methods have they used to study the problem?
• What theories have been used to explain, predict, or understand their research problem?
• What sources have they cited to support their conclusions?
• How have they used non-textual elements [e.g., charts, graphs, figures, etc.] to illustrate key points?

When you begin to write your literature review section, you'll be glad you dug deeper into how the research was designed and constructed because it establishes a means for developing more substantial analysis and interpretation of the research problem.

Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1 998.

Yet Another Writing Tip

When Do I Know I Can Stop Looking and Move On?

Here are several strategies you can utilize to assess whether you've thoroughly reviewed the literature:

• Look for repeating patterns in the research findings . If the same thing is being said, just by different people, then this likely demonstrates that the research problem has hit a conceptual dead end. At this point consider: Does your study extend current research?  Does it forge a new path? Or, does is merely add more of the same thing being said?
• Look at sources the authors cite to in their work . If you begin to see the same researchers cited again and again, then this is often an indication that no new ideas have been generated to address the research problem.
• Search Google Scholar to identify who has subsequently cited leading scholars already identified in your literature review [see next sub-tab]. This is called citation tracking and there are a number of sources that can help you identify who has cited whom, particularly scholars from outside of your discipline. Here again, if the same authors are being cited again and again, this may indicate no new literature has been written on the topic.

Onwuegbuzie, Anthony J. and Rebecca Frels. Seven Steps to a Comprehensive Literature Review: A Multimodal and Cultural Approach . Los Angeles, CA: Sage, 2016; Sutton, Anthea. Systematic Approaches to a Successful Literature Review . Los Angeles, CA: Sage Publications, 2016.

• << Previous: Theoretical Framework
• Next: Citation Tracking >>
• Last Updated: Sep 4, 2024 9:40 AM
• URL: https://libguides.usc.edu/writingguide

Research Methods: Literature Reviews

• Annotated Bibliographies
• Literature Reviews
• Scoping Reviews
• Systematic Reviews
• Scholarship of Teaching and Learning
• Persuasive Arguments
• Subject Specific Methodology

A literature review involves researching, reading, analyzing, evaluating, and summarizing scholarly literature (typically journals and articles) about a specific topic. The results of a literature review may be an entire report or article OR may be part of a article, thesis, dissertation, or grant proposal. A literature review helps the author learn about the history and nature of their topic, and identify research gaps and problems.

Steps & Elements

Problem formulation

• Determine your topic and its components by asking a question
• Research: locate literature related to your topic to identify the gap(s) that can be addressed
• Read: read the articles or other sources of information
• Analyze: assess the findings for relevancy
• Evaluating: determine how the article are relevant to your research and what are the key findings
• Synthesis: write about the key findings and how it is relevant to your research

Elements of a Literature Review

• Summarize subject, issue or theory under consideration, along with objectives of the review
• Divide works under review into categories (e.g. those in support of a particular position, those against, those offering alternative theories entirely)
• Explain how each work is similar to and how it varies from the others
• Conclude which pieces are best considered in their argument, are most convincing of their opinions, and make the greatest contribution to the understanding and development of an area of research

Writing a Literature Review Resources

• How to Write a Literature Review From the Wesleyan University Library
• Write a Literature Review From the University of California Santa Cruz Library. A Brief overview of a literature review, includes a list of stages for writing a lit review.
• Literature Reviews From the University of North Carolina Writing Center. Detailed information about writing a literature review.
• Undertaking a literature review: a step-by-step approach Cronin, P., Ryan, F., & Coughan, M. (2008). Undertaking a literature review: A step-by-step approach. British Journal of Nursing, 17(1), p.38-43

Literature Review Tutorial

• << Previous: Annotated Bibliographies
• Next: Scoping Reviews >>
• Last Updated: Jul 8, 2024 3:13 PM
• URL: https://guides.auraria.edu/researchmethods

1100 Lawrence Street Denver, CO 80204 303-315-7700 Ask Us Directions

• Privacy Policy

Home » Literature Review – Types Writing Guide and Examples

Literature Review – Types Writing Guide and Examples

Table of Contents

Literature Review

Definition:

A literature review is a comprehensive and critical analysis of the existing literature on a particular topic or research question. It involves identifying, evaluating, and synthesizing relevant literature, including scholarly articles, books, and other sources, to provide a summary and critical assessment of what is known about the topic.

Types of Literature Review

Types of Literature Review are as follows:

• Narrative literature review : This type of review involves a comprehensive summary and critical analysis of the available literature on a particular topic or research question. It is often used as an introductory section of a research paper.
• Systematic literature review: This is a rigorous and structured review that follows a pre-defined protocol to identify, evaluate, and synthesize all relevant studies on a specific research question. It is often used in evidence-based practice and systematic reviews.
• Meta-analysis: This is a quantitative review that uses statistical methods to combine data from multiple studies to derive a summary effect size. It provides a more precise estimate of the overall effect than any individual study.
• Scoping review: This is a preliminary review that aims to map the existing literature on a broad topic area to identify research gaps and areas for further investigation.
• Critical literature review : This type of review evaluates the strengths and weaknesses of the existing literature on a particular topic or research question. It aims to provide a critical analysis of the literature and identify areas where further research is needed.
• Conceptual literature review: This review synthesizes and integrates theories and concepts from multiple sources to provide a new perspective on a particular topic. It aims to provide a theoretical framework for understanding a particular research question.
• Rapid literature review: This is a quick review that provides a snapshot of the current state of knowledge on a specific research question or topic. It is often used when time and resources are limited.
• Thematic literature review : This review identifies and analyzes common themes and patterns across a body of literature on a particular topic. It aims to provide a comprehensive overview of the literature and identify key themes and concepts.
• Realist literature review: This review is often used in social science research and aims to identify how and why certain interventions work in certain contexts. It takes into account the context and complexities of real-world situations.
• State-of-the-art literature review : This type of review provides an overview of the current state of knowledge in a particular field, highlighting the most recent and relevant research. It is often used in fields where knowledge is rapidly evolving, such as technology or medicine.
• Integrative literature review: This type of review synthesizes and integrates findings from multiple studies on a particular topic to identify patterns, themes, and gaps in the literature. It aims to provide a comprehensive understanding of the current state of knowledge on a particular topic.
• Umbrella literature review : This review is used to provide a broad overview of a large and diverse body of literature on a particular topic. It aims to identify common themes and patterns across different areas of research.
• Historical literature review: This type of review examines the historical development of research on a particular topic or research question. It aims to provide a historical context for understanding the current state of knowledge on a particular topic.
• Problem-oriented literature review : This review focuses on a specific problem or issue and examines the literature to identify potential solutions or interventions. It aims to provide practical recommendations for addressing a particular problem or issue.
• Mixed-methods literature review : This type of review combines quantitative and qualitative methods to synthesize and analyze the available literature on a particular topic. It aims to provide a more comprehensive understanding of the research question by combining different types of evidence.

Parts of Literature Review

Parts of a literature review are as follows:

Introduction

The introduction of a literature review typically provides background information on the research topic and why it is important. It outlines the objectives of the review, the research question or hypothesis, and the scope of the review.

Literature Search

This section outlines the search strategy and databases used to identify relevant literature. The search terms used, inclusion and exclusion criteria, and any limitations of the search are described.

Literature Analysis

The literature analysis is the main body of the literature review. This section summarizes and synthesizes the literature that is relevant to the research question or hypothesis. The review should be organized thematically, chronologically, or by methodology, depending on the research objectives.

Critical Evaluation

Critical evaluation involves assessing the quality and validity of the literature. This includes evaluating the reliability and validity of the studies reviewed, the methodology used, and the strength of the evidence.

The conclusion of the literature review should summarize the main findings, identify any gaps in the literature, and suggest areas for future research. It should also reiterate the importance of the research question or hypothesis and the contribution of the literature review to the overall research project.

The references list includes all the sources cited in the literature review, and follows a specific referencing style (e.g., APA, MLA, Harvard).

How to write Literature Review

Here are some steps to follow when writing a literature review:

• Define your research question or topic : Before starting your literature review, it is essential to define your research question or topic. This will help you identify relevant literature and determine the scope of your review.
• Conduct a comprehensive search: Use databases and search engines to find relevant literature. Look for peer-reviewed articles, books, and other academic sources that are relevant to your research question or topic.
• Evaluate the sources: Once you have found potential sources, evaluate them critically to determine their relevance, credibility, and quality. Look for recent publications, reputable authors, and reliable sources of data and evidence.
• Organize your sources: Group the sources by theme, method, or research question. This will help you identify similarities and differences among the literature, and provide a structure for your literature review.
• Analyze and synthesize the literature : Analyze each source in depth, identifying the key findings, methodologies, and conclusions. Then, synthesize the information from the sources, identifying patterns and themes in the literature.
• Write the literature review : Start with an introduction that provides an overview of the topic and the purpose of the literature review. Then, organize the literature according to your chosen structure, and analyze and synthesize the sources. Finally, provide a conclusion that summarizes the key findings of the literature review, identifies gaps in knowledge, and suggests areas for future research.
• Edit and proofread: Once you have written your literature review, edit and proofread it carefully to ensure that it is well-organized, clear, and concise.

Examples of Literature Review

Here’s an example of how a literature review can be conducted for a thesis on the topic of “ The Impact of Social Media on Teenagers’ Mental Health”:

• Start by identifying the key terms related to your research topic. In this case, the key terms are “social media,” “teenagers,” and “mental health.”
• Use academic databases like Google Scholar, JSTOR, or PubMed to search for relevant articles, books, and other publications. Use these keywords in your search to narrow down your results.
• Evaluate the sources you find to determine if they are relevant to your research question. You may want to consider the publication date, author’s credentials, and the journal or book publisher.
• Begin reading and taking notes on each source, paying attention to key findings, methodologies used, and any gaps in the research.
• Organize your findings into themes or categories. For example, you might categorize your sources into those that examine the impact of social media on self-esteem, those that explore the effects of cyberbullying, and those that investigate the relationship between social media use and depression.
• Synthesize your findings by summarizing the key themes and highlighting any gaps or inconsistencies in the research. Identify areas where further research is needed.
• Use your literature review to inform your research questions and hypotheses for your thesis.

For example, after conducting a literature review on the impact of social media on teenagers’ mental health, a thesis might look like this:

“Using a mixed-methods approach, this study aims to investigate the relationship between social media use and mental health outcomes in teenagers. Specifically, the study will examine the effects of cyberbullying, social comparison, and excessive social media use on self-esteem, anxiety, and depression. Through an analysis of survey data and qualitative interviews with teenagers, the study will provide insight into the complex relationship between social media use and mental health outcomes, and identify strategies for promoting positive mental health outcomes in young people.”

Reference: Smith, J., Jones, M., & Lee, S. (2019). The effects of social media use on adolescent mental health: A systematic review. Journal of Adolescent Health, 65(2), 154-165. doi:10.1016/j.jadohealth.2019.03.024

Reference Example: Author, A. A., Author, B. B., & Author, C. C. (Year). Title of article. Title of Journal, volume number(issue number), page range. doi:0000000/000000000000 or URL

Applications of Literature Review

some applications of literature review in different fields:

• Social Sciences: In social sciences, literature reviews are used to identify gaps in existing research, to develop research questions, and to provide a theoretical framework for research. Literature reviews are commonly used in fields such as sociology, psychology, anthropology, and political science.
• Natural Sciences: In natural sciences, literature reviews are used to summarize and evaluate the current state of knowledge in a particular field or subfield. Literature reviews can help researchers identify areas where more research is needed and provide insights into the latest developments in a particular field. Fields such as biology, chemistry, and physics commonly use literature reviews.
• Health Sciences: In health sciences, literature reviews are used to evaluate the effectiveness of treatments, identify best practices, and determine areas where more research is needed. Literature reviews are commonly used in fields such as medicine, nursing, and public health.
• Humanities: In humanities, literature reviews are used to identify gaps in existing knowledge, develop new interpretations of texts or cultural artifacts, and provide a theoretical framework for research. Literature reviews are commonly used in fields such as history, literary studies, and philosophy.

Role of Literature Review in Research

Here are some applications of literature review in research:

• Identifying Research Gaps : Literature review helps researchers identify gaps in existing research and literature related to their research question. This allows them to develop new research questions and hypotheses to fill those gaps.
• Developing Theoretical Framework: Literature review helps researchers develop a theoretical framework for their research. By analyzing and synthesizing existing literature, researchers can identify the key concepts, theories, and models that are relevant to their research.
• Selecting Research Methods : Literature review helps researchers select appropriate research methods and techniques based on previous research. It also helps researchers to identify potential biases or limitations of certain methods and techniques.
• Data Collection and Analysis: Literature review helps researchers in data collection and analysis by providing a foundation for the development of data collection instruments and methods. It also helps researchers to identify relevant data sources and identify potential data analysis techniques.
• Communicating Results: Literature review helps researchers to communicate their results effectively by providing a context for their research. It also helps to justify the significance of their findings in relation to existing research and literature.

Purpose of Literature Review

Some of the specific purposes of a literature review are as follows:

• To provide context: A literature review helps to provide context for your research by situating it within the broader body of literature on the topic.
• To identify gaps and inconsistencies: A literature review helps to identify areas where further research is needed or where there are inconsistencies in the existing literature.
• To synthesize information: A literature review helps to synthesize the information from multiple sources and present a coherent and comprehensive picture of the current state of knowledge on the topic.
• To identify key concepts and theories : A literature review helps to identify key concepts and theories that are relevant to your research question and provide a theoretical framework for your study.
• To inform research design: A literature review can inform the design of your research study by identifying appropriate research methods, data sources, and research questions.

Characteristics of Literature Review

Some Characteristics of Literature Review are as follows:

• Identifying gaps in knowledge: A literature review helps to identify gaps in the existing knowledge and research on a specific topic or research question. By analyzing and synthesizing the literature, you can identify areas where further research is needed and where new insights can be gained.
• Establishing the significance of your research: A literature review helps to establish the significance of your own research by placing it in the context of existing research. By demonstrating the relevance of your research to the existing literature, you can establish its importance and value.
• Informing research design and methodology : A literature review helps to inform research design and methodology by identifying the most appropriate research methods, techniques, and instruments. By reviewing the literature, you can identify the strengths and limitations of different research methods and techniques, and select the most appropriate ones for your own research.
• Supporting arguments and claims: A literature review provides evidence to support arguments and claims made in academic writing. By citing and analyzing the literature, you can provide a solid foundation for your own arguments and claims.
• I dentifying potential collaborators and mentors: A literature review can help identify potential collaborators and mentors by identifying researchers and practitioners who are working on related topics or using similar methods. By building relationships with these individuals, you can gain valuable insights and support for your own research and practice.
• Keeping up-to-date with the latest research : A literature review helps to keep you up-to-date with the latest research on a specific topic or research question. By regularly reviewing the literature, you can stay informed about the latest findings and developments in your field.

Advantages of Literature Review

There are several advantages to conducting a literature review as part of a research project, including:

• Establishing the significance of the research : A literature review helps to establish the significance of the research by demonstrating the gap or problem in the existing literature that the study aims to address.
• Identifying key concepts and theories: A literature review can help to identify key concepts and theories that are relevant to the research question, and provide a theoretical framework for the study.
• Supporting the research methodology : A literature review can inform the research methodology by identifying appropriate research methods, data sources, and research questions.
• Providing a comprehensive overview of the literature : A literature review provides a comprehensive overview of the current state of knowledge on a topic, allowing the researcher to identify key themes, debates, and areas of agreement or disagreement.
• Identifying potential research questions: A literature review can help to identify potential research questions and areas for further investigation.
• Avoiding duplication of research: A literature review can help to avoid duplication of research by identifying what has already been done on a topic, and what remains to be done.
• Enhancing the credibility of the research : A literature review helps to enhance the credibility of the research by demonstrating the researcher’s knowledge of the existing literature and their ability to situate their research within a broader context.

Limitations of Literature Review

Limitations of Literature Review are as follows:

• Limited scope : Literature reviews can only cover the existing literature on a particular topic, which may be limited in scope or depth.
• Publication bias : Literature reviews may be influenced by publication bias, which occurs when researchers are more likely to publish positive results than negative ones. This can lead to an incomplete or biased picture of the literature.
• Quality of sources : The quality of the literature reviewed can vary widely, and not all sources may be reliable or valid.
• Time-limited: Literature reviews can become quickly outdated as new research is published, making it difficult to keep up with the latest developments in a field.
• Subjective interpretation : Literature reviews can be subjective, and the interpretation of the findings can vary depending on the researcher’s perspective or bias.
• Lack of original data : Literature reviews do not generate new data, but rather rely on the analysis of existing studies.
• Risk of plagiarism: It is important to ensure that literature reviews do not inadvertently contain plagiarism, which can occur when researchers use the work of others without proper attribution.

About the author

Muhammad Hassan

Researcher, Academic Writer, Web developer

You may also like

Data Interpretation – Process, Methods and...

Research Techniques – Methods, Types and Examples

Institutional Review Board – Application Sample...

Appendix in Research Paper – Examples and...

Future Research – Thesis Guide

Dissertation – Format, Example and Template

Warning: The NCBI web site requires JavaScript to function. more...

An official website of the United States government

The .gov means it's official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you're on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• Browse Titles

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Lau F, Kuziemsky C, editors. Handbook of eHealth Evaluation: An Evidence-based Approach [Internet]. Victoria (BC): University of Victoria; 2017 Feb 27.

Handbook of eHealth Evaluation: An Evidence-based Approach [Internet].

Chapter 9 methods for literature reviews.

Guy Paré and Spyros Kitsiou .

9.1. Introduction

Literature reviews play a critical role in scholarship because science remains, first and foremost, a cumulative endeavour ( vom Brocke et al., 2009 ). As in any academic discipline, rigorous knowledge syntheses are becoming indispensable in keeping up with an exponentially growing eHealth literature, assisting practitioners, academics, and graduate students in finding, evaluating, and synthesizing the contents of many empirical and conceptual papers. Among other methods, literature reviews are essential for: (a) identifying what has been written on a subject or topic; (b) determining the extent to which a specific research area reveals any interpretable trends or patterns; (c) aggregating empirical findings related to a narrow research question to support evidence-based practice; (d) generating new frameworks and theories; and (e) identifying topics or questions requiring more investigation ( Paré, Trudel, Jaana, & Kitsiou, 2015 ).

Literature reviews can take two major forms. The most prevalent one is the “literature review” or “background” section within a journal paper or a chapter in a graduate thesis. This section synthesizes the extant literature and usually identifies the gaps in knowledge that the empirical study addresses ( Sylvester, Tate, & Johnstone, 2013 ). It may also provide a theoretical foundation for the proposed study, substantiate the presence of the research problem, justify the research as one that contributes something new to the cumulated knowledge, or validate the methods and approaches for the proposed study ( Hart, 1998 ; Levy & Ellis, 2006 ).

The second form of literature review, which is the focus of this chapter, constitutes an original and valuable work of research in and of itself ( Paré et al., 2015 ). Rather than providing a base for a researcher’s own work, it creates a solid starting point for all members of the community interested in a particular area or topic ( Mulrow, 1987 ). The so-called “review article” is a journal-length paper which has an overarching purpose to synthesize the literature in a field, without collecting or analyzing any primary data ( Green, Johnson, & Adams, 2006 ).

When appropriately conducted, review articles represent powerful information sources for practitioners looking for state-of-the art evidence to guide their decision-making and work practices ( Paré et al., 2015 ). Further, high-quality reviews become frequently cited pieces of work which researchers seek out as a first clear outline of the literature when undertaking empirical studies ( Cooper, 1988 ; Rowe, 2014 ). Scholars who track and gauge the impact of articles have found that review papers are cited and downloaded more often than any other type of published article ( Cronin, Ryan, & Coughlan, 2008 ; Montori, Wilczynski, Morgan, Haynes, & Hedges, 2003 ; Patsopoulos, Analatos, & Ioannidis, 2005 ). The reason for their popularity may be the fact that reading the review enables one to have an overview, if not a detailed knowledge of the area in question, as well as references to the most useful primary sources ( Cronin et al., 2008 ). Although they are not easy to conduct, the commitment to complete a review article provides a tremendous service to one’s academic community ( Paré et al., 2015 ; Petticrew & Roberts, 2006 ). Most, if not all, peer-reviewed journals in the fields of medical informatics publish review articles of some type.

The main objectives of this chapter are fourfold: (a) to provide an overview of the major steps and activities involved in conducting a stand-alone literature review; (b) to describe and contrast the different types of review articles that can contribute to the eHealth knowledge base; (c) to illustrate each review type with one or two examples from the eHealth literature; and (d) to provide a series of recommendations for prospective authors of review articles in this domain.

9.2. Overview of the Literature Review Process and Steps

As explained in Templier and Paré (2015) , there are six generic steps involved in conducting a review article:

• formulating the research question(s) and objective(s),
• searching the extant literature,
• screening for inclusion,
• assessing the quality of primary studies,
• extracting data, and
• analyzing data.

Although these steps are presented here in sequential order, one must keep in mind that the review process can be iterative and that many activities can be initiated during the planning stage and later refined during subsequent phases ( Finfgeld-Connett & Johnson, 2013 ; Kitchenham & Charters, 2007 ).

Formulating the research question(s) and objective(s): As a first step, members of the review team must appropriately justify the need for the review itself ( Petticrew & Roberts, 2006 ), identify the review’s main objective(s) ( Okoli & Schabram, 2010 ), and define the concepts or variables at the heart of their synthesis ( Cooper & Hedges, 2009 ; Webster & Watson, 2002 ). Importantly, they also need to articulate the research question(s) they propose to investigate ( Kitchenham & Charters, 2007 ). In this regard, we concur with Jesson, Matheson, and Lacey (2011) that clearly articulated research questions are key ingredients that guide the entire review methodology; they underscore the type of information that is needed, inform the search for and selection of relevant literature, and guide or orient the subsequent analysis. Searching the extant literature: The next step consists of searching the literature and making decisions about the suitability of material to be considered in the review ( Cooper, 1988 ). There exist three main coverage strategies. First, exhaustive coverage means an effort is made to be as comprehensive as possible in order to ensure that all relevant studies, published and unpublished, are included in the review and, thus, conclusions are based on this all-inclusive knowledge base. The second type of coverage consists of presenting materials that are representative of most other works in a given field or area. Often authors who adopt this strategy will search for relevant articles in a small number of top-tier journals in a field ( Paré et al., 2015 ). In the third strategy, the review team concentrates on prior works that have been central or pivotal to a particular topic. This may include empirical studies or conceptual papers that initiated a line of investigation, changed how problems or questions were framed, introduced new methods or concepts, or engendered important debate ( Cooper, 1988 ). Screening for inclusion: The following step consists of evaluating the applicability of the material identified in the preceding step ( Levy & Ellis, 2006 ; vom Brocke et al., 2009 ). Once a group of potential studies has been identified, members of the review team must screen them to determine their relevance ( Petticrew & Roberts, 2006 ). A set of predetermined rules provides a basis for including or excluding certain studies. This exercise requires a significant investment on the part of researchers, who must ensure enhanced objectivity and avoid biases or mistakes. As discussed later in this chapter, for certain types of reviews there must be at least two independent reviewers involved in the screening process and a procedure to resolve disagreements must also be in place ( Liberati et al., 2009 ; Shea et al., 2009 ). Assessing the quality of primary studies: In addition to screening material for inclusion, members of the review team may need to assess the scientific quality of the selected studies, that is, appraise the rigour of the research design and methods. Such formal assessment, which is usually conducted independently by at least two coders, helps members of the review team refine which studies to include in the final sample, determine whether or not the differences in quality may affect their conclusions, or guide how they analyze the data and interpret the findings ( Petticrew & Roberts, 2006 ). Ascribing quality scores to each primary study or considering through domain-based evaluations which study components have or have not been designed and executed appropriately makes it possible to reflect on the extent to which the selected study addresses possible biases and maximizes validity ( Shea et al., 2009 ). Extracting data: The following step involves gathering or extracting applicable information from each primary study included in the sample and deciding what is relevant to the problem of interest ( Cooper & Hedges, 2009 ). Indeed, the type of data that should be recorded mainly depends on the initial research questions ( Okoli & Schabram, 2010 ). However, important information may also be gathered about how, when, where and by whom the primary study was conducted, the research design and methods, or qualitative/quantitative results ( Cooper & Hedges, 2009 ). Analyzing and synthesizing data : As a final step, members of the review team must collate, summarize, aggregate, organize, and compare the evidence extracted from the included studies. The extracted data must be presented in a meaningful way that suggests a new contribution to the extant literature ( Jesson et al., 2011 ). Webster and Watson (2002) warn researchers that literature reviews should be much more than lists of papers and should provide a coherent lens to make sense of extant knowledge on a given topic. There exist several methods and techniques for synthesizing quantitative (e.g., frequency analysis, meta-analysis) and qualitative (e.g., grounded theory, narrative analysis, meta-ethnography) evidence ( Dixon-Woods, Agarwal, Jones, Young, & Sutton, 2005 ; Thomas & Harden, 2008 ).

9.3. Types of Review Articles and Brief Illustrations

EHealth researchers have at their disposal a number of approaches and methods for making sense out of existing literature, all with the purpose of casting current research findings into historical contexts or explaining contradictions that might exist among a set of primary research studies conducted on a particular topic. Our classification scheme is largely inspired from Paré and colleagues’ (2015) typology. Below we present and illustrate those review types that we feel are central to the growth and development of the eHealth domain.

9.3.1. Narrative Reviews

The narrative review is the “traditional” way of reviewing the extant literature and is skewed towards a qualitative interpretation of prior knowledge ( Sylvester et al., 2013 ). Put simply, a narrative review attempts to summarize or synthesize what has been written on a particular topic but does not seek generalization or cumulative knowledge from what is reviewed ( Davies, 2000 ; Green et al., 2006 ). Instead, the review team often undertakes the task of accumulating and synthesizing the literature to demonstrate the value of a particular point of view ( Baumeister & Leary, 1997 ). As such, reviewers may selectively ignore or limit the attention paid to certain studies in order to make a point. In this rather unsystematic approach, the selection of information from primary articles is subjective, lacks explicit criteria for inclusion and can lead to biased interpretations or inferences ( Green et al., 2006 ). There are several narrative reviews in the particular eHealth domain, as in all fields, which follow such an unstructured approach ( Silva et al., 2015 ; Paul et al., 2015 ).

Despite these criticisms, this type of review can be very useful in gathering together a volume of literature in a specific subject area and synthesizing it. As mentioned above, its primary purpose is to provide the reader with a comprehensive background for understanding current knowledge and highlighting the significance of new research ( Cronin et al., 2008 ). Faculty like to use narrative reviews in the classroom because they are often more up to date than textbooks, provide a single source for students to reference, and expose students to peer-reviewed literature ( Green et al., 2006 ). For researchers, narrative reviews can inspire research ideas by identifying gaps or inconsistencies in a body of knowledge, thus helping researchers to determine research questions or formulate hypotheses. Importantly, narrative reviews can also be used as educational articles to bring practitioners up to date with certain topics of issues ( Green et al., 2006 ).

Recently, there have been several efforts to introduce more rigour in narrative reviews that will elucidate common pitfalls and bring changes into their publication standards. Information systems researchers, among others, have contributed to advancing knowledge on how to structure a “traditional” review. For instance, Levy and Ellis (2006) proposed a generic framework for conducting such reviews. Their model follows the systematic data processing approach comprised of three steps, namely: (a) literature search and screening; (b) data extraction and analysis; and (c) writing the literature review. They provide detailed and very helpful instructions on how to conduct each step of the review process. As another methodological contribution, vom Brocke et al. (2009) offered a series of guidelines for conducting literature reviews, with a particular focus on how to search and extract the relevant body of knowledge. Last, Bandara, Miskon, and Fielt (2011) proposed a structured, predefined and tool-supported method to identify primary studies within a feasible scope, extract relevant content from identified articles, synthesize and analyze the findings, and effectively write and present the results of the literature review. We highly recommend that prospective authors of narrative reviews consult these useful sources before embarking on their work.

Darlow and Wen (2015) provide a good example of a highly structured narrative review in the eHealth field. These authors synthesized published articles that describe the development process of mobile health (m-health) interventions for patients’ cancer care self-management. As in most narrative reviews, the scope of the research questions being investigated is broad: (a) how development of these systems are carried out; (b) which methods are used to investigate these systems; and (c) what conclusions can be drawn as a result of the development of these systems. To provide clear answers to these questions, a literature search was conducted on six electronic databases and Google Scholar . The search was performed using several terms and free text words, combining them in an appropriate manner. Four inclusion and three exclusion criteria were utilized during the screening process. Both authors independently reviewed each of the identified articles to determine eligibility and extract study information. A flow diagram shows the number of studies identified, screened, and included or excluded at each stage of study selection. In terms of contributions, this review provides a series of practical recommendations for m-health intervention development.

9.3.2. Descriptive or Mapping Reviews

The primary goal of a descriptive review is to determine the extent to which a body of knowledge in a particular research topic reveals any interpretable pattern or trend with respect to pre-existing propositions, theories, methodologies or findings ( King & He, 2005 ; Paré et al., 2015 ). In contrast with narrative reviews, descriptive reviews follow a systematic and transparent procedure, including searching, screening and classifying studies ( Petersen, Vakkalanka, & Kuzniarz, 2015 ). Indeed, structured search methods are used to form a representative sample of a larger group of published works ( Paré et al., 2015 ). Further, authors of descriptive reviews extract from each study certain characteristics of interest, such as publication year, research methods, data collection techniques, and direction or strength of research outcomes (e.g., positive, negative, or non-significant) in the form of frequency analysis to produce quantitative results ( Sylvester et al., 2013 ). In essence, each study included in a descriptive review is treated as the unit of analysis and the published literature as a whole provides a database from which the authors attempt to identify any interpretable trends or draw overall conclusions about the merits of existing conceptualizations, propositions, methods or findings ( Paré et al., 2015 ). In doing so, a descriptive review may claim that its findings represent the state of the art in a particular domain ( King & He, 2005 ).

In the fields of health sciences and medical informatics, reviews that focus on examining the range, nature and evolution of a topic area are described by Anderson, Allen, Peckham, and Goodwin (2008) as mapping reviews . Like descriptive reviews, the research questions are generic and usually relate to publication patterns and trends. There is no preconceived plan to systematically review all of the literature although this can be done. Instead, researchers often present studies that are representative of most works published in a particular area and they consider a specific time frame to be mapped.

An example of this approach in the eHealth domain is offered by DeShazo, Lavallie, and Wolf (2009). The purpose of this descriptive or mapping review was to characterize publication trends in the medical informatics literature over a 20-year period (1987 to 2006). To achieve this ambitious objective, the authors performed a bibliometric analysis of medical informatics citations indexed in medline using publication trends, journal frequencies, impact factors, Medical Subject Headings (MeSH) term frequencies, and characteristics of citations. Findings revealed that there were over 77,000 medical informatics articles published during the covered period in numerous journals and that the average annual growth rate was 12%. The MeSH term analysis also suggested a strong interdisciplinary trend. Finally, average impact scores increased over time with two notable growth periods. Overall, patterns in research outputs that seem to characterize the historic trends and current components of the field of medical informatics suggest it may be a maturing discipline (DeShazo et al., 2009).

9.3.3. Scoping Reviews

Scoping reviews attempt to provide an initial indication of the potential size and nature of the extant literature on an emergent topic (Arksey & O’Malley, 2005; Daudt, van Mossel, & Scott, 2013 ; Levac, Colquhoun, & O’Brien, 2010). A scoping review may be conducted to examine the extent, range and nature of research activities in a particular area, determine the value of undertaking a full systematic review (discussed next), or identify research gaps in the extant literature ( Paré et al., 2015 ). In line with their main objective, scoping reviews usually conclude with the presentation of a detailed research agenda for future works along with potential implications for both practice and research.

Unlike narrative and descriptive reviews, the whole point of scoping the field is to be as comprehensive as possible, including grey literature (Arksey & O’Malley, 2005). Inclusion and exclusion criteria must be established to help researchers eliminate studies that are not aligned with the research questions. It is also recommended that at least two independent coders review abstracts yielded from the search strategy and then the full articles for study selection ( Daudt et al., 2013 ). The synthesized evidence from content or thematic analysis is relatively easy to present in tabular form (Arksey & O’Malley, 2005; Thomas & Harden, 2008 ).

One of the most highly cited scoping reviews in the eHealth domain was published by Archer, Fevrier-Thomas, Lokker, McKibbon, and Straus (2011) . These authors reviewed the existing literature on personal health record ( phr ) systems including design, functionality, implementation, applications, outcomes, and benefits. Seven databases were searched from 1985 to March 2010. Several search terms relating to phr s were used during this process. Two authors independently screened titles and abstracts to determine inclusion status. A second screen of full-text articles, again by two independent members of the research team, ensured that the studies described phr s. All in all, 130 articles met the criteria and their data were extracted manually into a database. The authors concluded that although there is a large amount of survey, observational, cohort/panel, and anecdotal evidence of phr benefits and satisfaction for patients, more research is needed to evaluate the results of phr implementations. Their in-depth analysis of the literature signalled that there is little solid evidence from randomized controlled trials or other studies through the use of phr s. Hence, they suggested that more research is needed that addresses the current lack of understanding of optimal functionality and usability of these systems, and how they can play a beneficial role in supporting patient self-management ( Archer et al., 2011 ).

9.3.4. Forms of Aggregative Reviews

Healthcare providers, practitioners, and policy-makers are nowadays overwhelmed with large volumes of information, including research-based evidence from numerous clinical trials and evaluation studies, assessing the effectiveness of health information technologies and interventions ( Ammenwerth & de Keizer, 2004 ; Deshazo et al., 2009 ). It is unrealistic to expect that all these disparate actors will have the time, skills, and necessary resources to identify the available evidence in the area of their expertise and consider it when making decisions. Systematic reviews that involve the rigorous application of scientific strategies aimed at limiting subjectivity and bias (i.e., systematic and random errors) can respond to this challenge.

Systematic reviews attempt to aggregate, appraise, and synthesize in a single source all empirical evidence that meet a set of previously specified eligibility criteria in order to answer a clearly formulated and often narrow research question on a particular topic of interest to support evidence-based practice ( Liberati et al., 2009 ). They adhere closely to explicit scientific principles ( Liberati et al., 2009 ) and rigorous methodological guidelines (Higgins & Green, 2008) aimed at reducing random and systematic errors that can lead to deviations from the truth in results or inferences. The use of explicit methods allows systematic reviews to aggregate a large body of research evidence, assess whether effects or relationships are in the same direction and of the same general magnitude, explain possible inconsistencies between study results, and determine the strength of the overall evidence for every outcome of interest based on the quality of included studies and the general consistency among them ( Cook, Mulrow, & Haynes, 1997 ). The main procedures of a systematic review involve:

• Formulating a review question and developing a search strategy based on explicit inclusion criteria for the identification of eligible studies (usually described in the context of a detailed review protocol).
• Searching for eligible studies using multiple databases and information sources, including grey literature sources, without any language restrictions.
• Selecting studies, extracting data, and assessing risk of bias in a duplicate manner using two independent reviewers to avoid random or systematic errors in the process.
• Analyzing data using quantitative or qualitative methods.
• Presenting results in summary of findings tables.
• Interpreting results and drawing conclusions.

Many systematic reviews, but not all, use statistical methods to combine the results of independent studies into a single quantitative estimate or summary effect size. Known as meta-analyses , these reviews use specific data extraction and statistical techniques (e.g., network, frequentist, or Bayesian meta-analyses) to calculate from each study by outcome of interest an effect size along with a confidence interval that reflects the degree of uncertainty behind the point estimate of effect ( Borenstein, Hedges, Higgins, & Rothstein, 2009 ; Deeks, Higgins, & Altman, 2008 ). Subsequently, they use fixed or random-effects analysis models to combine the results of the included studies, assess statistical heterogeneity, and calculate a weighted average of the effect estimates from the different studies, taking into account their sample sizes. The summary effect size is a value that reflects the average magnitude of the intervention effect for a particular outcome of interest or, more generally, the strength of a relationship between two variables across all studies included in the systematic review. By statistically combining data from multiple studies, meta-analyses can create more precise and reliable estimates of intervention effects than those derived from individual studies alone, when these are examined independently as discrete sources of information.

The review by Gurol-Urganci, de Jongh, Vodopivec-Jamsek, Atun, and Car (2013) on the effects of mobile phone messaging reminders for attendance at healthcare appointments is an illustrative example of a high-quality systematic review with meta-analysis. Missed appointments are a major cause of inefficiency in healthcare delivery with substantial monetary costs to health systems. These authors sought to assess whether mobile phone-based appointment reminders delivered through Short Message Service ( sms ) or Multimedia Messaging Service ( mms ) are effective in improving rates of patient attendance and reducing overall costs. To this end, they conducted a comprehensive search on multiple databases using highly sensitive search strategies without language or publication-type restrictions to identify all rct s that are eligible for inclusion. In order to minimize the risk of omitting eligible studies not captured by the original search, they supplemented all electronic searches with manual screening of trial registers and references contained in the included studies. Study selection, data extraction, and risk of bias assessments were performed inde­­pen­dently by two coders using standardized methods to ensure consistency and to eliminate potential errors. Findings from eight rct s involving 6,615 participants were pooled into meta-analyses to calculate the magnitude of effects that mobile text message reminders have on the rate of attendance at healthcare appointments compared to no reminders and phone call reminders.

Meta-analyses are regarded as powerful tools for deriving meaningful conclusions. However, there are situations in which it is neither reasonable nor appropriate to pool studies together using meta-analytic methods simply because there is extensive clinical heterogeneity between the included studies or variation in measurement tools, comparisons, or outcomes of interest. In these cases, systematic reviews can use qualitative synthesis methods such as vote counting, content analysis, classification schemes and tabulations, as an alternative approach to narratively synthesize the results of the independent studies included in the review. This form of review is known as qualitative systematic review.

A rigorous example of one such review in the eHealth domain is presented by Mickan, Atherton, Roberts, Heneghan, and Tilson (2014) on the use of handheld computers by healthcare professionals and their impact on access to information and clinical decision-making. In line with the methodological guide­lines for systematic reviews, these authors: (a) developed and registered with prospero ( www.crd.york.ac.uk/ prospero / ) an a priori review protocol; (b) conducted comprehensive searches for eligible studies using multiple databases and other supplementary strategies (e.g., forward searches); and (c) subsequently carried out study selection, data extraction, and risk of bias assessments in a duplicate manner to eliminate potential errors in the review process. Heterogeneity between the included studies in terms of reported outcomes and measures precluded the use of meta-analytic methods. To this end, the authors resorted to using narrative analysis and synthesis to describe the effectiveness of handheld computers on accessing information for clinical knowledge, adherence to safety and clinical quality guidelines, and diagnostic decision-making.

In recent years, the number of systematic reviews in the field of health informatics has increased considerably. Systematic reviews with discordant findings can cause great confusion and make it difficult for decision-makers to interpret the review-level evidence ( Moher, 2013 ). Therefore, there is a growing need for appraisal and synthesis of prior systematic reviews to ensure that decision-making is constantly informed by the best available accumulated evidence. Umbrella reviews , also known as overviews of systematic reviews, are tertiary types of evidence synthesis that aim to accomplish this; that is, they aim to compare and contrast findings from multiple systematic reviews and meta-analyses ( Becker & Oxman, 2008 ). Umbrella reviews generally adhere to the same principles and rigorous methodological guidelines used in systematic reviews. However, the unit of analysis in umbrella reviews is the systematic review rather than the primary study ( Becker & Oxman, 2008 ). Unlike systematic reviews that have a narrow focus of inquiry, umbrella reviews focus on broader research topics for which there are several potential interventions ( Smith, Devane, Begley, & Clarke, 2011 ). A recent umbrella review on the effects of home telemonitoring interventions for patients with heart failure critically appraised, compared, and synthesized evidence from 15 systematic reviews to investigate which types of home telemonitoring technologies and forms of interventions are more effective in reducing mortality and hospital admissions ( Kitsiou, Paré, & Jaana, 2015 ).

9.3.5. Realist Reviews

Realist reviews are theory-driven interpretative reviews developed to inform, enhance, or supplement conventional systematic reviews by making sense of heterogeneous evidence about complex interventions applied in diverse contexts in a way that informs policy decision-making ( Greenhalgh, Wong, Westhorp, & Pawson, 2011 ). They originated from criticisms of positivist systematic reviews which centre on their “simplistic” underlying assumptions ( Oates, 2011 ). As explained above, systematic reviews seek to identify causation. Such logic is appropriate for fields like medicine and education where findings of randomized controlled trials can be aggregated to see whether a new treatment or intervention does improve outcomes. However, many argue that it is not possible to establish such direct causal links between interventions and outcomes in fields such as social policy, management, and information systems where for any intervention there is unlikely to be a regular or consistent outcome ( Oates, 2011 ; Pawson, 2006 ; Rousseau, Manning, & Denyer, 2008 ).

To circumvent these limitations, Pawson, Greenhalgh, Harvey, and Walshe (2005) have proposed a new approach for synthesizing knowledge that seeks to unpack the mechanism of how “complex interventions” work in particular contexts. The basic research question — what works? — which is usually associated with systematic reviews changes to: what is it about this intervention that works, for whom, in what circumstances, in what respects and why? Realist reviews have no particular preference for either quantitative or qualitative evidence. As a theory-building approach, a realist review usually starts by articulating likely underlying mechanisms and then scrutinizes available evidence to find out whether and where these mechanisms are applicable ( Shepperd et al., 2009 ). Primary studies found in the extant literature are viewed as case studies which can test and modify the initial theories ( Rousseau et al., 2008 ).

The main objective pursued in the realist review conducted by Otte-Trojel, de Bont, Rundall, and van de Klundert (2014) was to examine how patient portals contribute to health service delivery and patient outcomes. The specific goals were to investigate how outcomes are produced and, most importantly, how variations in outcomes can be explained. The research team started with an exploratory review of background documents and research studies to identify ways in which patient portals may contribute to health service delivery and patient outcomes. The authors identified six main ways which represent “educated guesses” to be tested against the data in the evaluation studies. These studies were identified through a formal and systematic search in four databases between 2003 and 2013. Two members of the research team selected the articles using a pre-established list of inclusion and exclusion criteria and following a two-step procedure. The authors then extracted data from the selected articles and created several tables, one for each outcome category. They organized information to bring forward those mechanisms where patient portals contribute to outcomes and the variation in outcomes across different contexts.

9.3.6. Critical Reviews

Lastly, critical reviews aim to provide a critical evaluation and interpretive analysis of existing literature on a particular topic of interest to reveal strengths, weaknesses, contradictions, controversies, inconsistencies, and/or other important issues with respect to theories, hypotheses, research methods or results ( Baumeister & Leary, 1997 ; Kirkevold, 1997 ). Unlike other review types, critical reviews attempt to take a reflective account of the research that has been done in a particular area of interest, and assess its credibility by using appraisal instruments or critical interpretive methods. In this way, critical reviews attempt to constructively inform other scholars about the weaknesses of prior research and strengthen knowledge development by giving focus and direction to studies for further improvement ( Kirkevold, 1997 ).

Kitsiou, Paré, and Jaana (2013) provide an example of a critical review that assessed the methodological quality of prior systematic reviews of home telemonitoring studies for chronic patients. The authors conducted a comprehensive search on multiple databases to identify eligible reviews and subsequently used a validated instrument to conduct an in-depth quality appraisal. Results indicate that the majority of systematic reviews in this particular area suffer from important methodological flaws and biases that impair their internal validity and limit their usefulness for clinical and decision-making purposes. To this end, they provide a number of recommendations to strengthen knowledge development towards improving the design and execution of future reviews on home telemonitoring.

9.4. Summary

Table 9.1 outlines the main types of literature reviews that were described in the previous sub-sections and summarizes the main characteristics that distinguish one review type from another. It also includes key references to methodological guidelines and useful sources that can be used by eHealth scholars and researchers for planning and developing reviews.

Typology of Literature Reviews (adapted from Paré et al., 2015).

As shown in Table 9.1 , each review type addresses different kinds of research questions or objectives, which subsequently define and dictate the methods and approaches that need to be used to achieve the overarching goal(s) of the review. For example, in the case of narrative reviews, there is greater flexibility in searching and synthesizing articles ( Green et al., 2006 ). Researchers are often relatively free to use a diversity of approaches to search, identify, and select relevant scientific articles, describe their operational characteristics, present how the individual studies fit together, and formulate conclusions. On the other hand, systematic reviews are characterized by their high level of systematicity, rigour, and use of explicit methods, based on an “a priori” review plan that aims to minimize bias in the analysis and synthesis process (Higgins & Green, 2008). Some reviews are exploratory in nature (e.g., scoping/mapping reviews), whereas others may be conducted to discover patterns (e.g., descriptive reviews) or involve a synthesis approach that may include the critical analysis of prior research ( Paré et al., 2015 ). Hence, in order to select the most appropriate type of review, it is critical to know before embarking on a review project, why the research synthesis is conducted and what type of methods are best aligned with the pursued goals.

9.5. Concluding Remarks

In light of the increased use of evidence-based practice and research generating stronger evidence ( Grady et al., 2011 ; Lyden et al., 2013 ), review articles have become essential tools for summarizing, synthesizing, integrating or critically appraising prior knowledge in the eHealth field. As mentioned earlier, when rigorously conducted review articles represent powerful information sources for eHealth scholars and practitioners looking for state-of-the-art evidence. The typology of literature reviews we used herein will allow eHealth researchers, graduate students and practitioners to gain a better understanding of the similarities and differences between review types.

We must stress that this classification scheme does not privilege any specific type of review as being of higher quality than another ( Paré et al., 2015 ). As explained above, each type of review has its own strengths and limitations. Having said that, we realize that the methodological rigour of any review — be it qualitative, quantitative or mixed — is a critical aspect that should be considered seriously by prospective authors. In the present context, the notion of rigour refers to the reliability and validity of the review process described in section 9.2. For one thing, reliability is related to the reproducibility of the review process and steps, which is facilitated by a comprehensive documentation of the literature search process, extraction, coding and analysis performed in the review. Whether the search is comprehensive or not, whether it involves a methodical approach for data extraction and synthesis or not, it is important that the review documents in an explicit and transparent manner the steps and approach that were used in the process of its development. Next, validity characterizes the degree to which the review process was conducted appropriately. It goes beyond documentation and reflects decisions related to the selection of the sources, the search terms used, the period of time covered, the articles selected in the search, and the application of backward and forward searches ( vom Brocke et al., 2009 ). In short, the rigour of any review article is reflected by the explicitness of its methods (i.e., transparency) and the soundness of the approach used. We refer those interested in the concepts of rigour and quality to the work of Templier and Paré (2015) which offers a detailed set of methodological guidelines for conducting and evaluating various types of review articles.

To conclude, our main objective in this chapter was to demystify the various types of literature reviews that are central to the continuous development of the eHealth field. It is our hope that our descriptive account will serve as a valuable source for those conducting, evaluating or using reviews in this important and growing domain.

• Ammenwerth E., de Keizer N. An inventory of evaluation studies of information technology in health care. Trends in evaluation research, 1982-2002. International Journal of Medical Informatics. 2004; 44 (1):44–56. [ PubMed : 15778794 ]
• Anderson S., Allen P., Peckham S., Goodwin N. Asking the right questions: scoping studies in the commissioning of research on the organisation and delivery of health services. Health Research Policy and Systems. 2008; 6 (7):1–12. [ PMC free article : PMC2500008 ] [ PubMed : 18613961 ] [ CrossRef ]
• Archer N., Fevrier-Thomas U., Lokker C., McKibbon K. A., Straus S.E. Personal health records: a scoping review. Journal of American Medical Informatics Association. 2011; 18 (4):515–522. [ PMC free article : PMC3128401 ] [ PubMed : 21672914 ]
• Arksey H., O’Malley L. Scoping studies: towards a methodological framework. International Journal of Social Research Methodology. 2005; 8 (1):19–32.
• A systematic, tool-supported method for conducting literature reviews in information systems. Paper presented at the Proceedings of the 19th European Conference on Information Systems ( ecis 2011); June 9 to 11; Helsinki, Finland. 2011.
• Baumeister R. F., Leary M.R. Writing narrative literature reviews. Review of General Psychology. 1997; 1 (3):311–320.
• Becker L. A., Oxman A.D. In: Cochrane handbook for systematic reviews of interventions. Higgins J. P. T., Green S., editors. Hoboken, nj : John Wiley & Sons, Ltd; 2008. Overviews of reviews; pp. 607–631.
• Borenstein M., Hedges L., Higgins J., Rothstein H. Introduction to meta-analysis. Hoboken, nj : John Wiley & Sons Inc; 2009.
• Cook D. J., Mulrow C. D., Haynes B. Systematic reviews: Synthesis of best evidence for clinical decisions. Annals of Internal Medicine. 1997; 126 (5):376–380. [ PubMed : 9054282 ]
• Cooper H., Hedges L.V. In: The handbook of research synthesis and meta-analysis. 2nd ed. Cooper H., Hedges L. V., Valentine J. C., editors. New York: Russell Sage Foundation; 2009. Research synthesis as a scientific process; pp. 3–17.
• Cooper H. M. Organizing knowledge syntheses: A taxonomy of literature reviews. Knowledge in Society. 1988; 1 (1):104–126.
• Cronin P., Ryan F., Coughlan M. Undertaking a literature review: a step-by-step approach. British Journal of Nursing. 2008; 17 (1):38–43. [ PubMed : 18399395 ]
• Darlow S., Wen K.Y. Development testing of mobile health interventions for cancer patient self-management: A review. Health Informatics Journal. 2015 (online before print). [ PubMed : 25916831 ] [ CrossRef ]
• Daudt H. M., van Mossel C., Scott S.J. Enhancing the scoping study methodology: a large, inter-professional team’s experience with Arksey and O’Malley’s framework. bmc Medical Research Methodology. 2013; 13 :48. [ PMC free article : PMC3614526 ] [ PubMed : 23522333 ] [ CrossRef ]
• Davies P. The relevance of systematic reviews to educational policy and practice. Oxford Review of Education. 2000; 26 (3-4):365–378.
• Deeks J. J., Higgins J. P. T., Altman D.G. In: Cochrane handbook for systematic reviews of interventions. Higgins J. P. T., Green S., editors. Hoboken, nj : John Wiley & Sons, Ltd; 2008. Analysing data and undertaking meta-analyses; pp. 243–296.
• Deshazo J. P., Lavallie D. L., Wolf F.M. Publication trends in the medical informatics literature: 20 years of “Medical Informatics” in mesh . bmc Medical Informatics and Decision Making. 2009; 9 :7. [ PMC free article : PMC2652453 ] [ PubMed : 19159472 ] [ CrossRef ]
• Dixon-Woods M., Agarwal S., Jones D., Young B., Sutton A. Synthesising qualitative and quantitative evidence: a review of possible methods. Journal of Health Services Research and Policy. 2005; 10 (1):45–53. [ PubMed : 15667704 ]
• Finfgeld-Connett D., Johnson E.D. Literature search strategies for conducting knowledge-building and theory-generating qualitative systematic reviews. Journal of Advanced Nursing. 2013; 69 (1):194–204. [ PMC free article : PMC3424349 ] [ PubMed : 22591030 ]
• Grady B., Myers K. M., Nelson E. L., Belz N., Bennett L., Carnahan L. … Guidelines Working Group. Evidence-based practice for telemental health. Telemedicine Journal and E Health. 2011; 17 (2):131–148. [ PubMed : 21385026 ]
• Green B. N., Johnson C. D., Adams A. Writing narrative literature reviews for peer-reviewed journals: secrets of the trade. Journal of Chiropractic Medicine. 2006; 5 (3):101–117. [ PMC free article : PMC2647067 ] [ PubMed : 19674681 ]
• Greenhalgh T., Wong G., Westhorp G., Pawson R. Protocol–realist and meta-narrative evidence synthesis: evolving standards ( rameses ). bmc Medical Research Methodology. 2011; 11 :115. [ PMC free article : PMC3173389 ] [ PubMed : 21843376 ]
• Gurol-Urganci I., de Jongh T., Vodopivec-Jamsek V., Atun R., Car J. Mobile phone messaging reminders for attendance at healthcare appointments. Cochrane Database System Review. 2013; 12 cd 007458. [ PMC free article : PMC6485985 ] [ PubMed : 24310741 ] [ CrossRef ]
• Hart C. Doing a literature review: Releasing the social science research imagination. London: SAGE Publications; 1998.
• Higgins J. P. T., Green S., editors. Cochrane handbook for systematic reviews of interventions: Cochrane book series. Hoboken, nj : Wiley-Blackwell; 2008.
• Jesson J., Matheson L., Lacey F.M. Doing your literature review: traditional and systematic techniques. Los Angeles & London: SAGE Publications; 2011.
• King W. R., He J. Understanding the role and methods of meta-analysis in IS research. Communications of the Association for Information Systems. 2005; 16 :1.
• Kirkevold M. Integrative nursing research — an important strategy to further the development of nursing science and nursing practice. Journal of Advanced Nursing. 1997; 25 (5):977–984. [ PubMed : 9147203 ]
• Kitchenham B., Charters S. ebse Technical Report Version 2.3. Keele & Durham. uk : Keele University & University of Durham; 2007. Guidelines for performing systematic literature reviews in software engineering.
• Kitsiou S., Paré G., Jaana M. Systematic reviews and meta-analyses of home telemonitoring interventions for patients with chronic diseases: a critical assessment of their methodological quality. Journal of Medical Internet Research. 2013; 15 (7):e150. [ PMC free article : PMC3785977 ] [ PubMed : 23880072 ]
• Kitsiou S., Paré G., Jaana M. Effects of home telemonitoring interventions on patients with chronic heart failure: an overview of systematic reviews. Journal of Medical Internet Research. 2015; 17 (3):e63. [ PMC free article : PMC4376138 ] [ PubMed : 25768664 ]
• Levac D., Colquhoun H., O’Brien K. K. Scoping studies: advancing the methodology. Implementation Science. 2010; 5 (1):69. [ PMC free article : PMC2954944 ] [ PubMed : 20854677 ]
• Levy Y., Ellis T.J. A systems approach to conduct an effective literature review in support of information systems research. Informing Science. 2006; 9 :181–211.
• Liberati A., Altman D. G., Tetzlaff J., Mulrow C., Gøtzsche P. C., Ioannidis J. P. A. et al. Moher D. The prisma statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: Explanation and elaboration. Annals of Internal Medicine. 2009; 151 (4):W-65. [ PubMed : 19622512 ]
• Lyden J. R., Zickmund S. L., Bhargava T. D., Bryce C. L., Conroy M. B., Fischer G. S. et al. McTigue K. M. Implementing health information technology in a patient-centered manner: Patient experiences with an online evidence-based lifestyle intervention. Journal for Healthcare Quality. 2013; 35 (5):47–57. [ PubMed : 24004039 ]
• Mickan S., Atherton H., Roberts N. W., Heneghan C., Tilson J.K. Use of handheld computers in clinical practice: a systematic review. bmc Medical Informatics and Decision Making. 2014; 14 :56. [ PMC free article : PMC4099138 ] [ PubMed : 24998515 ]
• Moher D. The problem of duplicate systematic reviews. British Medical Journal. 2013; 347 (5040) [ PubMed : 23945367 ] [ CrossRef ]
• Montori V. M., Wilczynski N. L., Morgan D., Haynes R. B., Hedges T. Systematic reviews: a cross-sectional study of location and citation counts. bmc Medicine. 2003; 1 :2. [ PMC free article : PMC281591 ] [ PubMed : 14633274 ]
• Mulrow C. D. The medical review article: state of the science. Annals of Internal Medicine. 1987; 106 (3):485–488. [ PubMed : 3813259 ] [ CrossRef ]
• Evidence-based information systems: A decade later. Proceedings of the European Conference on Information Systems ; 2011. Retrieved from http://aisel ​.aisnet.org/cgi/viewcontent ​.cgi?article ​=1221&context ​=ecis2011 .
• Okoli C., Schabram K. A guide to conducting a systematic literature review of information systems research. ssrn Electronic Journal. 2010
• Otte-Trojel T., de Bont A., Rundall T. G., van de Klundert J. How outcomes are achieved through patient portals: a realist review. Journal of American Medical Informatics Association. 2014; 21 (4):751–757. [ PMC free article : PMC4078283 ] [ PubMed : 24503882 ]
• Paré G., Trudel M.-C., Jaana M., Kitsiou S. Synthesizing information systems knowledge: A typology of literature reviews. Information & Management. 2015; 52 (2):183–199.
• Patsopoulos N. A., Analatos A. A., Ioannidis J.P. A. Relative citation impact of various study designs in the health sciences. Journal of the American Medical Association. 2005; 293 (19):2362–2366. [ PubMed : 15900006 ]
• Paul M. M., Greene C. M., Newton-Dame R., Thorpe L. E., Perlman S. E., McVeigh K. H., Gourevitch M.N. The state of population health surveillance using electronic health records: A narrative review. Population Health Management. 2015; 18 (3):209–216. [ PubMed : 25608033 ]
• Pawson R. Evidence-based policy: a realist perspective. London: SAGE Publications; 2006.
• Pawson R., Greenhalgh T., Harvey G., Walshe K. Realist review—a new method of systematic review designed for complex policy interventions. Journal of Health Services Research & Policy. 2005; 10 (Suppl 1):21–34. [ PubMed : 16053581 ]
• Petersen K., Vakkalanka S., Kuzniarz L. Guidelines for conducting systematic mapping studies in software engineering: An update. Information and Software Technology. 2015; 64 :1–18.
• Petticrew M., Roberts H. Systematic reviews in the social sciences: A practical guide. Malden, ma : Blackwell Publishing Co; 2006.
• Rousseau D. M., Manning J., Denyer D. Evidence in management and organizational science: Assembling the field’s full weight of scientific knowledge through syntheses. The Academy of Management Annals. 2008; 2 (1):475–515.
• Rowe F. What literature review is not: diversity, boundaries and recommendations. European Journal of Information Systems. 2014; 23 (3):241–255.
• Shea B. J., Hamel C., Wells G. A., Bouter L. M., Kristjansson E., Grimshaw J. et al. Boers M. amstar is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. Journal of Clinical Epidemiology. 2009; 62 (10):1013–1020. [ PubMed : 19230606 ]
• Shepperd S., Lewin S., Straus S., Clarke M., Eccles M. P., Fitzpatrick R. et al. Sheikh A. Can we systematically review studies that evaluate complex interventions? PLoS Medicine. 2009; 6 (8):e1000086. [ PMC free article : PMC2717209 ] [ PubMed : 19668360 ]
• Silva B. M., Rodrigues J. J., de la Torre Díez I., López-Coronado M., Saleem K. Mobile-health: A review of current state in 2015. Journal of Biomedical Informatics. 2015; 56 :265–272. [ PubMed : 26071682 ]
• Smith V., Devane D., Begley C., Clarke M. Methodology in conducting a systematic review of systematic reviews of healthcare interventions. bmc Medical Research Methodology. 2011; 11 (1):15. [ PMC free article : PMC3039637 ] [ PubMed : 21291558 ]
• Sylvester A., Tate M., Johnstone D. Beyond synthesis: re-presenting heterogeneous research literature. Behaviour & Information Technology. 2013; 32 (12):1199–1215.
• Templier M., Paré G. A framework for guiding and evaluating literature reviews. Communications of the Association for Information Systems. 2015; 37 (6):112–137.
• Thomas J., Harden A. Methods for the thematic synthesis of qualitative research in systematic reviews. bmc Medical Research Methodology. 2008; 8 (1):45. [ PMC free article : PMC2478656 ] [ PubMed : 18616818 ]
• Reconstructing the giant: on the importance of rigour in documenting the literature search process. Paper presented at the Proceedings of the 17th European Conference on Information Systems ( ecis 2009); Verona, Italy. 2009.
• Webster J., Watson R.T. Analyzing the past to prepare for the future: Writing a literature review. Management Information Systems Quarterly. 2002; 26 (2):11.
• Whitlock E. P., Lin J. S., Chou R., Shekelle P., Robinson K.A. Using existing systematic reviews in complex systematic reviews. Annals of Internal Medicine. 2008; 148 (10):776–782. [ PubMed : 18490690 ]

This publication is licensed under a Creative Commons License, Attribution-Noncommercial 4.0 International License (CC BY-NC 4.0): see https://creativecommons.org/licenses/by-nc/4.0/

• Cite this Page Paré G, Kitsiou S. Chapter 9 Methods for Literature Reviews. In: Lau F, Kuziemsky C, editors. Handbook of eHealth Evaluation: An Evidence-based Approach [Internet]. Victoria (BC): University of Victoria; 2017 Feb 27.
• PDF version of this title (4.5M)

In this Page

• Introduction
• Overview of the Literature Review Process and Steps
• Types of Review Articles and Brief Illustrations
• Concluding Remarks

Related information

• PMC PubMed Central citations
• PubMed Links to PubMed

Recent Activity

• Chapter 9 Methods for Literature Reviews - Handbook of eHealth Evaluation: An Ev... Chapter 9 Methods for Literature Reviews - Handbook of eHealth Evaluation: An Evidence-based Approach

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

Connect with NLM

National Library of Medicine 8600 Rockville Pike Bethesda, MD 20894

Web Policies FOIA HHS Vulnerability Disclosure

Help Accessibility Careers

• +44 (0) 207 391 9032

Recent Posts

What is a literature review definition, types, and examples.

• Why Is Your CV Getting Rejected and How to Avoid It
• Where to Find Images for Presentations
• What Is an Internship? Everything You Should Know
• How Long Should a Thesis Statement Be?
• How to Write a Character Analysis Essay
• Best Colours for Your PowerPoint Presentation: Top Colour Combinations
• How to Write a Nursing Essay – With Examples
• Top 5 Essential Skills You Should Build As An International Student
• How Professional Editing Services Can Take Your Writing to the Next Level
• Academic News
• Custom Essays
• Dissertation Writing
• Essay Marking
• Essay Writing
• Essay Writing Companies
• Model Essays
• Model Exam Answers
• Oxbridge Essays Updates
• PhD Writing
• Significant Academics
• Student News
• Study Skills
• University Applications
• University Essays
• University Life
• Writing Tips

Since 2006, Oxbridge Essays has been the UK’s leading paid essay-writing and dissertation service

We have helped 10,000s of undergraduate, Masters and PhD students to maximise their grades in essays, dissertations, model-exam answers, applications and other materials. If you would like a free chat about your project with one of our UK staff, then please just reach out on one of the methods below.

A literature review is an essential part of any academic research paper, thesis, or dissertation. It provides a thorough examination of existing research on a particular topic, allowing the researcher to identify gaps, areas of agreement or disagreement, and emerging trends in the field. In this post, we’ll delve into the definition of a literature review, explore the different types of literature reviews, and provide examples of literature review structures that can guide your own work. Additionally, we’ll offer tips on how to craft a compelling literature review that strengthens the foundation of your research.

Literature Review Meaning

The term "literature review" refers to a comprehensive survey of the scholarly works, books, journal articles, and other sources relevant to a particular research topic. Its primary purpose is to offer a critical evaluation of the existing body of knowledge. The literature review helps set the context for the research question, showing what has already been explored and where gaps in knowledge or methodological limitations may exist. By examining various sources, you can assess how your research fits into the broader conversation within your field. The literature review also provides the foundation for your argument, helping to justify the importance of your research and explain how it contributes to the ongoing academic discussion.

Why Is a Literature Review Important?

A literature review is not just a summary of previous research but a critical analysis of the work that has been done in a particular area of study. It helps demonstrate your understanding of the topic and situates your work within the existing academic landscape. By conducting a literature review, you ensure that your research is not redundant and identify the unique contributions your study can make. Furthermore, the literature review informs your methodology, highlighting which methods have been successful in previous studies and which have encountered limitations. By understanding what has worked before, you can avoid potential pitfalls and build upon the successes of earlier researchers.

Literature Review Structure

The structure of a literature review can vary depending on the nature of your research and the field of study. However, the most common literature review structure includes several key components:

• Introduction :This section outlines the scope of the literature review, defines the key terms, and states the overall purpose of the review. It provides the reader with an understanding of what the review will cover.
• Thematic Organisation : The literature is often organised thematically, grouping together works that address similar aspects of the research topic. Themes can relate to theoretical approaches, methodologies, or different interpretations of key issues.
• Critical Evaluation : The body of the literature review should not only summarise the existing research but also critically evaluate it. This might involve identifying strengths and weaknesses in methodologies, assessing the reliability of findings, and discussing how well the research supports the claims made.
• Conclusion : The conclusion should summarise the main findings of the review, restate the key themes, and highlight gaps in the research that your study will address. It should also reflect on how the literature review has shaped your own research design.

Types of Literature Reviews

There are several different forms of literature reviews, each with a distinct focus and structure. Understanding these types can help you choose the approach that best fits your research needs. Here are some of the most common types of literature reviews:

• Narrative Literature Review : This is the most traditional form of literature review. It provides a comprehensive summary and analysis of the literature on a particular topic. Narrative reviews are often broad in scope and provide an overview of key themes and trends.
• Systematic Literature Review : This type of review involves a rigorous, structured process that aims to identify all relevant studies on a specific research question. Systematic reviews follow a clearly defined methodology, including specific criteria for selecting and analysing studies. They are commonly used in fields such as healthcare, where a comprehensive synthesis of evidence is needed.
• Scoping Review : Clearly outline your main argument or position. This should guide the direction of your essay.
• Scoping Review : A scoping review is used to map the key concepts, sources, and evidence in a research area. It is often the first step before a systematic review and is useful for identifying gaps in the literature and guiding further research.
• Meta-Analysis : This is a form of literature review that uses statistical techniques to combine the results of multiple studies. Meta-analyses are typically used to provide an overall estimate of the effect size for a particular intervention or phenomenon.
• Integrative Review : An integrative review synthesises qualitative and quantitative data to provide a more holistic view of the research on a particular topic. It aims to generate new perspectives by integrating findings from different types of studies.
• Critical Review : This type of literature review goes beyond merely describing the literature. A critical review analyses and synthesises the research, evaluating its strengths and weaknesses and offering new insights and perspectives on the topic.

Short Example of a Literature Review

Below is an example of the literature review from a dissertation on climate change policies. The example demonstrates how to structure a literature review and critically engage with the literature:

Introduction of the Literature Review

Climate change has been a topic of growing concern over the past few decades, with numerous policies introduced globally to mitigate its effects. This review examines the existing literature on climate change policies, focusing on the effectiveness of carbon pricing, renewable energy subsidies, and regulatory approaches. The review aims to highlight the strengths and limitations of these policies and identify gaps in the research that future studies should address.

Thematic Organisation

The literature is organised into three main themes: carbon pricing mechanisms, renewable energy subsidies, and regulatory approaches to emissions reduction. Each theme is analysed in detail, examining the key findings of previous research and assessing the impact of these policies on greenhouse gas emissions.

Critical Evaluation

The review finds that while carbon pricing mechanisms have been effective in reducing emissions in some contexts, their success is heavily dependent on political and economic factors. Renewable energy subsidies have contributed to significant increases in renewable energy capacity, but their long-term sustainability remains in question. Regulatory approaches, while often politically contentious, have proven to be effective in certain jurisdictions.

The literature review concludes that although significant progress has been made in the development of climate change policies, further research is needed to evaluate the long-term impacts of these policies and to explore new approaches that may be more effective in reducing emissions.

Key Considerations

Writing a literature review can be a complex task, but it is a vital part of the research process. By understanding the meaning of a literature review, familiarising yourself with different forms of literature reviews, and following a clear structure, you can create a review that enhances your research project and demonstrates your knowledge of the field.

Top 10 tips for writing a dissertation methodology

Advice for successfully writing a dissertation, how to structure your dissertation in 2024, writing services.

• Essay Plans
• Critical Reviews
• Literature Reviews
• Presentations
• Dissertation Title Creation
• Dissertation Proposals
• Dissertation Chapters
• PhD Proposals
• Journal Publication
• CV Writing Service
• Business Proofreading Services

Editing Services

• Proofreading Service
• Editing Service
• Academic Editing Service

Additional Services

• Marking Services
• Consultation Calls
• Personal Statements
• Tutoring Services

Our Company

• Frequently Asked Questions
• Become a Writer

Terms & Policies

• Fair Use Policy
• Policy for Students in England
• Privacy Policy
• Terms & Conditions
• Editing Service Examples
• [email protected]
• Contact Form

Payment Methods

Cryptocurrency payments.

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• Systematic Review | Definition, Example, & Guide

Systematic Review | Definition, Example & Guide

Published on June 15, 2022 by Shaun Turney . Revised on November 20, 2023.

A systematic review is a type of review that uses repeatable methods to find, select, and synthesize all available evidence. It answers a clearly formulated research question and explicitly states the methods used to arrive at the answer.

They answered the question “What is the effectiveness of probiotics in reducing eczema symptoms and improving quality of life in patients with eczema?”

In this context, a probiotic is a health product that contains live microorganisms and is taken by mouth. Eczema is a common skin condition that causes red, itchy skin.

Table of contents

What is a systematic review, systematic review vs. meta-analysis, systematic review vs. literature review, systematic review vs. scoping review, when to conduct a systematic review, pros and cons of systematic reviews, step-by-step example of a systematic review, other interesting articles, frequently asked questions about systematic reviews.

A review is an overview of the research that’s already been completed on a topic.

What makes a systematic review different from other types of reviews is that the research methods are designed to reduce bias . The methods are repeatable, and the approach is formal and systematic:

• Formulate a research question
• Develop a protocol
• Search for all relevant studies
• Apply the selection criteria
• Extract the data
• Synthesize the data
• Write and publish a report

Although multiple sets of guidelines exist, the Cochrane Handbook for Systematic Reviews is among the most widely used. It provides detailed guidelines on how to complete each step of the systematic review process.

Systematic reviews are most commonly used in medical and public health research, but they can also be found in other disciplines.

Systematic reviews typically answer their research question by synthesizing all available evidence and evaluating the quality of the evidence. Synthesizing means bringing together different information to tell a single, cohesive story. The synthesis can be narrative ( qualitative ), quantitative , or both.

Prevent plagiarism. Run a free check.

Systematic reviews often quantitatively synthesize the evidence using a meta-analysis . A meta-analysis is a statistical analysis, not a type of review.

A meta-analysis is a technique to synthesize results from multiple studies. It’s a statistical analysis that combines the results of two or more studies, usually to estimate an effect size .

A literature review is a type of review that uses a less systematic and formal approach than a systematic review. Typically, an expert in a topic will qualitatively summarize and evaluate previous work, without using a formal, explicit method.

Although literature reviews are often less time-consuming and can be insightful or helpful, they have a higher risk of bias and are less transparent than systematic reviews.

Similar to a systematic review, a scoping review is a type of review that tries to minimize bias by using transparent and repeatable methods.

However, a scoping review isn’t a type of systematic review. The most important difference is the goal: rather than answering a specific question, a scoping review explores a topic. The researcher tries to identify the main concepts, theories, and evidence, as well as gaps in the current research.

Sometimes scoping reviews are an exploratory preparation step for a systematic review, and sometimes they are a standalone project.

Here's why students love Scribbr's proofreading services

Discover proofreading & editing

A systematic review is a good choice of review if you want to answer a question about the effectiveness of an intervention , such as a medical treatment.

To conduct a systematic review, you’ll need the following:

• A precise question , usually about the effectiveness of an intervention. The question needs to be about a topic that’s previously been studied by multiple researchers. If there’s no previous research, there’s nothing to review.
• If you’re doing a systematic review on your own (e.g., for a research paper or thesis ), you should take appropriate measures to ensure the validity and reliability of your research.
• Access to databases and journal archives. Often, your educational institution provides you with access.
• Time. A professional systematic review is a time-consuming process: it will take the lead author about six months of full-time work. If you’re a student, you should narrow the scope of your systematic review and stick to a tight schedule.
• Bibliographic, word-processing, spreadsheet, and statistical software . For example, you could use EndNote, Microsoft Word, Excel, and SPSS.

A systematic review has many pros .

• They minimize research bias by considering all available evidence and evaluating each study for bias.
• Their methods are transparent , so they can be scrutinized by others.
• They’re thorough : they summarize all available evidence.
• They can be replicated and updated by others.

Systematic reviews also have a few cons .

• They’re time-consuming .
• They’re narrow in scope : they only answer the precise research question.

The 7 steps for conducting a systematic review are explained with an example.

Step 1: Formulate a research question

Formulating the research question is probably the most important step of a systematic review. A clear research question will:

• Allow you to more effectively communicate your research to other researchers and practitioners
• Guide your decisions as you plan and conduct your systematic review

A good research question for a systematic review has four components, which you can remember with the acronym PICO :

• Population(s) or problem(s)
• Intervention(s)
• Comparison(s)

You can rearrange these four components to write your research question:

• What is the effectiveness of I versus C for O in P ?

Sometimes, you may want to include a fifth component, the type of study design . In this case, the acronym is PICOT .

• Type of study design(s)
• The population of patients with eczema
• The intervention of probiotics
• In comparison to no treatment, placebo , or non-probiotic treatment
• The outcome of changes in participant-, parent-, and doctor-rated symptoms of eczema and quality of life
• Randomized control trials, a type of study design

Their research question was:

• What is the effectiveness of probiotics versus no treatment, a placebo, or a non-probiotic treatment for reducing eczema symptoms and improving quality of life in patients with eczema?

Step 2: Develop a protocol

A protocol is a document that contains your research plan for the systematic review. This is an important step because having a plan allows you to work more efficiently and reduces bias.

Your protocol should include the following components:

• Background information : Provide the context of the research question, including why it’s important.
• Research objective (s) : Rephrase your research question as an objective.
• Selection criteria: State how you’ll decide which studies to include or exclude from your review.
• Search strategy: Discuss your plan for finding studies.
• Analysis: Explain what information you’ll collect from the studies and how you’ll synthesize the data.

If you’re a professional seeking to publish your review, it’s a good idea to bring together an advisory committee . This is a group of about six people who have experience in the topic you’re researching. They can help you make decisions about your protocol.

It’s highly recommended to register your protocol. Registering your protocol means submitting it to a database such as PROSPERO or ClinicalTrials.gov .

Step 3: Search for all relevant studies

Searching for relevant studies is the most time-consuming step of a systematic review.

To reduce bias, it’s important to search for relevant studies very thoroughly. Your strategy will depend on your field and your research question, but sources generally fall into these four categories:

• Databases: Search multiple databases of peer-reviewed literature, such as PubMed or Scopus . Think carefully about how to phrase your search terms and include multiple synonyms of each word. Use Boolean operators if relevant.
• Handsearching: In addition to searching the primary sources using databases, you’ll also need to search manually. One strategy is to scan relevant journals or conference proceedings. Another strategy is to scan the reference lists of relevant studies.
• Gray literature: Gray literature includes documents produced by governments, universities, and other institutions that aren’t published by traditional publishers. Graduate student theses are an important type of gray literature, which you can search using the Networked Digital Library of Theses and Dissertations (NDLTD) . In medicine, clinical trial registries are another important type of gray literature.
• Experts: Contact experts in the field to ask if they have unpublished studies that should be included in your review.

At this stage of your review, you won’t read the articles yet. Simply save any potentially relevant citations using bibliographic software, such as Scribbr’s APA or MLA Generator .

• Databases: EMBASE, PsycINFO, AMED, LILACS, and ISI Web of Science
• Handsearch: Conference proceedings and reference lists of articles
• Gray literature: The Cochrane Library, the metaRegister of Controlled Trials, and the Ongoing Skin Trials Register
• Experts: Authors of unpublished registered trials, pharmaceutical companies, and manufacturers of probiotics

Step 4: Apply the selection criteria

Applying the selection criteria is a three-person job. Two of you will independently read the studies and decide which to include in your review based on the selection criteria you established in your protocol . The third person’s job is to break any ties.

To increase inter-rater reliability , ensure that everyone thoroughly understands the selection criteria before you begin.

If you’re writing a systematic review as a student for an assignment, you might not have a team. In this case, you’ll have to apply the selection criteria on your own; you can mention this as a limitation in your paper’s discussion.

You should apply the selection criteria in two phases:

• Based on the titles and abstracts : Decide whether each article potentially meets the selection criteria based on the information provided in the abstracts.
• Based on the full texts: Download the articles that weren’t excluded during the first phase. If an article isn’t available online or through your library, you may need to contact the authors to ask for a copy. Read the articles and decide which articles meet the selection criteria.

It’s very important to keep a meticulous record of why you included or excluded each article. When the selection process is complete, you can summarize what you did using a PRISMA flow diagram .

Next, Boyle and colleagues found the full texts for each of the remaining studies. Boyle and Tang read through the articles to decide if any more studies needed to be excluded based on the selection criteria.

When Boyle and Tang disagreed about whether a study should be excluded, they discussed it with Varigos until the three researchers came to an agreement.

Step 5: Extract the data

Extracting the data means collecting information from the selected studies in a systematic way. There are two types of information you need to collect from each study:

• Information about the study’s methods and results . The exact information will depend on your research question, but it might include the year, study design , sample size, context, research findings , and conclusions. If any data are missing, you’ll need to contact the study’s authors.
• Your judgment of the quality of the evidence, including risk of bias .

You should collect this information using forms. You can find sample forms in The Registry of Methods and Tools for Evidence-Informed Decision Making and the Grading of Recommendations, Assessment, Development and Evaluations Working Group .

Extracting the data is also a three-person job. Two people should do this step independently, and the third person will resolve any disagreements.

They also collected data about possible sources of bias, such as how the study participants were randomized into the control and treatment groups.

Step 6: Synthesize the data

Synthesizing the data means bringing together the information you collected into a single, cohesive story. There are two main approaches to synthesizing the data:

• Narrative ( qualitative ): Summarize the information in words. You’ll need to discuss the studies and assess their overall quality.
• Quantitative : Use statistical methods to summarize and compare data from different studies. The most common quantitative approach is a meta-analysis , which allows you to combine results from multiple studies into a summary result.

Generally, you should use both approaches together whenever possible. If you don’t have enough data, or the data from different studies aren’t comparable, then you can take just a narrative approach. However, you should justify why a quantitative approach wasn’t possible.

Boyle and colleagues also divided the studies into subgroups, such as studies about babies, children, and adults, and analyzed the effect sizes within each group.

Step 7: Write and publish a report

The purpose of writing a systematic review article is to share the answer to your research question and explain how you arrived at this answer.

Your article should include the following sections:

• Abstract : A summary of the review
• Introduction : Including the rationale and objectives
• Methods : Including the selection criteria, search method, data extraction method, and synthesis method
• Results : Including results of the search and selection process, study characteristics, risk of bias in the studies, and synthesis results
• Discussion : Including interpretation of the results and limitations of the review
• Conclusion : The answer to your research question and implications for practice, policy, or research

To verify that your report includes everything it needs, you can use the PRISMA checklist .

Once your report is written, you can publish it in a systematic review database, such as the Cochrane Database of Systematic Reviews , and/or in a peer-reviewed journal.

In their report, Boyle and colleagues concluded that probiotics cannot be recommended for reducing eczema symptoms or improving quality of life in patients with eczema. Note Generative AI tools like ChatGPT can be useful at various stages of the writing and research process and can help you to write your systematic review. However, we strongly advise against trying to pass AI-generated text off as your own work.

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

• Student’s  t -distribution
• Normal distribution
• Null and Alternative Hypotheses
• Chi square tests
• Confidence interval
• Quartiles & Quantiles
• Cluster sampling
• Stratified sampling
• Data cleansing
• Reproducibility vs Replicability
• Peer review
• Prospective cohort study

Research bias

• Implicit bias
• Cognitive bias
• Placebo effect
• Hawthorne effect
• Hindsight bias
• Affect heuristic
• Social desirability bias

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a thesis, dissertation , or research paper , in order to situate your work in relation to existing knowledge.

A literature review is a survey of credible sources on a topic, often used in dissertations , theses, and research papers . Literature reviews give an overview of knowledge on a subject, helping you identify relevant theories and methods, as well as gaps in existing research. Literature reviews are set up similarly to other  academic texts , with an introduction , a main body, and a conclusion .

An  annotated bibliography is a list of  source references that has a short description (called an annotation ) for each of the sources. It is often assigned as part of the research process for a  paper .  

A systematic review is secondary research because it uses existing research. You don’t collect new data yourself.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

Turney, S. (2023, November 20). Systematic Review | Definition, Example & Guide. Scribbr. Retrieved September 10, 2024, from https://www.scribbr.com/methodology/systematic-review/

Is this article helpful?

Shaun Turney

Other students also liked, how to write a literature review | guide, examples, & templates, how to write a research proposal | examples & templates, what is critical thinking | definition & examples, get unlimited documents corrected.

✔ Free APA citation check included ✔ Unlimited document corrections ✔ Specialized in correcting academic texts

• Skip to main content
• Skip to "About this site"

Language selection

• Français
• Search and menus

Analytical Studies: Methods and References Innovation Ecosystem Performance Indicators: Review of the Literature

Skip to text

Acknowledgements

1 introduction, 2 review of innovation ecosystem indicators, 3 innovation ecosystem performance measurement framework, 4 discussion and conclusion.

Text begins

The authors would like to thank Danny Leung, Amélie Lafrance-Cooke, Ryan Kelly, Meade Conor and Catherine Beaudry for their valuable comments and suggestions.

The concept of innovation ecosystems has recently gained increasing interest among researchers, practitioners and policy makers. This article uses a systematic literature review approach by drawing on studies that bring together the most recent knowledge on innovation ecosystem performance indicators. Based on the indicators identified in these studies, the paper builds an integrated framework for analyzing the performance of innovation ecosystems based on their inputs and outputs. The results of this study allow researchers to observe the actors, activities and products that support the development of the ecosystem, including human capital, research and development, institutions, infrastructure, the business environment, financial support, products and innovation linkages, employment, and production.

As nations have made significant progress in improving their macroeconomic and institutional frameworks, attention has shifted to other engines of productivity, with the emergence of technology and innovation as core elements of the overall development process. One of the drivers of increased prosperity and economic efficiency is the extent to which an economy can adopt existing technologies or develop new technologies to improve the productivity of its industries (see, e.g. , Acs and Armington, 2004).

The concept of an innovation ecosystem has been widely discussed in the fields of strategy, innovation, and entrepreneurship, as well as among researchers, with a rapidly growing literature (see, e.g. , Gomes et al. , 2018). Researchers have developed a series of definitions related to innovation ecosystems, but a common definition would be to consider them as a dynamic set of actors (firms, universities, inventors, etc. ), activities and resources, evolving through institutions and relationships, that are important for the innovation performance of a region or country (see, e.g. , Granstrand and Holgersson, 2020).

This study explores some of the conceptual indicators used to measure the performance of innovation ecosystems. The aim of this review is to list the indicators most used in the literature on innovation ecosystems. Thus, the common framework does not suggest the construction of new indicators or the selection of ones; it describes the most widely used indicators. These indicators are based on high-quality statistics and sound, measurable analytical principles. However, this literature review considers that the choice of indicators, according to their use in the literature, may be influenced by the period of publication of the articles, or the scope (macroeconomic or microeconomic) of the articles.

This literature review on innovation ecosystems was commissioned by Statistics Canada to address the findings of the science, technology and innovation (STI) Data Gaps Initiative and to respond to the data needs of stakeholders by developing a broader, integrated measurement framework for profiling and mapping innovation ecosystems in Canada (Statistics Canada, 2021). Governments have progressively sought to adopt the concept of innovation ecosystems as a tool for promoting national and regional competitiveness, innovation, and growth. Note  1 For example, the National Research Council (NRC) has also launched a few initiatives since the early 2000s to support the growth of innovative firms clustered around NRC research institutes in different regions of Canada. In 2017, the Canadian government considered actions related to its Global Innovation Clusters in the budget, which include accelerating innovation through the provision of $950 million over five years to support several business-led innovations in superclusters.

These initiatives address the problem of Canada’s poor performance in turning its excellent research and technology into commercialized innovations (Beaudry and Solar-Pelletier, 2020). The shift to a digital economy in Canada since the COVID-19 pandemic has contributed to a surge in technology adoption through more dynamic, IT-intensive and entrepreneurial services. Note  2 It is therefore important to measure the performance of an innovation ecosystem to assess the effectiveness of the implementation of these initiatives. The performance analysis must be conducted over a given period, so the evolution of the performance over time can be observed and investment expenditures to develop innovation ecosystems can be adjusted. It is also important to be able to compare innovation ecosystems to have a precise development strategy.

The paper makes several contributions to the literature on innovation ecosystems. First, it shows how the literature measures the performance of innovation ecosystem indicators by identifying some relevant articles, which may be useful for researchers interested in identifying such measures. Second, the paper proposes a standardized and integrated framework that can be replicated in different contexts, especially in Canada. The common framework includes 8 indicator categories and 38 variables. The framework broadly follows in the footsteps of the Oslo Manual, but it differs by considering the full complexity of an innovation ecosystem and by not limiting innovation solely to the business perspective ( OECD and Eurostat, 2018). Third, the paper identifies several research opportunities based on the common innovation ecosystem performance measurement framework.

The remainder of the paper is organized as follows. Section 2 refers to a detailed literature review of innovation ecosystem indicators. Section 3 presents the common innovation ecosystem performance measurement framework. Section 4 presents a discussion of the common framework (with a synthetic example), shares opportunities for further research and concludes the study.

The concept of an innovation ecosystem is inspired by theories that study how individuals and organizations interact and collaborate formally and informally. The theoretical underpinnings of the concept cover industry clusters, as well as geographic, social and cognitive proximities, all of which have been shown to have a positive impact on a business’s propensity to innovate. A well-functioning ecosystem is necessary to increase the effectiveness of entrepreneurial and innovative activities, create jobs, and set the conditions for economic prosperity (Teece, 2007). Therefore, it is important to maintain and expand the impact of the ecosystem, which requires mapping and creating metrics to quantify and identify performance gaps, and possibly correct potential blockages (Adner, 2006).

2.1 Innovation input and output

This review proposes a common framework, illustrated in Figure 1, which the reader can use to compare and analyze the different papers studied in the following sections. These papers have analyzed innovation ecosystems in clusters, superclusters, small and large regions, and countries. The figure allows readers to examine the evolution of the frameworks proposed in the literature on innovation indicators, as well as their common features and specificities. These frameworks are built with indicators using reliable statistics and rigorous analysis methods, so that they are measurable and comparable at different geographical scales and over time. Most of these indicators help stimulate policy debates and highlight new dynamics, and this is why they are mostly developed by researchers from government agencies.

The literature on the performance of innovation ecosystems proposes indicators that can be adapted to an input–output structure. The input–output structure is based on the idea that the performance of an innovation ecosystem depends on its importance in terms of employment and production and is moderated by favourable conditions and significant investments. The inputs are the conditions that favour the creation of innovation, and outputs are the direct outcomes and indirect economic improvements that result from innovation.

The title of figure 1 is “Common Innovation Ecosystem Framework”. It gives the components of an innovation ecosystem framework.

Figure 1 consists of two main branches (top and bottom). The main branch at the top (inputs) is made up of the two conditions favoring the emergence of innovations. On the one hand (top), these are enabling conditions, i.e. human capital and research, infrastructure and institutions, and innovation links; and on the other hand (bottom), market conditions, i.e. financial support and business dynamics.

The main branch at the bottom (outputs) is made up of the two conditions accounting for the effect of innovation production. These are, on the one hand (top), innovation impacts, which are knowledge outputs; and on the other hand (bottom), economic impacts, which are employment and local production.

Source: Authors’ calculations.

Innovation inputs are divided into two groups. The first group includes enabling conditions, such as human capital, research and development (R&D), infrastructure, institutions, and innovation linkages. Human capital shapes the degree to which a country’s workforce can participate in innovative activities. Higher levels of human capital are associated with higher levels of innovation and faster technology diffusion (Heunks, 1998; Hadjimanolis, 2000; Barker and Mueller, 2002; Romijn and Albaladejo, 2002; Crescenzi, 2005; Arvanitis and Stucki, 2012). Sen (1999) describes education as a resource for more effective participation in the economic and political life of the nation. This can include improvements in education and training, which create a pool of skilled workers who can assimilate and improve imported technologies or adapt them to local conditions.

The global economy has become more sophisticated, and it is now evident that to be competitive, it is essential to enhance the human capital endowments and research capabilities of the workforce, whose members must have access to new knowledge and be continuously trained in new processes and the use of the latest technologies. The issue of the integration of a country and its private sector into the global economy has also become increasingly important over the past decade, in particular in the discussion of interactions between R&D actors.

In an increasingly interdependent regional and global economy, a more open focus on close linkages between foreign academics, entrepreneurs and innovators improves growth prospects through increased efficiency and productivity (Niosi and Bas, 2001; Santoro and Gopalakrishnan, 2001; Feller et al. , 2002; Busom and Fernández-Ribas, 2008; Eom and Lee, 2010; Soetanto and Jack, 2011; Arvanitis and Stucki, 2012). Greater integration into the economy also serves as an important channel for absorbing technological advances, including those from abroad, such as improvements in management practices, and positive effects on human capital development.

Porter (1990, 1998) provides useful information in his analysis of the role of education in bringing an economy’s productive system to full scale. He highlights the importance of close collaboration between educational institutions and potential employers, as universities and other institutions of higher learning are called upon to adapt to the changing needs of industry. An effective R&D system is therefore essential to transfer knowledge and develop innovations (see Jaffe, 1989; Audretsch and Feldman, 1996; Cohen et al. , 2002; Keller, 2002; Bottazzi and Peri, 2002; Bode, 2004; Döring and Schnellenbach, 2006; Woodward et al. , 2006; Drucker and Goldstein, 2007; Kirchhoff et al. , 2007).

The stability of the institutional environment and the quality of infrastructure are seen as critical to private sector development. Good regulation of institutions generally means good public management and, inevitably, fewer wasted resources, the surplus of which could contribute more directly to improved productivity and growth. High-quality infrastructure, especially in machinery and equipment that integrates new technologies such as information and communications technology (ICT) assets, is essential to stimulate innovation and improve the skills of employees, thus contributing to the productivity and competitiveness of businesses (Porter, 1990, 1998; Arthurs et al. , 2009).

The second group of innovation inputs is market conditions, which include business dynamics, and finance and support. Firms develop and implement new processes that increase productivity and competitiveness by turning ideas and inventions into new goods and services that feed markets (Porter, 1990). This creates more high-value jobs and contributes to increased national wealth that can support public investments in education, health, infrastructure, and social programs. Innovation plays a key role in the competitiveness of businesses in the modern global economy. With the right knowledge and skills, entrepreneurs can better assimilate cutting-edge technologies and changing business practices, making them more likely to choose innovation-based business strategies that boost their competitiveness.

Firms benefit from a variety of research talent and financial support to design and create new knowledge, products, processes, and other innovation activities and to effectively use ICT to improve productivity. While the decision to pursue an innovation-based business strategy rests with firms, governments, and research organizations play an important role in supporting business innovation by providing financial resources, directly and indirectly. This support provides firms with an environment of reliable access to talent, knowledge, and capital to support idea development and commercialization activities (see Cheshire and Magrini, 2000; Salter and Martin, 2001; Warda, 2001; Bilbao-Osorio and Rodriguez-Pose, 2004; Wolfe and Gertler, 2004; Rodríguez-Pose and Crescenzi, 2008).

Innovation outputs are also divided into two groups. The first group is innovation impacts, which include knowledge outputs. Knowledge outputs are the observed effects of innovations. Some outputs, such as product innovations, can have a direct influence on markets, while other innovations, such as process innovations, improve the quality or marketing of services, thereby enhancing the visibility or reputation of these services ( OECD and Eurostat, 2018).

The second group of innovation outputs is economic impacts, which include employment and production. Firms’ investments in innovation are rewarded when they result in increased productivity, greater production capacity, wages, and employment, as well as increased export market share in R&D-intensive industries. Thus, firms that invest in innovation will be more profitable and contribute to the growth of the local economy. Innovation is widely regarded as a driver of productivity, which in turn is essential to higher wages, profitability for investors and improved economic welfare in the long run (Bednarzik, 2000; Kolko, 2000; Acs and Armington, 2004; Hecker, 2005; Cukier et al. , 2016).

In the next sections, the review presents studies on innovation ecosystem indicators that analyze their performance from the finest level (cluster) to the broadest level (country). See Appendix Table A.1 for an outline of these studies.

2.2 Cluster innovation ecosystem

Firms gain a competitive advantage not only from their own capabilities, but also from resources and capabilities located in the business environment geographically close to the firm. Some empirical research has shown that clustering can have significant positive effects on firm productivity, innovation, profitability, growth, and resilience (Beaudry and Breschi, 2003; Duranton and Puga, 2004; Boschma, 2005; Gordon and McCann, 2005; Martin et al. , 2011; Combes and Gobillon, 2015; Delgado and Porter, 2017). An industrial cluster refers to a group of businesses and organizations in a sector that are geographically located together, are interconnected, share common elements and are complementary to each other (Porter, 1990). This definition of clusters sets the stage for the innovation ecosystem framework used in this analysis (Granstrand and Holgersson, 2020).

The first framework analyzed is that of Arthurs et al. (2009), who propose a simple cluster framework consisting of 34 variables. They analyze the implications of this framework for the current state of eight NRC cluster initiatives by exploring some of the conceptual issues and methodological challenges encountered in the analysis of the NRC -supported clusters. Note  3 The NRC has launched several initiatives to support the growth of innovative firms clustered around their research institutes in different regions of Canada.

Much of the analytical and policy work on clusters has been based on a diverse set of quantitative measures operating at very different conceptual and spatial scales. The authors explain that STI statistics and derived indexes are inadequate in capturing the core structures and relationships that are critical to understanding the state and performance of a cluster. Indeed, these statistics do not reflect emerging technology areas, tacit knowledge, and market linkages. They are sometimes unavailable at the level of geographic disaggregation required for small clusters because of privacy restrictions. Consequently, their methodology for cluster analysis relies mainly on interviews and surveys of firms and innovators because they provide rich information on how individual clusters perform.

Their framework is built on earlier work (Porter, 1990, 1998) and incorporates the findings of the Innovation Systems Research Network regarding clusters in the Canadian context (Wolfe and Gertler, 2004). Innovation inputs consider enabling conditions and investments by firms to develop their innovation capabilities. The authors consider the conditions of the cluster environment that influence its performance, such as access to skilled human capital; current infrastructure, including the quality of transportation; and regulations that shape the business climate. Firms’ investments include support from organizations (such as NRC and government policies), customers and competitors in the development of the innovation ecosystem (here the cluster). The authors also emphasize the importance of interactions within the cluster in terms of innovation activity and add an international dimension to this interaction.

Regarding the output measures that characterize cluster performance, the authors measure cluster employment in terms of the number, size and structure of firms by adding spin-off firms. They consider the exports of firms and their growth in terms of production. They complete their framework with a measure of cluster dynamism in terms of direct innovation outputs and R&D spending. Note  4

One feature of this framework, compared with others, is that it views innovation ecosystems as a dynamically evolving system with life cycles that can be latent, developing, established or transformational. The needs and interactions of cluster actors differ according to the stage of development of the cluster, and cluster policies must evolve accordingly, according to the authors. Furthermore, compared with the present paper’s common framework, they do not consider the investments in R&D that are important to guarantee an efficient innovation capacity for the ecosystem. They also do not consider ICT assets as a useful infrastructure for innovation that can help companies and ecosystem actors generate ideas, be productive and create innovation. Although their set of indicators is relatively simple to understand, the indicators are based on opinions and generally do not generate quantitative results and are resource intensive. In addition, the authors do not explain precisely how to construct many of their variables, making replication difficult.

The next framework analyzed is that of Beaudry and Solar-Pelletier (2020), which focuses on superclusters. Superclusters are much closer to the concept of innovation ecosystems because they are generally more technology oriented.

Beyond the geographical proximity of interconnected businesses within a sector, the supercluster considers the density of a knowledge network of actors around a core technology and the ability to collaborate with customers, suppliers, and universities. The authors’ framework, from Innovation, Science and Economic Development Canada, focuses on disruptive technologies such as big data analytics, artificial intelligence, advanced materials, additive manufacturing and blockchain. This interest is also driven by the Canadian government’s Global Innovation Clusters program to resolve Canada’s innovation paradox, which highlights the difficulty of translating scientific and technological performance into effective solutions and commercial success. Supercluster partners will be able to help strengthen regional innovation ecosystems, improve the growth and competitiveness of participating businesses, and maximize economic benefits (including well-paying jobs and prosperity for Canada) through the adoption of new and potentially disruptive technologies within innovation ecosystems (ISED, 2024).

Thus, according to the authors, Canada’s innovation policy framework must be rethought to adapt to new ways of organizing and governing innovation. Their proposed framework consists of 15 variables. This framework includes only two innovation inputs, namely the firm’s R&D expenditures, to measure the firm’s commitment to innovation. However, it includes several measures of innovation output because the aim of the study was to focus on direct measures of innovation to compare different superclusters. The authors measure the innovation inputs by the number of collaborative projects or professional actors working together on the same project, between private, academic, and public organizations. They also measure business investment in R&D. The authors propose measuring innovation outputs by the capacity of firms to produce effective innovation linkages and by their ability to build the best teams and mobilize the right set of resources to foster innovation. Second, they consider a measure of knowledge output as the number of new products or processes.

Furthermore, the authors include the measures of employment and production of their innovation ecosystem in their framework. On the one hand, they look at how the ecosystem grows in terms of jobs and firms created, and high-growth firms that contribute to more than 50% of new jobs and sales in given sectors. On the other hand, they measure the output of the ecosystem by its ability to create wealth and increase economic growth and competitiveness through a rise in exports, productivity, and gross domestic product (GDP).

As mentioned above, the conditions for innovation are not developed in this framework compared with the present study’s common framework. The authors are limited to the sole investment of firms in R&D, which is not sufficient to understand the environment necessary for the creation of innovation in ecosystems. The authors recognize that this framework does not consider the human capital devoted to R&D or to tasks related to commercialization, and the mobility of the labour force in the innovation process. They also recognize the difficulties of their indicators in measuring the successful adoption of technologies that is likely to occur through informal relationships and the sharing of tacit knowledge.

2.3 Regional innovation ecosystems

The indicators in the previous frameworks are measured either at the firm level, to assess the performance of firms within clusters relative to more isolated firms, or at the cluster level, to examine the overall performance of the organizations that form a cluster. However, it is beneficial for both levels to be examined together to assess the extent to which their agreement is beneficial to the business and its environment. This is what the regional ecosystem framework allows, by promoting coherent and effective coordination of innovation that can take place at the subnational level ( e.g. , provincial, regional or city level). Thus, it enables an alternative structure of linkages and relationships beyond geographically bounded clusters.

Cukier et al. (2016) are mapping the innovation ecosystem in eastern Ontario to better understand the breadth of innovation services that can help build a competitive advantage to attract businesses and investors and help stimulate the business environment. Their framework, called the Innovation Ecosystem Scorecard, is built on an exhaustive document review, a comprehensive data analysis and discussions with key stakeholders to develop a better overall understanding of the ecosystem’s drivers, with a particular focus on the dynamics of innovation in small communities. Note  5 Since the authors are interested in innovation ecosystems in small regions, they do not look at the innovation linkages and the support and funding provided in these regions, which are generally lower, on average. They focus more on providing a wider range of indicators that can describe more important aspects of human capital, economic dynamics, productivity, and employment, and economic well-being than previous frameworks.

They build their framework on an analysis inspired by the U.S. Economic Development Administration’s framework based on business and economic development data, including job supply and demand (U.S. Economic Development Administration, 2010). This approach has allowed them to develop an inventory of key players and intermediaries in the ecosystem, including investors, large employers, incubators, business service providers and government agencies. They also use an assessment of innovation models and methods such as the Global Entrepreneurship Monitor to understand the conditions that support entrepreneurship and innovation performance, such as the availability of financing, government policies and programs, education, R&D transfer, business and physical infrastructure, and cultural and social norms. Finally, the authors supplement their study with consultations with key stakeholders to understand the components of the ecosystem and their assessment of current programs and needs.

They developed a framework consisting of 22 variables. This framework proposes a diverse set of variables whose purpose is to measure the inputs and outputs of innovation. Among the most extensive categories are (1) human capital, which examines the characteristics of the regional population and workforce (high educational attainment, young adults, and innovation-related occupations and jobs); (2) economic dynamics, which addresses local business conditions (size of existing establishments and churn of establishments); and (3) employment and output, which assesses economic growth, regional attractiveness and economic well-being (high-tech employment, population and wealth growth, worker and owner earnings, etc. ).

However, compared with the present study’s common framework, this framework does not develop (or develops very few) indicators measuring research assets within human capital or ICT assets within the infrastructure supporting innovation activities. Neither does it develop direct measures of innovation, or the resources and funds available to entrepreneurs and firms to innovate.

The framework used by Cukier and co-authors, from the U.S. Economic Development Administration, has been updated with a more extensive version. This version is presented by Slaper et al. (2016), from the Indiana Business Research Center, who propose a set of almost 70 variables that can help regional leaders reach a strong consensus on regional strategic direction in the United States. The choice of variables was based on a broad coverage of the empirical and theoretical literature. Since it is a more regional definition of the term “innovation ecosystem,” it encompasses a larger dimension of the network concept, with innovation sites, incubators, university–business research partnerships, investment capital networks and relevant workforce development programs.

The proposed indicators are more comprehensive in each of the subcategories. The framework the authors propose also uses categories based on innovation inputs and outputs to measure innovation capacity and production potential. The inputs include human capital and knowledge creation, which indicates the extent to which a region’s population can engage in innovative activities. They also include business dynamics, which are composed of a measure of firm dynamics that assesses the competitiveness of a region by tracking the entry and exit of individual firms, and a measure of firm profile that assesses local business conditions and the resources available to entrepreneurs and firms.

Outputs are divided into two categories: the employment and productivity index that describes economic growth, regional attractiveness or direct outcomes of innovative activity, and the economic well-being index that explores living standards and other economic outcomes. Compared with Cukier et al. (2016), the authors add a measure of local business conditions and resources available to entrepreneurs and firms. In addition, they add a state context and social capital index. The former identifies the financial support and useful business dynamics that help understand the innovative environment. The l identifies the regional advantages of collaborative networks that underpin a community’s ability to address its challenges. These two additions remain optional, because the theory behind them is still under development and the data are not available at fine spatial scales, especially the social capital data. Therefore, in the paper, only the first set of state context variables has been included.

The variables measuring the current level of human capital, investment in innovation, and the performance of innovation activity in terms of employment and output are more varied and consider more aspects than the previous framework. However, compared with the present study’s common framework, the authors do not consider innovation linkages; however, they briefly discuss them in the social capital index.

2.4 Country innovation ecosystem performance

This section analyzes proposed frameworks on the performance of innovation ecosystems at the country level. It places greater emphasis on the role played by governments, the infrastructure and institutional framework, and international cooperation to improve local innovation production. These studies therefore support investments to create globally competitive scale and capacity in key areas of strength and opportunity, as well as a seamless integration of organizations, activities, and funding mechanisms across the innovation ecosystem.

Cannon et al. (2015) propose an analytical framework from the Science, Technology and Innovation Council that examines 28 indicators measuring the performance of innovation ecosystems in Canada. They present an innovation ecosystem framework in which the government plays a central role. The framework proposes greater government investment with innovation support programs designed to encourage collaboration across the innovation ecosystem. Governments play an important role in supporting and encouraging business innovation. They should therefore routinely provide more direct support to high-risk, high-reward business R&D, especially in industries of economic importance to Canada. Higher education institutions should also leverage government programs to increase their research and innovation capacity.

Thus, the authors emphasize human capital, research and business resources, and innovation support and finance as important inputs for innovation. They place a strong emphasis on R&D expenditures by business enterprises that are most closely linked to product and process innovation. Increased investment in R&D must be accompanied by increased investment in other knowledge assets, including talent in the business, engineering, science, and health sectors.

The authors measure the performance of innovation ecosystems on innovation output in large businesses and in high-growth small and medium-sized enterprises (SMEs) with the potential to become major players. They recommend increasing the number of large innovative firms to improve future competitiveness and employment growth, because large firms are often more productive and tend to invest and export more than small firms.

However, their framework has little regard for ICT assets in infrastructure and institutions and accounts very little for ICT assets and the business dynamics needed to develop coherent and effective innovation capital. Moreover, with the focus on the inputs of innovation, the outputs of innovation are not well developed, especially with respect to the linkages between innovations and the effect on employment.

In their framework, similar to the present study’s common framework, Hollanders and Es-Sadki (2021) focus on the conditions that capture the key drivers of innovation performance in 27 European countries and distinguish between human resources, attractive research systems and infrastructure through digitization. They also focus on investments, which capture investments in the public and business sectors, distinguishing between funding and support, business investments, and the use of IT.

This paper follows in the footsteps of the Oslo Manual, which mainly considers innovation from the firm’s point of view ( OECD and Eurostat, 2018). Although firms play an important role in innovation, universities, non-governmental organizations, not-for-profit organizations, autonomous researchers, and others are also known to contribute to the performance of an innovation ecosystem. The common framework reflects this complex picture.

Innovation outcomes are measured by innovation activities that capture different aspects of innovation in the business sector and distinguish between innovators. They also include impacts that capture the effects of firms’ innovation activities and distinguish between impacts on employment and impacts on output, such as sales and environmental sustainability. Using this set of variables, the authors classify the innovation performance of European countries as innovation leaders, strong innovators, moderate innovators, and emerging innovators.

In addition to these variables, a contextual analysis of the impact of structural differences between countries was assessed. To better understand the differences in performance between the innovation indicators used in the main measurement framework, a set of contextual indicators was added. These contextual indicators measure differences in the performance and structure of the economy, business activities and entrepreneurship, the introduction of innovation, the institutional and legal environment, climate change performance, and demographics. These variables were added to the authors’ framework, including those that allow for international comparison, bringing the number of variables to 51. This provides the authors with a framework whose variables are distributed in a balanced way between the different categories of innovation inputs and outputs, allowing them to consider a varied set of key concepts related to innovation ecosystems.

The Organisation for Economic Co-operation and Development (OECD) (2017) has also proposed a framework similar to the present study’s common framework, with the aim of helping governments of OECD and other countries (60 countries in total) to design more effective science, innovation and industry policies in a rapidly changing digital age. The selected indicators were developed to rely on high-quality statistics and robust analytical principles and be measurable internationally, over time, and with room for improvement. This framework focuses on the impact of knowledge and digital transformation in developed country economies in the context of today’s rapidly changing digital technology landscape. It proposes a set of over 80 indicators to measure the performance of innovation ecosystems. This set of indicators is derived from previous work by academics, the OECD , Eurostat and the World Bank.

Within innovation inputs, this study focuses on human capital through knowledge, talent and skills and examines the knowledge assets that many businesses and governments see as current and future sources of long-term sustainable growth. The authors also develop indicators measuring research excellence and collaboration to help inform the policy debate through a series of metrics on the variety and nature of knowledge dissemination mechanisms in the digital age. In addition, they consider business innovation by exploring the dynamism of the business sector and the framework conditions essential for innovation.

Unlike other frameworks, this one adds leadership and competitiveness by including indicators measuring how countries seek to develop their competitive strengths and the extent to which economies can integrate and specialize along global value chains. This framework, however, is not sufficiently balanced in terms of variables in the categories of innovation inputs and outputs, compared with the previous one. The study places a strong emphasis on innovation-enabling investments, especially in ICT assets, and completely omits tangible infrastructure and innovation-enabling institutions, especially for a national study.

López-Claros and Mata (2010) developed a set of 61 variables that allowed them to construct the Innovation Capacity Index to assess how successful countries have been in developing an innovation-friendly climate capacity. The authors combine indicators used by various international organizations, including the European Commission’s Joint Research Centre, the International Monetary Fund, the OECD , the United Nations and the World Bank. This informs policy makers and entrepreneurs around the world (131 countries) about the wide range of country-specific factors that underpin innovation.

Like the previous frameworks, the authors’ framework presents indicators that can be categorized into inputs and outputs of innovation. The authors present the factors that are essential to creating an environment conducive to innovation and the types of initiatives that will contribute in some way to boosting productivity and, hence, economic growth.

In terms of innovation inputs, the authors emphasize the institutional environment that favours innovation, notably through good governance, a good country policy assessment, and a regulatory and legal framework that favours entrepreneurship. In addition, they note the importance of a good level of human capital, training, and social inclusion. Well-developed human capital resources increase the potential for innovation, which in turn increases a country’s ability to innovate and achieve sustained productivity growth. Finally, the authors stress the importance of adopting and using ICT and investing in R&D. Regarding the outputs of innovation, the emphasis is only on direct measures of innovation and knowledge products, such as patents and trademarks.

This framework is very well informed in terms of infrastructure variables and institutions. However, compared with the present study’s common framework, it lacks measures of the entrepreneurial conditions that guarantee a good innovation capability. In addition, the authors omit measures of employment and innovation linkages from their framework. Also, some underlying factors ( e.g. , budget deficit, education spending and R&D intensity) are difficult to measure, requiring surveys to capture perceptions of firms or civil society.

The World Intellectual Property Organization (WIPO) (2021) provides a framework, similar to the present study’s common framework, based on new data and analysis on the state of global innovation. This framework allows readers and policy makers to compare the performance of the innovation ecosystems of over 132 economies.

As such, the framework is designed to provide the most comprehensive picture of innovation possible, with the index comprising approximately 80 indicators, including measures of each economy’s policy environment, education, infrastructure, and knowledge creation. It has the advantage of being supported by empirical studies examining the choice of indicators as inputs and outputs of innovation. This is notably the case of Araujo Reis et al. (2021), who examine the relationship between innovation input and output with the Global Innovation Index. They show that innovation input has a significant and positive effect on innovation outputs in countries.

The WIPO framework’s translation of an economy’s investments in innovation—in the form of R&D, education, infrastructure, and strong institutions supporting innovative activities—into innovation outcomes is no small task. This study develops an effective innovation system that balances knowledge creation, exploration, and investment (the inputs of innovation) with the generation of ideas and technologies for application, exploitation and impact (the outputs of innovation).

The WIPO study is unique in that it was developed and presented during the COVID-19 pandemic, which was a trigger for innovation in economies, including the manufacture and implementation of vaccines, teleworking, and online services. According to the authors of this study, the key indicators that have been most affected by the pandemic are science, global innovation products, scientific publications, R&D expenditures, international patent filings and venture capital (VC) operations.

This section returns in more detail to the composition of the common framework proposed in Figure 1. The different studies examined above, which cover fine units such as clusters to very large units such as countries, allow more than 400 indicators of innovation performance measures to be gathered, divided between innovation inputs and outputs.

After similar indicators were combined by removing duplicates and overlaps, the indicators were ranked according to bibliometric impact importance. The nine studies presented above, as well as the sources used in these studies, are considered to justify the choice of indicators. The sources employed to validate the use of an indicator are empirical or theoretical studies that have clearly demonstrated that an indicator has an effect on innovation (innovation input) or that an indicator is affected by innovation (innovation output). The most relevant sources, cited at least 100 times in the literature, are retained. Note  6 The more a journal article is cited, the more it can be said to have influenced subsequent scientific research. This leaves 153 papers.

The indicators are then classified based on those 153 papers, from the least used to the most used. Indicators with fewer than 5 citations are weakly used, indicators with 5 to 10 citations are moderately used, and indicators with more than 10 citations are strongly used. The range of four items is the standard deviation and the mean of the distribution of the number of items per indicator. The minimum number of articles cited per indicator is 1, and the maximum number of articles cited per indicator is 20, for a total of 97 indicators listed. From the set of 97 indicators, 12 are heavily used, 26 are moderately used and 59 are lightly used in the literature. In the proposed common framework, only medium- and high-use indicators are included to reflect their role in the literature. However, on request, users can access an appendix containing all the indicators investigated in the literature in this study, to view low-use indicators and adapt them to the context of their analysis. This leaves 38 variables, as illustrated below.

Common innovation ecosystem indicators

• Education expenditures
• International students
• Knowledge programs
• Researchers and technicians in R&D
• Scientific and technical articles
• Tertiary education Note †
• University-based knowledge spillovers Note †
• Young adult population
• Broadband connections Note †
• Doing business index
• ICT investment
• Local availability of capital Note †
• Logistics performance
• Business–university collaboration
• Business local collaboration
• Business international collaboration
• Business expenditures on R&D Note †
• Business incubators Note †
• Establishment size
• High-tech industry employment Note †
• Industry concentration
• Institutionally based start-ups Note †
• International workers
• Proprietorship rate Note †
• Academic expenditures on R&D Note †
• Foreign direct investment
• Government expenditures on R&D Note †
• Gross expenditures on R&D Note †
• Venture capital investment Note †
• Patent applications Note †
• Product and process innovations
• Designs, copyrights and trademarks
• Change in establishment births Note †
• Establishment churn
• Job growth to population growth ratio
• Exports in high-tech industries Note †
• Gross domestic product growth
• Proprietor income to wages and salaries

Notes: The rest of the indicators are moderately used in the literature. R&D = research and development; ICT = information and communications technology.

Source: Authors' calculations.

The fact that the majority of indicators are weakly replicated by other studies may indicate that they either are newly used or reflect contextual objectives of researchers. For example, some recent studies consider new indicators, such as those related to the effect of innovation on environmental performance. Furthermore, the availability of a variable does not mean that a particular indicator is robust. Some variables will be more widely used because they are more widely available, introducing potential bias in the results of the analysis.

To understand the bias, two alternative classifications are proposed. The first classification includes only recent papers, from the last 10 years, reducing the number of papers analyzed from 153 to 25. Indicators with fewer than three citations are weakly used, indicators with three to five citations are moderately used and indicators with more than five citations are strongly used. The interval of two between classes is the standard deviation of the distribution of the number of papers per indicator (the mean is three). The minimum number of articles cited per indicator is 1, and the maximum number of articles cited per indicator is 8, for a total of 81 indicators listed.

The number of moderately or heavily used indicators has dropped from 38 to 32. This new ranking, as presented in Appendix B, does not change the innovation outputs in relation to the common framework. However, it does change the innovation inputs, especially the indicators measuring the quality of infrastructure and institutions, which go from five to two. The indicators of market conditions also decrease. The measures of business incubators, start-ups and public spending on R&D disappear. In sum, considering only recent studies restricts the number of indicators without adding indicators different from the common framework. The common classification is not biased by older articles.

The second classification considers the fact that macro variables are generally easier to obtain than micro analyses, and this could result in a higher number of macro studies. This would result in more macro variables, which are less suitable for an analysis of ecosystem performance at the regional level. Therefore, a bibliometric analysis was conducted using only the microanalysis of the innovation ecosystem to see whether the list of indicators selected would be different.

The new classification reduces the number of papers analyzed from 153 to 142. Indicators with fewer than five citations are weakly used, indicators with four to eight citations are moderately used and indicators with more than eight citations are strongly used. The interval of three between classes is the standard deviation of the distribution of the number of papers per indicator (the mean is four). The minimum number of articles cited per indicator is 1, and the maximum number of articles cited per indicator is 16, for a total of 56 indicators listed.

The number of moderately or heavily used indicators has dropped from 38 to 30, as presented in Appendix C. A definite bias toward macroeconomic indicators is seen. Although there are only 11 macroeconomic studies, these construct 42% of the total indicators listed, 24% of the moderately or heavily used indicators. In particular, the enabling conditions variables decrease significantly from 16 to 8. The rest of the indicators remain more or less the same, and this classification adds a new one, the measure of establishment churn, which becomes an average indicator used in the literature.

The following points briefly discuss how to measure the indicators, especially in the Canadian context. For all the indicators described here, data sources for measuring them are available upon request from Statistics Canada at the national, provincial and sometimes municipal levels.

3.1 Human capital and research

Education expenditures: Spending on education is a good indicator of the priority and level of commitment a region places on education and human capital development, which have positive implications for innovation. Education provides the basic and advanced knowledge and skills that help individuals pursue and succeed in higher education, research and employment in innovation-related fields (see López-Claros and Mata, 2010; Cannon et al. , 2015; Slaper et al. , 2016; OECD , 2017; WIPO, 2021). To better represent innovation, education spending can be restricted to specific areas such as tertiary education or science, technology, engineering and mathematics (STEM) programs.

International students: International students reflect the importance of academic diversity as an active channel for the dissemination of knowledge (see Cannon et al. , 2015; Slaper et al. , 2016; OECD , 2017; Hollanders and Es-Sadki, 2021; WIPO, 2021). International students can be measured as the number of students from foreign countries. This number may contain international PhD students in particular. Not all international students stay—they may choose to return to their home country after graduation. Therefore, it may be better to count the number of postgraduation work permit applicants who stay and work in Canada after graduation.

Knowledge programs: These programs indicate the quality of learning outcomes and creative thinking of human capital (see López-Claros and Mata, 2010; Cannon et al. , 2015; OECD , 2017; Hollanders and Es-Sadki, 2021; WIPO, 2021). This indicator can be measured as the Programme for International Student Assessment scales in reading, mathematics and science or Programme for the International Assessment of Adult Competencies scales in numeracy, literacy and problem solving of the 15-year-old population or workers. However, this indicator is more suitable for a country innovation ecosystem.

Researchers and technicians in R&D: Researchers and other R&D personnel are an essential input to the performance of R&D. Researchers are professionals involved in the design or creation of new knowledge in business, government, higher education and private non-profit organizations. They conduct research and improve or develop ideas, models, techniques, tools, software or operating methods. The number of researchers can be weighted by population (see Porter, 1990, 1998; Wolfe and Gertler, 2004; Arthurs et al. , 2009; López-Claros and Mata, 2010; Cannon et al. , 2015; OECD , 2017; WIPO, 2021).

Scientific and technical articles: Publications are a measure of the effectiveness of the research system, as collaboration increases scientific productivity (see Cannon et al. , 2015; Slaper et al. , 2016; OECD , 2017; Hollanders and Es-Sadki, 2021; WIPO, 2021). This indicator can be measured as the number of scientific publications with at least one foreign-based co-author. It can be measured as the number of scientific and technical journal articles per million people indexed in the journal database, published in the following fields: physics, biology, chemistry, mathematics, clinical medicine, biomedical research, engineering and technology, and earth and space sciences.

Tertiary education: Government policy makers are especially interested in the supply of scientists, engineers and ICT experts because of their direct involvement in technical activities and the ongoing digital transformation (see Heunks, 1998; Hadjimanolis, 2000; Barker and Mueller, 2002; Romijn and Albaladejo, 2002; Wolfe and Gertler, 2004; Crescenzi, 2005; Arthurs et al. , 2009; Atkinson and Mayo, 2010; López-Claros and Mata, 2010; Arvanitis and Stucki, 2012; Cannon et al. , 2015; Cukier et al. , 2016; Slaper et al. , 2016; OECD , 2017; Hollanders and Es-Sadki, 2021). STEM graduates are often employed in management positions. Tertiary education can be measured as university graduates, especially from STEM programs.

University-based knowledge spillovers: This measure estimates how scientific knowledge spreads from universities to neighbouring regions. Since universities are less competitive and profit-driven than industries, their knowledge should spread more widely between institutions and regions. This indicator could also predict the level of patenting in a region. University knowledge spillovers are measured by the distance between the university and the chosen region, or by the number of universities in cities (see Audretsch and Feldman, 1996; Anselin et al. , 1997; Breschi and Lissoni, 2001; Salter and Martin, 2001; Cohen et al. , 2002; Keller, 2002; Bode, 2004; Woodward et al. , 2006; Drucker and Goldstein, 2007; Simonen and McCann, 2008; Casper, 2013; Slaper et al. , 2016; WIPO, 2021).

Young adult population: The young population is guaranteed to include university students and working professionals, who are the most likely to engage in innovative activities. These individuals are also more likely to be less risk averse and more entrepreneurial. Moreover, the growth in the younger population suggests that new residents are likely to enhance the innovative and entrepreneurial aspects of the core community (see Slaper et al. , 2011; Cannon et al. , 2015; Cukier et al. , 2016; Slaper et al. , 2016; Hollanders and Es-Sadki, 2021).

3.2 Infrastructure and institutions

Broadband connections: Broadband supplies high-speed Internet connections to businesses and consumers. Several studies suggest that broadband capacity has a significant positive effect on economic performance (see Crandall et al. , 2007; López-Claros and Mata, 2010; Lehr et al. , 2006; Cukier et al. , 2016; Slaper et al. , 2016; OECD , 2017; Hollanders and Es-Sadki, 2021). Access to high-speed Internet allows businesses and individuals to collaborate from virtually any location. Broadband connections can be measured by the number of residential broadband connections per 1,000 households.

Doing business index: This index reflects businesses’ perceptions of the ease of starting a business in their region (see Porter, 1990; Arthurs et al. , 2009; López-Claros and Mata, 2010; Hollanders and Es-Sadki, 2021; WIPO, 2021). This index would be based on the simple average of the scores for each of three indicators, such as all the procedures officially required, or commonly undertaken in practice, for an entrepreneur to formally start and operate an industrial or commercial enterprise, as well as the time and cost to complete these procedures.

ICT investment: ICT skills are highly relevant for innovation in an increasingly digital economy (see Arthurs et al. , 2009; López-Claros and Mata, 2010; Grundke et al. , 2017; OECD , 2017; Hollanders and Es-Sadki, 2021; WIPO, 2021). The share of investment in ICT is an indicator of the overall development of employee skills. It can be measured in monetary terms as external IT spending (technology products purchased) and internal IT spending (custom software and training), as well as spending on telecommunications and other office equipment (as a percentage of GDP ). It can also be measured in numbers by the number of ICT specialist employees.

Local availability of capital: Local availability of capital indicates the ability of local banks to lend to businesses. Areas with a higher concentration of local bank deposits are more likely to have higher rates of entrepreneurship, innovation, new business creation and overall economic success in a region (see Porter, 1990; Wolfe and Gertler, 2004; Benfratello et al. , 2008; Arthurs et al. , 2009; Kerr and Nanda, 2009; Ayyagari et al. , 2011; Slaper et al. , 2016; WIPO, 2021). Local availability can be measured by the share of local deposits at all banks in the region, which serves as a predictor of local lending, or total capital expenditures.

Logistics performance: Logistics performance can be measured as an index assessing the perception of businesses and residents on the quality of trade and transport infrastructure and the competence and quality of logistics services (see Porter, 1990, 1998; Arthurs et al. , 2009; López-Claros and Mata, 2010; Hollanders and Es-Sadki, 2021; WIPO, 2021).

3.3 Innovation linkages

Business–university collaboration: Innovation collaboration is a platform for disseminating knowledge and accelerating innovation development (see Porter, 1998; Arthurs et al. , 2009; OECD , 2017; Beaudry and Solar-Pelletier, 2020; WIPO, 2021). Academic collaboration involves the active participation of universities and businesses. This indicator can be constructed from a survey question to businesses on the extent to which they collaborate with universities in R&D. It can also be measured by the number of research publications co-authored by universities and private businesses.

Business local collaboration: Local collaboration involves the active participation of businesses with other businesses and institutions, excluding academic organizations (see Porter, 1998; Sorenson and Fleming, 2004; Singh, 2005; Sorenson et al. , 2006; Fleming et al. , 2006; Arthurs et al. , 2009; OECD , 2017; Beaudry and Solar-Pelletier, 2020; Hollanders and Es-Sadki, 2021). This indicator can be constructed from a survey question about the number of firms (usually innovative SMEs) that had cooperation agreements on innovation activities with other firms or institutions during the recent survey period. It can also be measured by the number of research publications co-authored by private or public-private firms.

Business international collaboration: International innovation collaboration refers to active cross-border participation in innovation collaboration (see Arthurs et al. , 2009; OECD , 2017; Beaudry and Solar-Pelletier, 2020; WIPO, 2021). This indicator can be constructed from a survey question asked to businesses about the extent to which they collaborate with foreign businesses or institutions. It can also be measured by the number (or share) of co-inventions (or co-publications) in patent families with inventors located in at least two different countries.

3.4 Business dynamics

Business expenditures on R&D: Business R&D spending reflects the participation of businesses in the creation of new knowledge, which leads to greater economic growth in the region through higher patenting and productivity levels (see Audretsch and Feldman, 1996; Cohen et al. , 2002; Keller, 2002; Bottazzi and Peri, 2002; Bilbao-Osorio and Rodriguez-Pose, 2004; Gulbrandsen and Smeby, 2005; Cannon et al. , 2015; Slaper et al. , 2016; Beaudry and Solar-Pelletier, 2020; OECD , 2017; Hollanders and Es-Sadki, 2021; WIPO, 2021). Business expenditures on R&D can be expressed as the share of total business R&D expenditure in GDP .

Business incubators: Incubators provide services to new businesses in the area and help them to survive and succeed, by transferring a flow of knowledge that increases their capacity for production and innovation (see Mian, 1996; Etzkowitz, 2002, 2003; Chan and Lau, 2005; Grimaldi and Grandi, 2005; Hansson et al. , 2005; Löfsten and Lindelöf, 2005; Markman et al. , 2005; Aerts et al. , 2007; Slaper et al. , 2016; Hollanders and Es-Sadki, 2021). This measure can be calculated by the proportion of business incubators in a region.

Establishment size: Small firms are highly adaptable and can easily change their processes to incorporate new ideas or technologies (see Acs and Audretsch, 1988, 1990; Porter, 1990; Arthurs et al. , 2009; Cukier et al. , 2016; Slaper et al. , 2016; OECD , 2017). Large firms would contribute positively to innovation through the increased availability of funds for R&D (see Acs and Audretsch, 1988; Porter, 1990; Hicks and Hegde, 2005; Arthurs et al. , 2009; Cukier et al. , 2016; Slaper et al. , 2016). Large and small establishments can be measured as the number of establishments with fewer than 50 employees (small firms) or more than 500 employees (large firms) per 10,000 workers. However, there may not be many differences between regions in terms of small businesses per employee. It may be relevant to consider the percentage of employees in small versus large businesses, which varies more from region to region.

High-tech industry employment: Innovative areas contain businesses that require a highly skilled and specialized workforce. Employees in high tech provide services to consumers directly, such as telecommunications, and provide raw materials to innovative businesses in all sectors of the economy (see Audretsch and Feldman, 1996; Klepper, 1996; Kolko, 2000; Feser, 2003; Florida, 2003; Wolfe and Gertler, 2004; Koo, 2005; Arthurs et al. , 2009; Belussi and Sedita, 2009; Tödtling et al. , 2009; Neffke et al. , 2011; Neumark et al. , 2011; Cannon et al. , 2015; Cukier et al. , 2016; Slaper et al. , 2016; OECD , 2017; WIPO, 2021). This is knowledge-intensive service employment, represented by the sum of those in managerial, professional and technical positions as a percentage of the total people employed.

Industry concentration: The concentration measure can help determine the extent to which a country’s industrial system is competitive or uncompetitive in different industrial subsectors. Some studies show that the market structure significantly influences the production of innovations. This market structure can be competitive or poorly diversified and concentrated (see Porter, 1998, 2000; Stuart and Sorenson, 2003; Wolfe and Gertler, 2004; Feldman et al. , 2005; Arthurs et al. , 2009; Glaeser and Kerr, 2009; Delgado et al. , 2010, 2014; Slaper et al. , 2016; OECD , 2017; WIPO, 2021). This measure of concentration can be measured in different ways. It can be measured by the Herfindahl-Hirschman Index for the national industry, defined as the sum of the squared shares of the subsectors, usually in total manufacturing output. It can also be measured by key indicators such as the number of competitors, the relative size of competitors (larger or smaller than the respondent firm or multinationals), or qualitative measures of the intensity of competition in the firm’s market.

Institutionally based start-ups: Start-ups are actively involved in licensing technologies produced by universities and other research institutes to create new and improved goods and services. Technology transfer is an inherently innovative activity, representing the transformation of new knowledge into economic, or marketable, knowledge. A high rate of institutionally based start-ups means more concentrated innovation activity (see Porter, 1990; Almeida and Kogut, 1997; Thurik and Wennekers, 1999; Carree and Thurik, 2003; Audretsch and Keilbach, 2004; Acs and Plummer, 2005; Audretsch and Lehmann, 2005; Slaper et al. , 2016; OECD , 2017). Institutional start-ups can be measured by the number of entities that universities and other non-profit research institutions have formed.

International workers: International workers can raise the level of human capital and foster technological progress. They can increase consumption, living standards and incomes in the long run. Their presence, in particular that of international STEM workers, is correlated with high rates of entrepreneurship and innovation (see Wadhwa et al. , 2008; Hart and Acs, 2011; Kerr, 2013; Langdon et al. , 2013; Slaper et al. , 2016). International workers can be measured as the number of inbound migrant workers relative to the working-age population, particularly in STEM occupations. STEM workers are better positioned to use existing innovations and create new ones.

Proprietorship rate: High proprietorship rates are associated with greater employment growth and entrepreneurial activity conducive to innovation (see Thurik and Wennekers, 1999; Audretsch and Thurik, 2001; Thurik et al. , 2008; Romero and Martínez-Román, 2012; Acs et al. , 2013; Slaper et al. , 2016). The proprietorship rate can be measured as the number of proprietors relative to the total number of employed people. It can also be measured by the proportion of self-employment in the region.

3.5 Finance and support

Academic expenditures on R&D: R&D within universities is a good predictor of the level of patenting. It predicts knowledge transfer to the private sector and subsequent innovation (see Audretsch and Feldman, 1996; Cheshire and Magrini, 2000; Cohen et al. , 2002; Keller, 2002; Bottazzi and Peri, 2002; Bilbao-Osorio and Rodriguez-Pose, 2004; Bode, 2004; Woodward et al. , 2006; Bercovitz and Feldman, 2007; Drucker and Goldstein, 2007; Cannon et al. , 2015; Slaper et al. , 2016; OECD , 2017). It can be measured as the total expenditure on education (current expenditure, capital expenditure and transfers) as a percentage of GDP .

Foreign direct investment: Foreign direct investment increases competition and gives rise to positive externalities and technology spillovers, thereby increasing dynamic efficiency (see Sjöholm, 1999; Branstetter, 2006; Blalock and Gertler, 2008; López-Claros and Mata, 2010; Slaper et al. , 2016; Hollanders and Es-Sadki, 2021; WIPO, 2021). This indicator helps determine the extent to which foreign or domestic firms invest in the region relative to others, improves knowledge and technology transfer, describes the openness of a region’s economy and community, and provides insights into a firm’s ability to improve productivity. Foreign direct investment can be measured as the average over recent years of net inflows characterized by the sum of equity, reinvested earnings, other long-term capital, and short-term capital as reported in the balance of payments. It can be expressed as a share of GDP or can also be divided by the working-age population.

Government expenditures on R&D: Government support for R&D has been found to lead to an increase in the number of patents. There is a significant relationship between industry funding and the quality and quantity of university research. R&D funding leads to greater knowledge creation and more opportunities for innovation (see Audretsch and Feldman, 1996; Salter and Martin, 2001; Cohen et al. , 2002; Keller, 2002; Bottazzi and Peri, 2002; Bilbao-Osorio and Rodriguez-Pose, 2004; Rodríguez-Pose and Crescenzi, 2008; Slaper et al. , 2016; Hollanders and Es-Sadki, 2021). Public funding of R&D includes direct funding through instruments such as grants and public contracts and indirect support through the tax system. It is therefore the sum of government tax support for R&D spending and direct funding of business R&D spending, which can be expressed as a percentage of GDP .

Gross expenditures on R&D: R&D spending is often used to forecast innovation and economic growth, and recent indexes have included R&D spending as a measure of innovation. R&D expenditures predict patenting and productivity levels and also have a positive effect on the creation of new businesses and other measures of innovation and economic development (see Audretsch and Feldman, 1996; Salter and Martin, 2001; Cohen et al. , 2002; Keller, 2002; Bottazzi and Peri, 2002; Bilbao-Osorio and Rodriguez-Pose, 2004; Rodríguez-Pose and Crescenzi, 2008; Arthurs et al. , 2009; López-Claros and Mata, 2010; Cannon et al. , 2015; Slaper et al. , 2016; OECD , 2017; WIPO, 2021). Gross expenditures on R&D represent the total amount of funds spent on R&D activities across all sectors (business, higher education, federal and provincial or territorial governments, private non-profit organizations, and foreign organizations).

VC investment: VC funds are used to bring new ideas to market, bring new technology to market or develop innovative businesses (see Kortum and Lerner, 2000; Engel and Keilbach, 2007; Mann and Sager, 2007; Hirukawa and Ueda, 2011; Rin et al. , 2013; Cannon et al. , 2015; Cukier et al. , 2016; Slaper et al. , 2016; OECD , 2017; Hollanders and Es-Sadki, 2021; WIPO, 2021). Higher VC activity is associated with higher levels of innovation, as measured by patents and total factor productivity growth. VC investment can be measured in dollars as the private capital raised for investment in businesses, which includes seed capital (seed plus start-up) and expansion and replacement capital. It can also be measured by the total number of VC deals. VC can be expressed as a percentage of GDP and averaged over three years.

3.6 Knowledge outputs

Patent applications: The number of patents is an established measure of regional innovation, as it predicts subsequent patent filings and the birth of new industries (see Jaffe, 1989; Jaffe et al. , 1993; Hall et al. , 2001; Agrawal and Henderson, 2002; Bottazzi and Peri, 2002; Bilbao-Osorio and Rodriguez-Pose, 2004; Corrado et al. , 2009; López-Claros and Mata, 2010; Cannon et al. , 2015; Cukier et al. , 2016; Slaper et al. , 2016; OECD , 2017; Hollanders and Es-Sadki, 2021; WIPO, 2021). Patent-related knowledge is shared across networks and spreads to neighbouring regions. This indicator can be measured as the number of patent applications filed by residents.

Product and process innovations: This is the direct measure of innovations. Product innovation is the creation and introduction of a new good or service to the market or an improved version of a previous product (see Wolfe and Gertler, 2004; Arthurs et al. , 2009; Corrado et al. , 2009; Cannon et al. , 2015; OECD , 2017; Beaudry and Solar-Pelletier, 2020; Hollanders and Es-Sadki, 2021). Process innovation is the introduction of a new or improved method of production or delivery.

Designs, copyrights and trademarks: Trademarks, copyrights and designs are an important indicator of innovation, especially for the service sector, because their establishment ensures consistent product quality, productivity growth and competitiveness, which is essential for innovation growth (see Corrado et al. , 2009; López-Claros and Mata, 2010; OECD , 2017; Hollanders and Es-Sadki, 2021; WIPO, 2021). This indicator can be measured as the number of individual trademarks, copyrights and designs filed.

3.7 Employment

Change in establishment births: New businesses stimulate and enhance competition among firms, which increases productivity and economic growth over the long term. Start-ups also play a key role in bringing about innovations, which often change the game, open new markets, and disrupt the status quo (see Fritsch, 2008; Neumark et al. , 2011; Hyatt and Spletzer, 2013; Criscuolo et al. , 2014; Decker et al. , 2014; Foster et al. , 2016; Slaper et al. , 2016; OECD , 2017; WIPO, 2021). This indicator can be measured as the number of newly registered establishments per 1,000 working-age people or the average of the number of new establishments less than 1 year old divided by the total number of active establishments.

Establishment churn: Technology and knowledge requirements that have changed, or even been eliminated, offer the opportunity to create new industries, processes, and jobs. Labour turnover is an indicator of the improved employment tenure of labour force workers. Workers are moving into more desirable and better-paying jobs. Similarly, labour turnover, whether measured by the creation of new businesses or the increase in the workforce of existing businesses, is an indicator of positive economic change in the region (see Criscuolo et al. , 2014; Decker et al. , 2014; Hathaway and Litan, 2014; Cukier et al. , 2016; OECD , 2017; Slaper et al. , 2016). Establishment turnover can be measured as the ratio of establishments that increase employment (with or without adding entries) to firms that reduce employment (with or without adding exits).

Job growth to population growth ratio: Employment growth is often used to measure economic growth and as an output of innovation. High employment growth relative to population growth indicates that jobs are being created faster than people are moving into an area. The ratio measures whether employment is growing faster or slower than the general population, indicating good innovation activity, as a firm’s performance is strongly correlated with its ability to innovate (see Dowrick and Nguyen, 1989; Acs and Armington, 2004; Boarnet, 2005; Cukier et al. , 2016; Slaper et al. , 2016; OECD , 2017; Beaudry and Solar-Pelletier, 2020).

3.8 Production

Exports in high-tech industries: Exports in high-tech industries measure technological competitiveness as the ability to commercialize the results of R&D and innovation in international markets. High-tech products are key drivers of economic growth, productivity, and welfare, and they are typically a source of high-value-added, well-paying jobs (see Porter, 1990, 1998; Wolfe and Gertler, 2004; Arthurs et al. , 2009; Cannon et al. , 2015; Slaper et al. , 2016; OECD , 2017; Beaudry and Solar-Pelletier, 2020; Hollanders and Es-Sadki, 2021; WIPO, 2021). Typically, these are companies in management, scientific and technical consulting services; scientific R&D services; chemicals; basic pharmaceuticals; weapons and ammunition; computer, electronic and optical products; electrical equipment, machinery and equipment; motor vehicles; trailers and semi-trailers; other transportation equipment, excluding shipbuilding, air- and spacecraft and related machinery; or medical and dental instruments. High-tech exports can be expressed as a percentage of total trade.

GDP growth: GDP can be a measure of economic performance, because it includes labour compensation and returns to capital. Innovative products or processes are undertaken because they would increase wages or profits (see Slaper et al. , 2011; Cukier et al. , 2016; Slaper et al. , 2016; Beaudry and Solar-Pelletier, 2020; Hollanders and Es-Sadki, 2021; WIPO, 2021). Since not all companies that contribute to GDP growth are innovative or active in innovation, GDP growth of high-tech companies can be considered instead. However, GDP may give a misleading reading for Canada in recent years, given population growth. GDP per hour worked ( i.e. , labour productivity) or other measures of productivity ( e.g. , multifactor or total factor productivity growth) could therefore be considered instead.

Proprietor income to wages and salaries: This measure examines the success of entrepreneurial activity by comparing owner income to total employee wages and salaries. A high ratio suggests the presence of profitable entrepreneurial activity, which may also indicate a more dynamic and innovative economy. The data on wage earnings and owner income are based on the location of the work. This allows for a stronger relationship between innovation activities and innovation rewards based on where the innovation activities took place (see Low et al. , 2005; Wong et al. , 2005; Hessels et al. , 2008; Koellinger, 2008; Goetz and Rupasingha, 2009; Minniti and Lévesque, 2010; Slaper et al. , 2016; OECD , 2017). Proprietor income to wages and salaries can be measured as the ratio of owner income to total wages and salaries.

4.1 Methodology of the innovation framework

According to the OECD ’s Handbook on Constructing Composite Indicators: Methodology and User Guide , the two main criteria for assessing composite indicators are ease of interpretation and the transparency of the methodology used. The selected indicators meet these criteria because they are widely used in the literature and at the international level, as well as on the methodological basis of simple popularity criteria. The common framework for analyzing the performance of innovation ecosystems is based on the use of factual data. Only 5 of 38 indicators (doing business, logistics performance and the three collaboration indicators) are based on indexes built on the perceptions of businesses or civil society. The framework is composed of 26 innovation input indicators and 12 innovation output indicators. It allows innovation ecosystems to be analyzed based on a variety of indicators, measuring different dimensions of innovation.

The literature review refers to indicators used to measure the performance of innovation ecosystems. These indicators can be used to construct a composite index to compare the evolution of a city’s innovation ecosystem over time or to compare two cities. A composite index can be created from all the indicators by using a simple or weighted average (based on their medium or high ranking) of the indicator scores.

Following the methodology of the European Innovation Scoreboard (see Hollanders and Es-Sadki, 2021) is proposed. It allows a simple and efficient construction of innovation indexes comparable in time and space. This methodology uses a scoring system to build a composite index. For each indicator, a base year is identified based on the availability of data from the observed sample.

Missing data are then treated by replacing, for example, missing values with those from the previous or next available year. Then, outliers are identified and replaced by the respective maximum and minimum values observed over the entire period of the observed sample. This methodology also transforms the data if they are highly skewed. Some indicators will be expressed as percentages, while others will have values that are not limited to an upper threshold. If these values are very volatile and asymmetric, they can be replaced by the square root of the indicator value. At this stage, it is possible to identify, for each indicator, the maximum score (the highest value of the sample over the whole observed period) and the minimum score (the lowest value of the sample over the whole observed period).

The values of the indicators are finally rescaled for each period to have them from 0 to 100. To do this, a formula is applied: (value−minimum(maximum)) × 100. Finally, for each observed period, a composite synthetic innovation index can be computed as the unweighted average of the rescaled scores for all indicators, where all indicators receive the same weighting (1/38 if data are available for all 38 indicators of the common framework). The European methodology proposes expressing performance scores relative to those of the European Union, which would be equivalent to expressing, for example, the composite index of Canadian cities relative to that of Canada.

4.2 Ecosystem performance analysis

Table 1 presents the indicators for two regions, A and B. The indicator values were randomly generated between the minimum and maximum values observed for each indicator at the provincial level in Canada from 2016 to 2018. They were generated for 50 regions to construct a good sample of regions. Only two regions are presented for illustrative purposes. The scores generated are at the indicator group and subgroup level. The value column gives the observed value of the indicator. The score column gives the score generated following the methodology presented above.

In the example presented, region B (58) has better overall innovation performance than region A (49). The framework allows the performance of these two ecosystems to be analyzed in more detail. Region A performs better in terms of innovation inputs than region B (54 versus 47), mainly because of better investment in the financial environment, as well as in the human and research capital of the ecosystem. By contrast, region B performs significantly better in all areas of innovation inputs than region A (68 versus 44), translating into better overall performance in terms of innovation. Another feature that this framework highlights is that region A’s better performance in terms of investment in innovation inputs does not translate well in terms of innovation outputs. This represents the major challenge for Canada in transforming its innovation investments into knowledge products. Given that these values were generated randomly, no further interpretive analysis can be done, but the common framework would allow researchers to assess, for example, which innovation input investment profile best translates into good innovation creation or existing innovation development.

The framework allows for an analysis of important categorical indexes for the inputs and outputs of innovation. This framework does not indicate how the volume of inputs affects the quantity of outputs. The OECD recommends using indicators cautiously, with the objective of providing complementary information to already established analyses on innovation ecosystems. Thus, this framework should be combined with empirical and theoretical analysis, including informed judgments and common sense.

However, the selected indicators allow regions to be ranked according to their innovation input and output performance. Spatial and temporal analysis can be performed on specific or combined dimensions of ecosystem performance. Through the common framework, readers, and users—especially in the public domain—will be able to see, very quickly, the dimensions measured in innovation ecosystems. As a source of information, therefore, this framework can assist in the development of various policies. It can be useful for quantifying and defining numerical targets and benchmarks. For example, a comparative analysis between regions with numerical indicators that are easy to understand can be used to motivate behaviour change, because one can compare oneself to others. The United Nations Development Programme’s Human Development Index classifications have inspired development practitioners to approach economic development in a broader dimension, involving composite indicators.

The analytical framework can also help develop common goals in public debate. Indexes and associated rankings are useful tools for focusing public attention on a particular set of policy issues. When supported by detailed data, they can provide valuable information about underlying strengths and weaknesses, which can then serve as a catalyst for further policy debate and efforts to improve specific areas of expertise.

In conclusion, this study covers the literature on performance indicators for innovation ecosystems at different scales of analysis. The selected indicators are easily computable, provided that the data are available. They are also regionally and internationally comparable, thanks to the methodology used to select them.

More than 400 indicators have been explored and combined into just over 100, classified according to their occurrence in the literature. The study provides a useful framework for assessment, which will be made more effective by a greater emphasis on improving the weakly developed dimensions of innovation. It is an interesting and comprehensive tool for policy makers, researchers, the private sector, and innovation actors. However, although this ranking lists the indicators most widely used in the literature, it does not include lesser-used indicators that reflect new trends and dynamics in local economies. To visualize low-use indicators and adapt them to the context of their analysis, readers can request access to the appendix containing all the indicators in the literature examined by this study.

Common innovation ecosystem indicators (recent studies)

• University-based knowledge spillovers
• Broadband connections
• High-tech industry employment
• Proprietorship rate

Common innovation ecosystem indicators (Microeconomics studies)

• Industry concentration Note †

Acs, Z. and Armington, C. (2004). Employment growth and entrepreneurial activity in cities. Regional Studies , 38(8):911–927.

Acs, Z., Audretsch, D., and Lehmann, E. E. (2013). The knowledge spillover theory of entrepreneurship. Small Business Economics , 41:757–774.

Acs, Z. J. and Audretsch, D. B. (1988). Innovation in large and small firms: An empirical analysis. The American Economic Review , 78(4):678–690.

Acs, Z. and Audretsch, D. B. (1990). Innovation and Small Firms . MIT Press, Cambridge, MA.

Acs, Z. J. and Plummer, L. A. (2005). Penetrating the "knowledge filter" in regional economies. The Annals of Regional Science , 39:439–456.

Adner, R. (2006). Match Your Innovation Strategy To Your Innovation Ecosystem. Harvard Business Review , 84:98–107; 148.

Aerts, K., Matthyssens, P., and Vandenbempt, K. (2007). Critical role and screening practices of European business incubators. Technovation , 27(5):254–267.

Agrawal, A. and Henderson, R. (2002). Putting patents in context: Exploring knowledge transfer from MIT. Management Science , 48(1):44–60.

Almeida, P. and Kogut, B. (1997). The exploration of technological diversity and the geographic localization of innovation: Start-up firms in the semiconductor industry Small Business Economics , 9(1):21–31.

Anselin, L., Varga, A., and Acs, Z. (1997). Local geographic spillovers between university research and high technology innovations. Journal of Urban Economics , 42(3):422–448.

Araujo Reis, D. Rodrigues de Moura, F., and Machado de Aragão, I. (2021). The linkage between input and output in the innovation ecosystem. Global Journal of Human-Social Science Research .

Arthurs, D., Cassidy, E., Davis, C., and Wolfe, D. (2009). Indicators to support innovation cluster policy. International Journal of Technology Management , 464:263–279.

Arvanitis, S. and Stucki, T. (2012). What determines the innovation capability of firm founders? Industrial and Corporate Change , 21(4):1049–1084.

Atkinson, R. D. and Mayo, M. J. (2010). Refueling the U.S. innovation economy: Fresh approaches to science, technology, engineering and mathematics (STEM) education.

Audretsch, D. and Keilbach, M. (2004). Entrepreneurship capital and economic performance. Regional Studies , 38(8):949–959.

Audretsch, D. and R. Thurik (2001), "Linking Entrepreneurship to Growth" , OECD Science, Technology and Industry Working Papers , No. 2001/02, OECD Publishing, Paris, https://doi.org/10.1787/736170038056.

Audretsch, D. B. and Feldman, M. P. (1996). Innovative clusters and the industry life cycle. Review of Industrial Organization , 11(2):253–273.

Audretsch, D. B. and Lehmann, E. E. (2005). Does the knowledge spillover theory of entrepreneurship hold for regions? Research Policy , 34(8):1191–1202. Regionalization of Innovation Policy.

Ayyagari, M., Demirgüç-Kunt, A., and Maksimovic, V. (2011). Firm innovation in emerging markets: The role of finance, governance, and competition. Journal of Financial and Quantitative Analysis , 46(6):1545–1580.

Barker, V. III and Mueller, G. C. (2002). CEO characteristics and firm R&D spending. Management Science , 48(6):782–801.

Beaudry, C. and Breschi, S. (2003). Are firms in clusters really more innovative? Economics of Innovation and New Technology , 12(4):325–342.

Beaudry, C., and Solar-Pelletier, L. (2020). The Superclusters Initiative: An Opportunity to Reinforce Innovation Ecosystems. IRPP Study 79 . Montreal: Institute for research on Public Policy .

Bednarzik, R. W. (2000).  The role of entrepreneurship in U.S. and European job growth. Monthly Labor Review , 123.

Belussi, F. and Sedita, S. R. (2009). Life cycle vs. multiple path dependency in industrial districts. European Planning Studies , 17(4):505–528.

Benfratello, L., Schiantarelli, F., and Sembenelli, A. (2008). Banks and innovation: Micro-econometric evidence on Italian firms. Journal of Financial Economics , 90(2):197–217.

Bercovitz, J. E. L. and Feldman, M. P. (2007). Fishing upstream: Firm innovation strategy and university research alliances. Research Policy , 36(7):930–948.

Bilbao-Osorio and Rodriguez-Pose, A. (2004). From R&D to innovation and economic growth in the EU. Growth and Change , 35(4):434–455.

Blalock, G. and Gertler, P. J. (2008). Welfare gains from foreign direct investment through technology transfer to local suppliers. Journal of International Economics , 74(2):402–421.

Boarnet, M. G. (2005). An empirical model of intrametropolitan population and employment growth. Papers in Regional Science , 73(2):135–152.

Bode, E. (2004). The spatial pattern of localized R&D spillovers: An empirical investigation for Germany. Journal of Economic Geography , 4(1):43–64.

Boschma, R. (2005). Proximity and innovation: A critical assessment. Regional Studies , 39(1):61–74.

Bottazzi, L. and Peri, G. (2002). Innovation and spillovers in regions: Evidence from European patent data. European Economic Review , 47(4):687–710.

Branstetter, L. (2006). Is foreign direct investment a channel of knowledge spillovers? Evidence from Japan’s FDI in the United States. Journal of International Economics , 68(2):325–344.

Breschi, S. and Lissoni, F. (2001). Knowledge spillovers and local innovation systems: A critical survey. Industrial and Corporate Change , 10(4):975–1005.

Busom, I. and Fernández-Ribas, A. (2008). The impact of firm participation in R&D programmes on R&D partnerships. Research Policy , 37(2):240–257.

Cannon, E., Forest, S., Chakma, A., Fortier, S., Garbutt, D., Gupta, A., Simair, M., Kang, C.-Y., Darkes, M., Knox, K., Laflamme, R., Romero, J., Scott, J., Treurnicht, I., Gracht, P., Venne-Man, N., Verschuren, A., and Secretariate, S. (2015). Science, Technology and Innovation Council to the Government of Canada. Technical report.

Carree, M. A. and Thurik, A. R. (2003). Chapter - The impact of entrepreneurship on economic growth. In Acs, Z. J. and Audretsch, D. B., editors, Handbook of Entrepreneurship Research: An Interdisciplinary Survey and Introduction , pages 437–471. Springer, Boston, MA.

Casper, S. (2013). The spill-over theory reversed: The impact of regional economies on the commercialization of university science. Research Policy , 42(1):1313–1324.

Chan, K. F. and Lau, T. (2005). Assessing technology incubator programs in the science park: the good, the bad and the ugly. Technovation , 25(10):1215–1228.

Cheshire, P. and Magrini, S. (2000). Endogenous processes in European regional growth: Convergence and policy. Growth and Change , 31(4):455–479.

Cohen, W. M., Nelson, R. R., and Walsh, J. P. (2002). Links and impacts: The influence of public research on industrial R&D. Management Science , 48(1):1–23.

Combes, P.-P. and Gobillon, L. (2015). Chapter 5 - The empirics of agglomeration economies. In Duranton, G., Henderson, J. V., and Strange, W. C., editors, Handbook of Regional and Urban Economics , volume 5, pages 247–348. Elsevier.

Corrado, C., Hulten, C., and Sichel, D. (2009). Intangible Capital and U.S. Economic Growth. Review of Income and Wealth , 55(3):661–685.

Crandall, R., Litan, R., and Lehr, W. (2007). The effects of broadband deployment on output and employment: A cross-sectional analysis of U.S. data. Issues in Economic Policy , 6.

Crescenzi, R. (2005). Innovation and regional growth in the enlarged Europe: The role of local innovative capabilities, peripherality, and education. Growth and Change , 36(1):471–507.

Criscuolo, C., P. Gal and C. Menon (2014), "The Dynamics of Employment Growth: New Evidence from 18 Countries", OECD Science, Technology and Industry Policy Papers , No. 14, OECD Publishing, Paris.

Cukier, W., Stolarik, K., Ngwenyama, O., N.,.and Elmi, M. (2016). Mapping the Innovation Ecosystem in Eastern Ontario. Technical report, Institute for Innovation and Technology Management.

Decker, R., Haltiwanger, J., Jarmin, R., and Miranda, J. (2014). The role of entrepreneurship in US job creation and economic dynamism. Journal of Economic Perspectives , 28(3):3–24.

Delgado, M. and Porter, M. E. (2017). Clusters and the Great Recession. In DRUID Conference Paper . Technical Report.

Delgado, M., Porter, M. E., and Stern, S. (2010). Clusters and entrepreneurship. Journal of Economic Geography , 10(4):495–518.

Delgado, M., Porter, M. E., and Stern, S. (2014). Clusters, convergence, and economic performance. Research Policy , 43(10):1785–1799.

Döring, T. and Schnellenbach, J. (2006). What do we know about geographical knowledge spillovers and regional growth?: A survey of the literature. Regional Studies , 40(3):375–395.

Dowrick, S. and Nguyen, D.-T. (1989). OECD comparative economic growth 1950-85: Catch-up and convergence. The American Economic Review , 79(5):1010–1030.

Drucker, J. and Goldstein, H. (2007). Assessing the regional economic development im- pacts of universities: A review of current approaches. International Regional Science Review , 30(1):20–46.

Duranton G, Puga D. 2004. Micro-Foundations of Urban Agglomeration Economies. In Handbook of Regional and Urban Economics , ed. J. V. Henderson and J.-F. Thisse, Amsterdam: Elsevier. Vol. 4: 2063-2117.

Engel, D. and Keilbach, M. (2007). Firm-level implications of early stage venture capital investment ― An empirical investigation. Journal of Empirical Finance , 14(2):150–167.

Eom, B.-Y. and Lee, K. (2010). Determinants of industry–academy linkages and, their impact on firm performance: The case of Korea as a latecomer in knowledge industrialization. Research Policy , 39(5):625–639. Special Section on Government as Entrepreneur.

Etzkowitz, H. (2002). Incubation of incubators: Innovation as a triple helix of university- industry-government networks. Science and Public Policy , 29(2):115–128.

Etzkowitz, H. (2003). Innovation in innovation: The triple helix of university-industry- government relations. Social Science Information , 42(3):293–337.

Feldman, M., Francis, J., and Bercovitz, J. (2005). Creating a cluster while building a firm: Entrepreneurs and the formation of industrial clusters. Regional Studies , 39(1):129–141.

Feller, I., Ailes, C. P., and Roessner, J. D. (2002). Impacts of research universities on technological innovation in industry: evidence from engineering research centers. Research Policy , 31(3):457–474.

Feser, E. J. (2003). What regions do rather than make: A proposed set of knowledge-based occupation clusters. Urban Studies , 40(10):1937–1958.

Fleming, L., King, C. III, and Juda, A. (2006). Small worlds and regional innovation. Organization Science , 18.

Florida, R. (2003). Cities and the creative class. City & Community , 2(1):3–19.

Foster, L., Grim, C., and Haltiwanger, J. (2016). Reallocation in the great recession: Cleansing or not? Journal of Labor Economics , 34(S1):S293–S331.

Fritsch, M. (2008). How does new business formation affect regional development? Introduction to the special issue. Small Business Economics , 30:1–14.

Glaeser, E. L. and Kerr, W. R. (2009). Local industrial conditions and entrepreneurship: How much of the spatial distribution can we explain? Journal of Economics & Management Strategy , 18(1):623–663.

Goetz, S. J. and Rupasingha, A. (2009) . Determinants of growth in non-farm proprietor densities in the US, 1990-2000. Small Business Economics , 32(4):425–438.

Gomes, L. A. d. V., Facin, A. L. F., Salerno, M. S., and Ikenami, R. K. (2018). Unpacking the innovation ecosystem construct: Evolution, gaps and trends. Technological Forecasting and Social Change , 136:30–48.

Gordon, I. R. and McCann, P. (2005). Innovation, agglomeration, and regional development. Journal of Economic Geography , 5(5):523–543.

Granstrand, O. and Holgersson, M. (2020). Innovation ecosystems: A conceptual review and a new definition. Technovation , 90-91:102098.

Grimaldi, R. and Grandi, A. (2005). Business incubators and new venture creation: An assessment of incubating models. Technovation , 25(2):111–121.

Grundke, R., Jamet, S., Kalamova, M., Keslair, F., and Squicciarini, M. (2017). Skills and global value chains: A characterization. OECD Science, Technology and Industry Working Papers , No. 2017/05, OECD Publishing, Paris.

Gulbrandsen, M. and Smeby, J.-C. (2005). Industry funding and university professors’ research performance. Research Policy , 34(6):932–950.

Hadjimanolis, A. (2000). A resource-based view of innovativeness in small firms. Technology Analysis & Strategic Management , 12(2):263–281.

Hall, B. H., Jaffe, A. B., and Trajtenberg, M. (2001). The NBER patent citations data file: Lessons, insights and methodological tools. NBER Working Paper 8498 .

Hansson, F., Husted, K., and Vestergaard, J. (2005). Second generation science parks: From structural holes jockeys to social capital catalysts of the knowledge society. Technovation , 25(9):1039–1049.

Hart, D. M. and Acs, Z. J. (2011). High-tech immigrant entrepreneurship in the United States. Economic Development Quarterly , 25(2):116–129.

Hathaway, I. and Litan, R. E. (2014). Declining business dynamism in the United States: A look at states and metros. Economic Studies at Brookings .

Hecker, D. E. (2005). High-technology employment: A NAICS-based update. Monthly Labor Review / U.S. Department of Labor, Bureau of Labor Statistics , 128:57–72.

Hessels, J., van Gelderen, M., and Thurik, R. (2008). Entrepreneurial aspirations, motivations, and their drivers. Small Business Economics , 31:323–339.

Heunks, F. J. (1998). Innovation, creativity and success. Small Business Economics , 10(3):263–272.

Hicks, D. and Hegde, D. (2005). Highly innovative small firms in the markets for technology. Research Policy , 34(5):703–716.

Hirukawa, M. and Ueda, M. (2011). Venture capital and innovation: Which is first? Pacific Economic Review , 16(4):421–465.

Hollanders, H. and Es-Sadki, N. (2021). European Innovation Scoreboard 2021. Technical report, European Union.

Hyatt, H. R. and Spletzer, J. R. (2013). The recent decline in employment dynamics. IZA Journal of Labor Economics , 2(5).

Innovation, Science and Economic Development Canada. (2024). Global Innovation Clusters. https://ised-isde.canada.ca/site/global-innovation-clusters/en.

Jaffe, A. (1989). The real effects of academic research. American Economic Review , 79(5):957–970.

Jaffe, A. B., Trajtenberg, M., and Henderson, R. (1993). Geographic localization of knowledge spillovers as evidenced by patent citations. The Quarterly Journal of Economics , 108(3):577–598.

Keller, W. (2002). Geographic localization of international technology diffusion. American Economic Review , 92(1):120–142.

Kerr, W. and Nanda, R. (2009). Financing constraints and entrepreneurship. Handbook of Research on Innovation and Entrepreneurship .

Kerr, W. R. (2013). U.S. high-skilled immigration, innovation, and entrepreneurship: Empirical approaches and evidence. Working Paper 19377, National Bureau of Economic Research.

Kirchhoff, B. A., Newbert, S. L., Hasan, I., and Armington, C. (2007). The influence of university R&D expenditures on new business formations and employment growth. Entrepreneurship Theory and Practice , 31(4):543–559.

Klepper, S. (1996). Entry, exit, growth, and innovation over the product life cycle. The American Economic Review , 86(3):562–583.

Koellinger, P. (2008). Why are some entrepreneurs more innovative than others? Small Business Economics , 31(1):21–37.

Kolko, J. (2000). The high-tech rural renaissance? Information technology, firm size and rural employment growth. Small Business Research Summary , 201.

Koo, J. (2005). Knowledge-based industry clusters: Evidenced by geographical patterns of patents in manufacturing. Urban Studies , 42(9):1487–1505.

Kortum, S. and Lerner, J. (2000). Assessing the contribution of venture capital to innovation. The RAND Journal of Economics , 31(4):674–692.

Langdon, D., McKittrick, G., Beede, D., Khan, B., and Doms, M. (2013). STEM: Good jobs now and for the future . U.S. Department of Commerce, Washington, DC.

Lehr, W., Osorio, C., Gillett, S. E., and Sirbu, M. A. (2006). Measuring broadband’s economic impact. (ESD-WP-2006-02).

Löfsten, H. and Lindelöf, P. (2005). R&D networks and product innovation patterns― academic and non-academic new technology-based firms on Science Parks. Technovation , 25(9):1025–1037.

López-Claros, A. and Mata, Y. (2010). Policies and Institutions Underpinning Country Innovation: Results from the Innovation Capacity Index. pages 3–63.

Low, S., Henderson, J., and Weiler, S. (2005). Gauging a region’s entrepreneurial potential. Economic Review , 90:61–89.

Mann, R. J. and Sager, T. W. (2007). Patents, venture capital, and software start-ups. Research Policy , 36(2):193–208.

Markman, G. D., Phan, P. H., Balkin, D. B., and Gianiodis, P. T. (2005). Entrepreneurship and university-based technology transfer. Journal of Business Venturing , 20(2):241–263. Special issue on Science Parks and incubators.

Martin, P., Mayer, T., and Mayneris, F. (2011). Public support to clusters: A firm level study of French “local productive systems.” Regional Science and Urban Economics , 41(2):108–123.

Mian, S. A. (1996). Assessing value-added contributions of university technology business incubators to tenant firms. Research Policy , 25(3):325–335.

Minniti, M. and Lévesque, M. (2010). Entrepreneurial types and economic growth. Journal of Business Venturing , 25(3):305–314.

Neffke, F., Henning, M., Boschma, R., Lundquist, K.-J., and Olander, L.-O. (2011). The dynamics of agglomeration externalities along the life cycle of industries. Regional Studies , 45(1):49–65.

Neumark, D., Wall, B., and Zhang, J. (2011). Do small businesses create more jobs? New evidence for the United States from the National Establishment Time Series. The Review of Economics and Statistics , 93(1):16–29.

Niosi, J. and Bas, T. G. (2001). The competencies of regions ― Canada’s clusters in biotechnology. Small Business Economics , 17:31–42.

OECD (2017). OECD Science, Technology and Industry Scoreboard 2017: The digital transformation . OECD Science, Technology and Industry Scoreboard. OECD Publishing, Paris.

OECD and Eurostat (2018). Oslo Manual 2018 .

Porter, M. E. (1990). The Competitive Advantage of Nations . The Free Press, New York.

Porter, M. E. (1998). Clusters and Competition: New Agendas for Companies, Governments, and Institutions . Clusters and Competition. Harvard Business Review Books, Cambridge, MA.

Porter, M. E. (2000). Location, competition, and economic development: Local clusters in a global economy. Economic Development Quarterly , 14(1):15–34.

Rin, M. D., Hellmann, T., and Puri, M. (2013). Chapter 8 - A survey of venture capital research. volume 2 of Handbook of the Economics of Finance , pages 573–648. Elsevier.

Rodríguez-Pose, A. and Crescenzi, R. (2008). Research and development, spillovers, innovation systems, and the genesis of regional growth in Europe. Regional Studies , 42(1):51–67.

Romero, I. and Martínez-Román, J. A. (2012). Self-employment and innovation. Exploring the determinants of innovative behavior in small businesses. Research Policy , 41(1):178–189.

Romijn, H. and Albaladejo, M. (2002). Determinants of innovation capability in small electronics and software firms in southeast England. Research Policy , 31(7):1053–1067.

Salter, A. J. and Martin, B. R. (2001). The economic benefits of publicly funded basic research: A critical review. Research Policy , 30(3):509–532.

Santoro, M. D. and Gopalakrishnan, S. (2001). Relationship dynamics between university research centers and industrial firms: Their impact on technology transfer activities. The Journal of Technology Transfer , 26(1-2):163–171.

Sen, A. (1999). Development as Freedom . Alfred Knopf, New York.

Simonen, J. and McCann, P. (2008). Firm innovation: The influence of R&D cooperation and the geography of human capital inputs. Journal of Urban Economics , 64(1):146–154.

Singh, J. (2005). Collaborative networks as determinants of knowledge diffusion patterns. Management Science , 51(5):756–770.

Sjöholm, F. (1999). Technology gap, competition and spillovers from direct foreign investment: Evidence from establishment data. The Journal of Development Studies , 36(1):53–73.

Slaper, T. F., Hart, N. R., Hall, T. J., and Thompson, M. F. (2011). The index of innovation: A new tool for regional analysis. Economic Development Quarterly , 25(1):36–53.

Slaper, T., van der Does, T., Egan, P., Ortuzar, G., and Strange, R. (2016). Driving Regional Innovation: The Innovation Index 2.0. Technical report, U.S. Economic Development Administration.

Soetanto, D. P, and Jack, S. L. (2011). Business incubators and the networks of technology- based firms. The Journal of Technology Transfer , 38.

Sorenson, O. and Fleming, L. (2004). Science and the diffusion of knowledge. Research Policy , 33(10):1615–1634.

Sorenson, O., Rivkin, J. W., and Fleming, L. (2006). Complexity, networks and knowledge flow. Research Policy , 35(7):994–1017.

Statistics Canada. (2021). Science, Technology and Innovation Data Gaps Initiative. What we heard - key takeaways. Unpublished report.

Stuart, T. and Sorenson, O. (2003). The geography of opportunity: Spatial heterogeneity in founding rates and the performance of biotechnology firms. Research Policy , 32(2):229–253. Special issue on technology entrepreneurship and contact information for corresponding authors.

Teece, D. J. (2007). Explicating dynamic capabilities: The nature and microfoundations of (sustainable) enterprise performance. Strategic Management Journal , 28(13):1319–1350.

Thurik, A. R., Carree, M. A., Andrévan Stel, and Audretsch, D. B. (2008). Does self-employment reduce unemployment? Journal of Business Venturing , 23(6):673–686. The Economics of Entrepreneurship.

Thurik, R. and Wennekers, S. (1999). Linking entrepreneurship and economic growth. Small Business Economics , 13:27–55.

Tödtling, F., Lehner, P., and Kaufmann, A. (2009). Do different types of innovation rely on specific kinds of knowledge interactions? Technovation , 29(1):59–71.

U.S. Department of Commerce, Economic Development Administration (2010). A Practitioner’s Guide to Economic Development Tools for Regional Competitiveness in a Knowledge-Based Economy.

Wadhwa, V., Saxenian, A., Rissing, B., and Gereffi, G. (2008). Skilled immigration and economic growth. Applied Research in Economic Development , 5(1):6–15.

Warda, J. (2001). Measuring the value of R&D tax treatment in OECD countries. STI Review , 27:185–211.

WIPO (2021). Global Innovation Index 2021: Tracking Innovation through the COVID-19 Crisis. World Intellectual Property Organization, Geneva.

Wolfe, D. A. and Gertler, M. S. (2004). Clusters from the inside and out: local dynamics and global linkages. Urban Studies , 41(5/6):1071–1093.

Wong, P. K., Ho, Y. P., and Autio, E. (2005). Entrepreneurship, innovation and economic growth: Evidence from gem data. Small Business Economics , 24(3):335–350.

Woodward, D., Figueiredo, O., and Guimarães, P. (2006). Beyond the Silicon Valley: University R&D and high-technology location. Journal of Urban Economics , 60(1):15–32.

More information

Note of appreciation.

Canada owes the success of its statistical system to a long-standing partnership between Statistics Canada, the citizens of Canada, its businesses, governments and other institutions. Accurate and timely statistical information could not be produced without their continued co-operation and goodwill.

Standards of service to the public

Statistics Canada is committed to serving its clients in a prompt, reliable and courteous manner. To this end, the Agency has developed standards of service which its employees observe in serving its clients.

Published by authority of the Minister responsible for Statistics Canada.

© His Majesty the King in Right of Canada as represented by the Minister of Industry, 2024

Use of this publication is governed by the Statistics Canada Open Licence Agreement .

Catalogue no. 11-633-X

Frequency: Occasional

A Systematic Literature Review on Flexible Strategies and Performance Indicators for Supply Chain Resilience

• REVIEW PAPERS
• Open access
• Published: 10 September 2024

Cite this article

You have full access to this open access article

• Ananna Paul 1 &
• Suvash C. Saha   ORCID: orcid.org/0000-0002-9962-8919 1  

Supply chain resilience is a widely useful concept for managing risk and disruption. Designing strategies for preparedness, response, and recovery can help businesses to mitigate risks and disruptions. Among them, flexible strategies can effectively improve supply chain resilience. In the literature, several studies have considered different types of flexible strategies and investigated their impacts on supply chain resilience. However, a systematic literature review (SLR) paper on this topic can further help to understand the scientific progress, research gaps, and avenues for future research. Hence, this study aims to explore how the literature has contributed to the area of flexible strategies and the impact on supply chain resilience performance. To achieve our objective, we apply an SLR methodology to identify themes such as research areas and key findings, contexts and industry sectors, methodologies, and key strategies and performance indicators in the connection between flexible strategies and supply chain resilience. The findings show that many studies connect flexible strategies to supply chain resilience. However, research gaps exist in analysing relationships between flexible strategies and performance, conducting comparative studies, developing dynamic resilience plans, applying flexible strategies, conducting theoretically grounded empirical studies, and applying multiple analytical tools to develop decision-making models for supply chain resilience. Finally, this study suggests several future research opportunities to advance the research on the topic. The findings can be a benchmark for researchers who are interested in conducting research in the area of flexible strategies and supply chain resilience.

Similar content being viewed by others

Flexibility in Enhancing Supply Chain Resilience: Developing a Resilience Capability Portfolio in the Event of Severe Disruption

Analysis of Theoretical Aspects of Supply Chain Resilience Determinants and Strategies

Supply Chain Resilience: A Literature Review and Gap Analysis

Explore related subjects.

• Artificial Intelligence

Avoid common mistakes on your manuscript.

Introduction

Supply chain management is critical in supplying, producing, and distributing goods and services to consumers and communities. However, any risks, disruptions, and uncertainties at any supply chain stage could make the whole operation vulnerable (Paul et al., 2017 ). The ultimate consequences could include delivery and supply delays, demand unfulfilment, and loss of revenue and business goodwill (Rahman et al., 2022 ). Hence, developing a resilient supply chain to absorb disruptions and keep operations going is important.

Supply chain resilience is defined by the preparedness and ability to respond to recover from and deal with disruptions (Ponis & Koronis, 2012 ; Ribeiro & Barbosa-Povoa, 2018 ; Tukamuhabwa et al., 2015 ). Preparedness means taking proactive actions, such as assessing risk and disruption factors and planning for strategies and resources (Paul & Chowdhury, 2020 ; Rahman et al., 2022 ). Meanwhile, response and recovery are reactive actions. Response includes the ability to quickly and accurately sense the impacts of a disruption and respond to mitigate such impacts (Scholten et al., 2020 ). For example, swiftly accessing alternative suppliers and emergency sources in case of a supply disruption can help mitigate the consequences. Recovery includes the planning and replanning for a future period after the occurrence of a disruption to bring the plan to the normal stage (Paul et al., 2017 ). For example, utilising alternative suppliers and resources to revise the supply chain plan for a certain period after the occurrence of supply disruption mitigates the impacts and helps restore the original plan. Recovery requires a sophisticated plan that utilises appropriate mitigation strategies. Preparedness, response, and recovery are well connected, as response and recovery can be difficult without good preparedness.

The flexible supply chain is a popular concept for managing variability in supply chains (Dhillon et al., 2023 ; Varma et al., 2024 ; Wadhwa et al., 2008 ). Variability includes changes in demand, processing time, lead time, and so on. Supply chain flexible strategies include flexibility in design, supply, manufacturing, transportation, and logistics. It also connects the flexibility of supply chain partners, such as flexible suppliers, manufacturing plants, logistics, and transportation.

Supply chain variabilities are well connected to risks and uncertainties. Flexible strategies can help manage supply chain uncertainties, risks, and variabilities (Tang & Tomlin, 2008 ; Yi et al., 2011 ). For example, utilising multiple suppliers and safety inventory can be useful to mitigate supply risks and uncertainties. The literature shows that flexible strategies effectively build resilient supply chains and can help manage risk and uncertainty and improve supply chain resilience by preparing well and/or enhancing capabilities to respond and recover (Chowdhury et al., 2024 ; Chunsheng et al., 2020 ; Dwivedi et al., 2023 ; Kamalahmadi et al., 2022 ; Kazancoglu et al., 2022 ; Mackay et al., 2020 ; Piprani et al., 2022 ; Rajesh, 2021 ; Sharma et al., 2023 ; Tang & Tomlin, 2008 ).

In the literature, several studies explore the usefulness of flexible strategies to improve supply chain resilience. Moreover, a few review papers exist in the literature which analysed supply chain resilience with drivers, vulnerabilities, risks and impacts, and robustness (Shishodia et al., 2023 ), supply chain resilience strategies (Rahman et al., 2022 ), framework, barriers, and strategies for supply chain resilience (Shashi et al., 2020 ), and recovery ability for supply chain resilience (Mandal, 2014 ). However, a systematic literature review (SLR) and content analysis of previously published papers on flexible strategies and supply chain resilience are non-existent. An SLR and content analysis are very helpful for researchers to understand the progress and development and plan for future research. Accordingly, this review article develops the following research questions (RQs).

RQ1: What contributions have been made in the connection between flexible strategies and supply chain resilience?

RQ2: What are the emerging research opportunities in the area of flexible strategies and supply chain resilience?

To answer the above RQs, this paper investigates flexible strategies and performance indicators for supply chain resilience by conducting an SLR and analysing articles under different themes, such as research area and key findings, context and industry sectors, methodologies, key dimensions, strategies, and performance indicators. Finally, this study also analyses the research gaps and suggests a number of meaningful future research opportunities.

The rest of the paper is organised as follows. Section “ Review Methodologies ” describes the review methodologies. Section “ Analysing Reviewed Articles ” analyses previous articles on flexible strategies for supply chain resilience. Research gaps and future research directions are provided in Sect. “ Research gaps and Future Research Opportunities ”. Finally, Sect. “ Conclusions ” provides conclusions and limitations of the study.

Review Methodologies

In this paper, an SLR process is utilised to analyse the content of the reviewed articles (Tranfield et al., 2003 ). An SLR provides a more accurate literature search and in-depth content analysis than other methods, such as generic and bibliometric reviews. It also helps in the systematic and critical analysis of the content of previously published articles.

In this paper, Scopus was the primary database to identify articles on flexible strategies and performance indicators for supply chain resilience. The following search criteria were used:

Keywords: flexible strategy, supply chain, resilience, performance.

Language: English.

Source type: Journal.

Search timeline: up to 2023.

The initial search using keywords identified a total of 138 articles. After filtering for language and source type, 46 articles were removed and 92 articles remained.

Next, we read the article’s title, abstract, and content and applied inclusion and exclusion criteria to finalise the articles. The inclusion criteria were: (i) articles focused on flexible strategies for different aspects of supply chain resilience, and (ii) both the keywords “flexible” or “flexibility” and “resilience” appeared in the main text. The exclusion criteria were if one or more keywords mentioned in the implications and/or in the reference list were available, but the article did not focus on the flexible strategies in supply chain resilience. After applying inclusion and exclusion criteria, 30 articles were removed and 62 articles remained.

Finally, other databases, such as Google Scholar and Web of Science, were used to search the articles. The reference check was also conducted to ensure that all relevant articles were included in the analysis. These checks did not include any new articles. A total of 62 articles were finalised for the analysis in this review. The review methodology is presented in Fig.  1 .

Review methodology

Analysing Reviewed Articles

This section analyses the finalised articles in key different dimensions, including subject areas, key contributions and findings, contexts of the studies, methodologies used, key sectors (manufacturing or service), different flexible strategies for supply chain resilience, and performance indicators for supply chain resilience.

Key Subject Areas

We analysed the subject areas for the 62 articles. As flexibility and supply chain resilience is a multidisciplinary research area, the articles were expected to contribute to several subject areas. Thus, we observed the common subject areas to be business, management and accounting, engineering, decision sciences, computer science, and social sciences. The key subject areas for the reviewed articles are presented in Fig.  2 .

Key subject areas of the reviewed articles

Key Contributions and Findings of Previous Studies

Over the last few years, many studies have contributed in the area of flexible strategies and supply chain resilience. We observed that eight articles used a literature review approach, while the remaining 54 were technical studies. This section delves into the details of previous contributions and findings.

Previously Published Review Articles

From the systematic review, we identified eight review articles in the area of supply chain resilience. The main contributions and findings of those review articles are summarised in Table  1 . The previous review articles analysed the literature in different supply chain resilience dimensions, including drivers, vulnerabilities, risks and impacts, and robustness (Shishodia et al., 2023 ), resilience strategies (Rahman et al., 2022 ), framework, barriers, and strategies (Shashi et al., 2020 ), and recovery (Mandal, 2014 ). Significant research gaps exist in reviewing the literature on how different flexible strategies are applied to improve supply chain resilience and the potential future research directions. This paper fills these gaps.

Table 1 shows that five articles used a systematic literature review approach, while others used bibliometric analysis and literature review along with expert opinions and conceptual modelling/framework.

Contributions and Findings of Technical Studies

We analysed the contributions and main findings of 54 technical studies and observed the following main areas of study.

Analysing resilience strategies using varieties of methodologies (Kummer et al., 2022 ; Nagariya et al., 2023 ; Purvis et al., 2016 ; Wang et al., 2016 ),

Analysing impacts of strategies on performance (Alvarenga et al., 2023 ; Hamidu et al., 2024 ; Isti’anah et al., 2021 ; Lin et al., 2023 ; Nguyen et al., 2022 ; Xu et al., 2023 ),

Exploring capabilities for supply chain resilience (Faruquee et al., 2023 ; Shweta et al., 2023 ; Um & Han, 2021 ; Zhou et al., 2022 ),

Evaluating critical factors, enablers, and antecedents for supply chain resilience (Das et al., 2022 ; Pu et al., 2023a , 2023b ; Sangari & Dashtpeyma, 2019 ),

Analysing impacts of disruption on supply chains (Ivanov, 2022 ),

Designing/re-designing supply chain networks to improve resilience (Alikhani et al., 2021 ; Carvalho et al., 2012 ; Fattahi et al., 2020 ), and

Selecting suppliers for supply chain resilience (Suryadi & Rau, 2023 ).

The main contributions and findings are summarised in Table  2 .

This section analyses different contexts used in the literature. The contexts include both industry sectors and regions of data collection and applications. We observed that 38 studies used a specific industry context, while 41 papers used a country/regional context in their studies.

Industry Context

Our analysis of the articles shows that both single and multiple sectors have been considered in previous studies. Fourteen studies considered multiple industry sectors, and 24 studies considered a single industry sector. The single industry sectors include maritime (Isti’anah et al., 2021 ; Praharsi et al., 2021 ; Zavitsas et al., 2018 ), food (Li et al., 2022 ; Purvis et al., 2016 ), healthcare (Vimal 2022a ; Shweta et al., 2023 ), and textile and apparel sectors (Fahimnia et al., 2018 ; Nagariya et al., 2023 ). The other single industry sectors are container handling, delivery services, e-commerce of clothing and grocery, industrialised construction, copper industry, retail, ICT industry, automotive, sportswear, and electronic sectors.

Previous studies also considered multiple industry sectors. For example, Alvarenga et al. ( 2023 ) considered multiple sectors, including chemical and petroleum, food and beverage, and machinery sectors. Maharjan and Kato ( 2023 ) considered multiple sectors, including manufacturing, assembly, agricultural machinery parts, apparel business, and trading companies. Zhou et al. ( 2022 ) considered multiple sectors, including electronics and appliances, metals, machinery and engineering, construction materials, textiles, and clothing. Gölgeci and Kuivalainen ( 2020 ) considered multiple sectors, including chemical and pharmaceutical, food and beverage, construction equipment, retail, textile, clothing, and apparel.

Country/Regional Context

Forty-one studies considered a specific country/regional context. Several studies considered global or multiple regions. For example, Alvarenga et al. ( 2023 ) considered a global context, including North America, Europe, Asia, Africa, South America, and Oceania countries. Faruquee et al. ( 2023 ) collected data from the USA and the UK. Das et al. ( 2022 ) collected data from countries in Asia, Europe, and the Americas.

The majority of the studies considered a single country/regional context. Among them, seven studies considered India (Altay et al., 2018 ; Vimal et al., 2022a , 2022b ; Nagariya et al., 2023 ; Rajesh, 2016 ; Shweta et al., 2023 ; Suryawanshi et al., 2021 ), four studies considered Iran (Alikhani et al., 2021 ; Fattahi et al., 2020 ; Moosavi & Hosseini, 2021 ; Suryadi & Rau, 2023 ), three studies considered China (Pu et al., 2023a , 2023b ; Zhu & Wu, 2022 ) and three studies considered Ghana (Hamidu et al., 2023a , 2023b , 2024 ) in the country context.

The details of industry sectors and country/regional contexts are presented in Table  3 .

Methodologies Used

Both qualitative and quantitative methods have been applied to analyse strategies and performance indicators in supply chain resilience. Qualitative methods include literature reviews (see Table  1 ), interviews (Chen et al., 2019 ; Lin et al., 2023 ; Maharjan & Kato, 2023 ; Purvis et al., 2016 ; Silva et al., 2023 ), conceptual modelling (Mackay et al., 2020 ), DMAIC framework (Praharsi et al., 2021 ), and FEWSION for the community resilience process (Ryan et al., 2021 ).

Quantitative methods include structural equation modelling (Alvarenga et al., 2023 ; Gölgeci & Kuivalainen, 2020 ; Pu et al., 2023a , 2023b ; Purvis et al., 2016 ; Um & Han, 2021 ), mathematical programming (Alikhani et al., 2021 ; Mao et al., 2020 ; Mikhail et al., 2019 ; Suryawanshi et al., 2021 ; Zavitsas et al., 2018 ), MCDM methods (Das et al., 2022 ; Shweta et al., 2023 ), simulation (Ivanov, 2022 ; Kummer et al., 2022 ; Moosavi & Hosseini, 2021 ; Tan et al., 2020 ), partial least squares (Altay et al., 2018 ), and regression analysis (Donadoni et al., 2018 ; Trabucco & De Giovanni, 2021 ).

Table 4 provides a summary of the methods used.

Several studies integrated multiple methods such as PLS-SEM (Ekanayake et al., 2021 ; Hamidu et al., 2023a , 2023b ; Nguyen et al., 2022 ), Fuzzy DEMATEL and best–worst method (Shweta et al., 2023 ), analytic hierarchy process and linear programming (Suryadi & Rau, 2023 ), analysis of variance and polynomial regression (Faruquee et al., 2023 ), best–worst method and fuzzy TOPSIS (Vima et al., 2022b ), Delphi method and best–worst method (Nagariya et al., 2023 ), AHP and DEMATEL (Das et al., 2022 ), mixed-integer linear programming and Monte Carlo simulation (Suryawanshi et al., 2021 ), interpretive structural modelling and fuzzy analytical network process (Sangari & Dashtpeyma, 2019 ), and discrete-event simulation and regression analysis (Macdonald et al., 2018 ).

Case studies were combined with other methods in several studies. For example, Purvis et al. ( 2016 ) conducted a case study in the UK’s food and drink sector to analyse supply chain resilience strategies. Maharjan and Kato ( 2023 ) included a case study from Japan’s manufacturing, agricultural, apparel, and trading companies to identify the current resilience status. Lin et al. ( 2023 ) provided a case study from delivery services in the UK to investigate supply chain resilience in responding to disruptions. Silva et al. ( 2023 ) discussed the findings from coffee-producing firms in Brazil to explore the relationship between sustainability and resilience. Carvalho et al. ( 2012 ) explained a case study from the automotive sector in Portugal to analyse the scenario-based design for supply chain resilience.

Key Sectors (Manufacturing or Service)

The reviewed articles show that previous studies considered both the manufacturing and service sectors as the key application areas. Figure  3 provides a summary of key sectors. Figure  3 shows that 49 out of 62 articles considered a sector, with most (35 articles) focusing on the manufacturing sector. Nine studies considered both manufacturing and service sectors, and only five considered the service sector. Sect. “ Contexts ” shows the specific contexts previous studies considered.

Summary of key sectors

Different Flexible Strategies for Supply Chain Resilience

We observed that numerous strategies have been used for supply chain resilience. We have categorised them as supply, manufacturing/operational strategies, transportation and distribution strategies, and supply chain levels.

The most common supply strategies were multiple suppliers/sourcing, improving collaboration with suppliers/partners, backup/alternative suppliers, supplier development, and building trust with suppliers. These strategies help to improve supply chain flexibility and supply chain resilience. For example, multiple suppliers/sourcing includes having multiple suppliers or sources of materials for mitigating risks and disruptions (Ekanayake et al., 2021 ; Mikhail et al., 2019 ; Praharsi et al., 2021 ; Rahman et al., 2022 ). It improves supply flexibility, further allowing for the diversification of the supply base. Similarly, another popular strategy in supply chain resilience is improving collaboration with suppliers/partners. It enhances communication processes, information, and resource sharing and working together to deal with risks and uncertainties in their supply chains (Chen et al., 2019 ; Faruquee et al., 2023 ; Sangari & Dashtpeyma, 2019 ; Silva et al., 2023 ).

Flexible transportation/distribution channels were the most widely applied transportation and distribution strategy. This includes flexible routes, flexible transportation capacities, and multiple distribution channels, spanning online, and physical distributions (Faruquee et al., 2023 ; Hohenstein et al., 2015 ; Massari & Giannoccaro, 2021 ; Suryadi & Rau, 2023 ). This strategy is very effective in improving resilience in transportation and distribution, particularly, and the supply chain, in general. The other flexible strategies included alternative shipment/transportation modes and backup distribution centres.

Strategies such as utilising extra capacity, resource allocation/reallocation, managing the quality of products, and using safety stock were widely applied in manufacturing/operations. Extra capacities in manufacturing plants improve production flexibilities and help mitigate supply and demand uncertainties (Altay et al., 2018 ; Fattahi et al., 2020 ; Rahman et al., 2022 ). Other strategies, such as resource allocation/reallocation, managing the quality of products, and using safety stock, are also effective in dealing with risk and disruption in supply chains and improving business reputation.

In supply chain-level strategies, the common strategies were adopting digital technologies, knowledge/information sharing, business continuity/contingency planning, and multi-skilled labour. The recent studies highlighted that adopting digital technologies at the supply chain level could improve communication, tracking, data analysis, and information processing (Alvarenga et al., 2023 ; Nagariya et al., 2023 ; Nguyen et al., 2022 ; Trabucco & De Giovanni, 2021 ). All these contribute to improving supply chain performance and resilience. Similarly, the literature proved that supply chain-level strategies help improve operational, financial, and reputational performance by enhancing supply chain resilience.

The full list of flexible strategies for supply chain resilience and their categories are presented in Table  5 .

Performance Indicators for Supply Chain Resilience

Supply chain resilience studies have used several performance indicators to measure performance, including financial, operational, reputational, and supply chain performance.

In supply chain resilience, financial performance indicators include cost efficiency, return on investment, market share, sales growth, profit, and return on sales and assets. Cost efficiency is the most significant performance indicator (Alikhani et al., 2021 ; Donadoni et al., 2018 ; Fattahi et al., 2020 ; Nagariya et al., 2023 ). Organisations set their desired price while maintaining the quality of products or services and improving customer satisfaction. Another significant performance indicator is profit (Hohenstein et al., 2015 ; Mikhail et al., 2019 ; Moosavi & Hosseini, 2021 ; Shashi et al., 2020 ). Profit is a goal for organisations to enhance overall performance. Return on investment (Gölgeci & Kuivalainen, 2020 ; Juan & Li, 2023 ; Trabucco & De Giovanni, 2021 ) and market share (Hohenstein et al., 2015 ; Juan & Li, 2023 ; Pu et al., 2023a , 2023b ; Zhou et al., 2022 ) are also used to evaluate organisational performance.

The most common operational performance indicators in supply chain resilience are on-time delivery, demand fulfilment, and enhanced operational efficiency and delivery time. On-time delivery (Rajesh, 2021 ; Shweta et al., 2023 ; Trabucco & De Giovanni, 2021 ) improves the efficiency of business processes and fulfils customer commitment. Customer order processing depends on demand fulfilment. Demand fulfilment (Moosavi & Hosseini, 2021 ; Rajesh, 2021 ; Tan et al., 2020 ) positively impacts the firm’s performance in the competitive market. Enhanced operational efficiency (Praharsi et al., 2021 ) and delivery time (Mao et al., 2020 ) increases customer satisfaction and improves business performance.

In supply chain resilience, reputational performance indicators include customer satisfaction, service-level improvement, customer loyalty, meeting customer satisfaction/request, quality performance, and corporate image. Service-level improvement (Hohenstein et al., 2015 ; Isti’anah et al., 2021 ; Praharsi et al., 2021 ) is one of the most important performance indicators. Maximising service level increases the overall performance of organisations. Customer satisfaction is the second most crucial reputational performance indicator (Gölgeci & Kuivalainen, 2020 ; Zhu & Wu, 2022 ). Customer satisfaction with a product/service enhances organisational reputation.

Resilience performance also depends on supply chain performance indicators such as restoring material flow, quickly moving to a desirable state, lead time reduction, supply chain visibility, recovery time, and response time. Among these indicators, lead time reduction (Donadoni et al., 2018 ; Ivanov, 2022 ; Nagariya et al., 2023 ), recovery time (Altay et al., 2018 ; Singh & Singh, 2019 ), and response time (Altay et al., 2018 ; Faruquee et al., 2023 ) are the significant performance indicators. Lead time reduction minimises the time duration of the product or service process. Reduction of recovery time and response time enhances the efficiency of organisational performance.

Table 6 summarises the list of performance indicators in supply chain resilience.

Mapping of Strategies and Performance Indicators

The literature review shows that flexible strategies are useful in improving supply chain performance. This section explains the mapping between different flexible strategies and performance indications and discusses the strategies that effectively improve or influence performance.

From the literature analysis, we have observed that “improving collaboration with suppliers/partners” influences all major resilience performances, including cost efficiency, return on investment, market share, profit, customer satisfaction, service-level improvement, on-time delivery, demand fulfilment, lead time reduction, recovery time, and response time (Chen et al., 2019 ; Donadoni et al., 2018 ; Faruquee et al., 2023 ; Hohenstein et al., 2015 ; Juan & Li, 2023 ; Ladeira et al., 2021 ; Moosavi & Hosseini, 2021 ; Praharsi et al., 2021 ; Shashi et al., 2020 ; Shweta et al., 2023 ; Suryadi & Rau, 2023 ; Zhou et al., 2022 ; Zhu & Wu, 2022 ).

Similarly, multiple suppliers/sourcing, backup/alternative suppliers, flexible transportation/distribution channels, utilising extra capacity, adopting digital technologies, knowledge/information sharing, and multi-skilled labour are effective in improving resilience performance in supply chain management.

Table 7 provides the mapping between different strategies and their influence on resilience performance indicators.

Research Gaps and Future Research Opportunities

We have observed the following research gaps from the literature review and have suggested future research opportunities.

Relationship Between Strategies and Performance In Supply Chain Resilience

Very few studies analysed the relationship between strategies and performance in supply chain resilience. While a few studies did, they only considered a limited number of strategies and performance indicators (Donadoni et al., 2018 ; Faruquee et al., 2023 ; Gölgeci & Kuivalainen, 2020 ; Isti’anah et al., 2021 ; Juan & Li, 2023 ; Mikhail et al., 2019 ; Nagariya et al., 2023 ; Praharsi et al., 2021 ; Pu et al., 2023a , 2023b ; Shishodia et al., 2023 ; Suryadi & Rau, 2023 ; Trabucco & De Giovanni, 2021 ; Wang et al., 2016 ; Zhou et al., 2022 ). For example, Shishodia et al. ( 2023 ) considered managing product quality, multiple sourcing, demand aggregation, flexible transportation systems, backup suppliers, fortification of partners, and risk sharing as strategies and cost efficiency and lead time reduction as performance indicators. Similar analyses were found in other studies. This makes the literature less comprehensive in analysing the thorough impacts of different strategies, individually and combined, on supply chain resilience performance.

To close this gap and improve the literature, we propose studies to consider the holistic list of strategies and performance indicators (as shown in Sects. “ Different Flexible Strategies for Supply Chain Resilience ” and “ Performance Indicators for Supply Chain Resilience ”) and analyse how major strategies influence major performance indicators in supply chain resilience.

Comparative Studies

There is a significant research gap in the literature regarding comparative studies. Very few studies considered both the manufacturing and service sectors and multiple industry sectors (Alikhani et al., 2021 ; Alvarenga et al., 2023 ; Nguyen et al., 2022 ; Singh & Singh, 2019 ; Zhu & Wu, 2022 ). However, the literature has research gaps for comparative studies between developed and developing economies, large and small and medium enterprises, and their longitudinal analyses. Hence, there is a gap in generalising the findings.

To contribute to this area, we suggest conducting the following studies.

Comparative studies of flexible strategies and/or performance indicators for developed and developing economies.

Comparative studies of flexible strategies and/or performance indicators between large, small, and medium enterprises.

Analysis of findings over time for different economies and enterprises.

Developing models for generalising the findings for different economies and enterprises.

Service Sectors

Service sectors get less attention in the literature even though they are dominant in many countries. Only a few studies considered service sectors (Fattahi et al., 2020 ; Isti’anah et al., 2021 ; Lin et al., 2023 ; Suryawanshi et al., 2021 ). Hence, the literature provided few findings on supply chain resilience and their strategies and performance indicators in service sectors.

We suggest conducting more studies for service sectors, including the analysis of different flexible strategies used by different service sectors and how they influence service performance to improve supply chain resilience.

Dynamic Plans for Supply Chain Resilience

Many studies have developed models and frameworks for analysis strategies and performance indicators in supply chain resilience (Juan & Li, 2023 ; Shishodia et al., 2023 ; Suryadi & Rau, 2023 ). Still, there is a gap in the literature on developing dynamic resilience plans for the changed environment. As risks and disruptions change over time, it is important to change the plan and its flexible strategies to ensure supply chains can deal with the impacts of the changing environment and improve resilience. These types of studies on flexible strategies and supply chain resilience are non-existent in the current literature.

To contribute to this area, we suggest developing the following studies.

Developing dynamic and flexible strategies for supply chain resilience for different disruption scenarios.

Analysing the impacts of dynamic strategies on resilience performance over time.

Developing dynamic supply chain resilience models for preparedness, response, and recovery considering different flexible strategies.

Comparing the findings for different flexible strategies to obtain the most suitable plans for dynamic supply chain resilience plans.

Theoretically Grounded Studies

Few studies developed theoretically grounded empirical models (Alvarenga et al., 2023 ; Gölgeci & Kuivalainen, 2020 ; Juan & Li, 2023 ; Ladeira et al., 2021 ; Pu et al., 2023a , 2023b ; Singh & Singh, 2019 ; Um & Han, 2021 ; Zhou et al., 2022 ; Zhu & Wu, 2022 ). However, there is a gap in the literature in relation to applying emergent theories such as the awareness–motivation–capability framework.

In the future, we propose considering theories from multiple disciplines to develop and test models to analyse the impacts of flexible strategies on supply chain resilience, including in dynamic and changed environments.

Analytical Studies

According to the literature review, different studies applied different analytical tools, such as mathematical programming and simulation approaches (Alikhani et al., 2021 ; Fattahi et al., 2020 ; Ivanov, 2022 ; Kummer et al., 2022 ; Mikhail et al., 2019 ; Pu et al., 2023a , 2023b ; Zavitsas et al., 2018 ). Integrating multiple analytical tools improves the quality of findings and the decision-making process in supply chain management. The flexible strategies and supply chain resilience literature has a gap in relation to integrating multiple analytical tools for analysing strategies and performance indicators.

In future, we propose applying multiple analytical tools to develop decision-making models for practitioners. We also suggest dividing the studies into different sections, applying analytical tools and connecting them again to improve the quality of findings.

Conclusions

The main objective of this study was to critically review the existing studies that considered flexible strategies for supply chain resilience. To fulfil this objective, we applied an SLR technique and analysed 62 related studies in the domain of contributions and findings, research contexts and business sectors, methodologies, different flexible strategies and performance indicators, and relationship mapping between flexible strategies and performance indicators.

The main contributions of this study are: (i) conducting an SLR in flexible strategies for supply chain resilience, which has not yet been explored in the literature, (ii) critically analysing the existing studies and presenting the findings, and (iii) proposing future research directions based on the identified research gaps.

The main findings indicated that more research is needed to analyse holistic relationships between flexible strategies and supply chain performance. Moreover, the service sector should be studied more, as it has been widely ignored in the literature thus far. Future research should also consider developing dynamic resilience plans using flexible strategies. Finally, more theoretically grounded and analytical studies should be conducted in the area of flexible strategies and supply chain resilience.

However, this review article has some limitations. First, we consider only journal articles published until 2023 and written in English. Second, the scope of the study was limited to flexible strategies and performance indicators used in the area of supply chain resilience. In the future, the timeline of published articles and the scope of the study can be further broadened. As this SLR paper provided a critical review, a summary of existing studies, and significant future research directions, the findings of the study can be used as a benchmark for future research in flexible strategies for supply chain resilience.

Key Questions

What contributions have been made in the connection between flexible strategies and supply chain resilience?

What are the emerging research opportunities in the area of flexible strategies and supply chain resilience?

There is no funding for this article.

Alikhani, R., Torabi, S. A., & Altay, N. (2021). Retail supply chain network design with concurrent resilience capabilities. International Journal of Production Economics, 234 , 108042. https://doi.org/10.1016/j.ijpe.2021.108042

Article   Google Scholar  

Altay, N., Gunasekaran, A., Dubey, R., & Childe, S. J. (2018). Agility and resilience as antecedents of supply chain performance under moderating effects of organizational culture within the humanitarian setting: A dynamic capability view. Production Planning and Control, 29 (14), 1158–1174. https://doi.org/10.1080/09537287.2018.1542174

Alvarenga, M. Z., de Oliveira, M. P. V., & de Oliveira, T. A. G. F. (2023). The impact of using digital technologies on supply chain resilience and robustness: The role of memory under the covid-19 outbreak. Supply Chain Management, 28 (5), 825–842. https://doi.org/10.1108/SCM-06-2022-0217

Carvalho, H., Barroso, A. P., MacHado, V. H., Azevedo, S., & Cruz-Machado, V. (2012). Supply chain redesign for resilience using simulation. Computers and Industrial Engineering, 62 (1), 329–341. https://doi.org/10.1016/j.cie.2011.10.003

Chen, H. Y., Das, A., & Ivanov, D. (2019). Building resilience and managing post-disruption supply chain recovery: Lessons from the information and communication technology industry. International Journal of Information Management, 49 , 330–342. https://doi.org/10.1016/j.ijinfomgt.2019.06.002

Chowdhury, M. M. H., Chowdhury, P., Quaddus, M., Rahman, K. W., & Shahriar, S. (2024). Flexibility in enhancing supply chain resilience: developing a resilience capability portfolio in the event of severe disruption. Global Journal of Flexible Systems Management , 25 (2), 395–417.

Chunsheng, L., Wong, C. W. Y., Yang, C. C., Shang, K. C., & Lirn, T. (2020). Value of supply chain resilience: Roles of culture, flexibility, and integration. International Journal of Physical Distribution & Logistics Management, 50 (1), 80–100. https://doi.org/10.1108/IJPDLM-02-2019-0041

Das, D., Datta, A., Kumar, P., Kazancoglu, Y., & Ram, M. (2022). Building supply chain resilience in the era of COVID-19: An AHP-DEMATEL approach. Operations Management Research, 15 (1–2), 249–267. https://doi.org/10.1007/s12063-021-00200-4

Dhillon, M. K., Rafi-ul-Shan, P. M., Amar, H., Sher, F., & Ahmed, S. (2023). Flexible green supply chain management in emerging economies: a systematic literature review. Global Journal of Flexible Systems Management, 24 (1), 1–28

Donadoni, M., Caniato, F., & Cagliano, R. (2018). Linking product complexity, disruption and performance: The moderating role of supply chain resilience. Supply Chain Forum, 19 (4), 300–310. https://doi.org/10.1080/16258312.2018.1551039

Dwivedi, A., Srivastava, S., Agrawal, D., Jha, A., & Paul, S. K. (2023). Analyzing the inter-relationships of business recovery challenges in the manufacturing industry: implications for post-pandemic supply chain resilience. Global Journal of Flexible Systems Management, 24 (1), 31–48. https://doi.org/10.1007/s40171-023-00365-w

Ekanayake, E. M. A. C., Shen, G. Q. P., Kumaraswamy, M. M., Owusu, E. K., & Saka, A. B. (2021). Modeling supply chain resilience in industrialized construction: A Hong Kong case. Journal of Construction Engineering and Management, 147 (11), 1–16. https://doi.org/10.1061/(asce)co.1943-7862.0002188

Fahimnia, B., & Jabbarzadeh, A. (2016). Marrying supply chain sustainability and resilience: A match made in heaven. Transportation Research Part e: Logistics and Transportation Review, 91 , 306–324. https://doi.org/10.1016/j.tre.2016.02.007

Fahimnia, B., Jabbarzadeh, A., & Sarkis, J. (2018). Greening versus resilience: A supply chain design perspective. Transportation Research Part e: Logistics and Transportation Review, 119 , 129–148. https://doi.org/10.1016/j.tre.2018.09.005

Faruquee, M., Paulraj, A., & Irawan, C. A. (2023). A typology of supply chain resilience: recognising the multi-capability nature of proactive and reactive contexts. Production Planning and Control , in press, 1–21. https://doi.org/10.1080/09537287.2023.2202151

Fattahi, M., Govindan, K., & Maihami, R. (2020). Stochastic optimization of disruption-driven supply chain network design with a new resilience metric. International Journal of Production Economics, 230 , 107755. https://doi.org/10.1016/j.ijpe.2020.107755

Gölgeci, I., & Kuivalainen, O. (2020). Does social capital matter for supply chain resilience? The role of absorptive capacity and marketing-supply chain management alignment. Industrial Marketing Management, 84 , 63–74. https://doi.org/10.1016/j.indmarman.2019.05.006

Grzybowska, K., & Stachowiak, A. (2022). Global changes and disruptions in supply chains—preliminary research to sustainable resilience of supply chains. Energies, 15 , 4579. https://doi.org/10.3390/en15134579

Hamidu, Z., Boachie-Mensah, F. O., & Issau, K. (2023a). Supply chain resilience and performance of manufacturing firms: Role of supply chain disruption. Journal of Manufacturing Technology Management, 34 (3), 361–382. https://doi.org/10.1108/JMTM-08-2022-0307

Hamidu, Z., Mensah, B. D., Issau, K., & Asafo-Adjei, E. (2023b). Does technological innovation matter in the nexus between supply chain resilience and performance of manufacturing firms in a developing economy? Journal of Manufacturing Technology Management, 34 (6), 981–1003. https://doi.org/10.1108/JMTM-11-2022-0384

Hamidu, Z., Issau, K., Boachie-Mensah, F. O., & Asafo-Adjei, E. (2024). On the interplay of supply chain network complexity on the nexus between supply chain resilience and performance. Benchmarking, 31 (5), 1590–1610. https://doi.org/10.1108/BIJ-09-2022-0551

Hohenstein, N. O., Feise, E., Hartmann, E., & Giunipero, L. (2015). Research on the phenomenon of supply chain resilience: A systematic review and paths for further investigation. International Journal of Physical Distribution and Logistics Management, 45 , 90–117. https://doi.org/10.1108/IJPDLM-05-2013-0128

Isti’anah, P. R., Praharsi, Y., Maharani, A., & Wee, H. M. (2021). Supply chain resilience analysis using the quality function deployment (QFD) approach in a freight forwarding company. Reliability: Theory and Applications, 16 (2), 15–26. https://doi.org/10.24412/1932-2321-2021-264-15-26

Ivanov, D. (2022). Blackout and supply chains: Cross-structural ripple effect, performance, resilience and viability impact analysis. Annals of Operations Research, in Press. https://doi.org/10.1007/s10479-022-04754-9

Juan, S. J., & Li, E. Y. (2023). Financial performance of firms with supply chains during the COVID-19 pandemic: The roles of dynamic capability and supply chain resilience. International Journal of Operations and Production Management, 43 (5), 712–737. https://doi.org/10.1108/IJOPM-04-2022-0249

Kamalahmadi, M., Shekarian, M., & Mellat Parast, M. (2022). The impact of flexibility and redundancy on improving supply chain resilience to disruptions. International Journal of Production Research, 60 (6), 1992–2020. https://doi.org/10.1080/00207543.2021.1883759

Kazancoglu, I., Ozbiltekin-Pala, M., Mangla, S. K., Kazancoglu, Y., & Jabeen, F. (2022). Role of flexibility, agility and responsiveness for sustainable supply chain resilience during COVID-19. Journal of Cleaner Production, 362 , 132431. https://doi.org/10.1016/j.jclepro.2022.132431

Kummer, Y., Fikar, C., Burtscher, J., Strobl, M., Fuchs, R., Domig, K. J., & Hirsch, P. (2022). Facilitating resilience during an african swine fever outbreak in the austrian pork supply chain through hybrid simulation modelling. Agriculture (Switzerland), 12 (3), 1–17. https://doi.org/10.3390/agriculture12030352

Ladeira, M. B., de Oliveira, M. P. V., de Sousa, P. R., & Barbosa, M. W. (2021). Firm’s supply chain agility enabling resilience and performance in turmoil times. International Journal of Agile Systems and Management, 14 (2), 224–253. https://doi.org/10.1504/IJASM.2021.118068

Li, Z., Liu, Q., Ye, C., Dong, M., & Zheng, Y. (2022). Achieving resilience: Resilient price and quality strategies of fresh food dual-channel supply chain considering the disruption. Sustainability (Switzerland), 14 (11), 6645. https://doi.org/10.3390/su14116645

Lin, Y., Chen, A., Zhong, S., Giannikas, V., Lomas, C., & Worth, T. (2023). Service supply chain resilience: A social-ecological perspective on last-mile delivery operations. International Journal of Operations and Production Management, 43 (1), 140–165. https://doi.org/10.1108/IJOPM-03-2022-0180

Macdonald, J. R., Zobel, C. W., Melnyk, S. A., & Griffis, S. E. (2018). Supply chain risk and resilience: Theory building through structured experiments and simulation. International Journal of Production Research, 56 (12), 4337–4355. https://doi.org/10.1080/00207543.2017.1421787

Mackay, J., Munoz, A., & Pepper, M. (2020). Conceptualising redundancy and flexibility towards supply chain robustness and resilience. Journal of Risk Research, 23 (12), 1541–1561. https://doi.org/10.1080/13669877.2019.1694964

Maharjan, R., & Kato, H. (2023). Logistics and supply chain resilience of Japanese companies: Perspectives from Impacts of the COVID-19 pandemic. Logistics, 7 (2), 27. https://doi.org/10.3390/logistics7020027

Mandal, S. (2014). Supply chain resilience: A state-of-the-art review and research directions. International Journal of Disaster Resilience in the Built Environment, 5 (4), 427–453. https://doi.org/10.1108/IJDRBE-03-2013-0003

Mao, X., Lou, X., Yuan, C., & Zhou, J. (2020). Resilience-based restoration model for supply chain networks. Mathematics, 8 (2), 163. https://doi.org/10.3390/math8020163

Massari, G. F., & Giannoccaro, I. (2021). Investigating the effect of horizontal coopetition on supply chain resilience in complex and turbulent environments. International Journal of Production Economics, 237 , 108150. https://doi.org/10.1016/j.ijpe.2021.108150

Mikhail, M., El-Beheiry, M., & Afia, N. (2019). Incorporating resilience determinants in supply chain network design model. Journal of Modelling in Management, 14 (3), 738–753. https://doi.org/10.1108/JM2-05-2018-0057

Moosavi, J., & Hosseini, S. (2021). Simulation-based assessment of supply chain resilience with consideration of recovery strategies in the COVID-19 pandemic context. Computers and Industrial Engineering, 160 , 107593. https://doi.org/10.1016/j.cie.2021.107593

Nagariya, R., Mukherjee, S., Baral, M. M., & Chittipaka, V. (2023). Analyzing blockchain-based supply chain resilience strategies: Resource-based perspective. International Journal of Productivity and Performance Management, in Press. https://doi.org/10.1108/IJPPM-07-2022-0330

Nguyen, D. N., Nguyen, T. T. H., Nguyen, T. T., Nguyen, X. H., Do, T. K. T., & Ngo, H. N. (2022). The effect of supply chain finance on supply chain risk, supply chain risk resilience, and performance of vietnam smes in global supply chain. Uncertain Supply Chain Management, 10 (1), 225–238. https://doi.org/10.5267/j.uscm.2021.9.005

Olivares-Aguila, J., & Vital-Soto, A. (2021). Supply chain resilience roadmaps for major disruptions. Logistics, 5 (4), 78. https://doi.org/10.3390/logistics5040078

Paul, S. K., & Chowdhury, P. (2020). Strategies for managing the impacts of disruptions during COVID-19: An example of toilet paper. Global Journal of Flexible Systems Management, 21 , 283–293. https://doi.org/10.1007/s40171-020-00248-4

Paul, S. K., Sarker, R., & Essam, D. (2017). A quantitative model for disruption mitigation in a supply chain. European Journal of Operational Research, 257 (3), 881–895. https://doi.org/10.1016/j.ejor.2016.08.035

Piprani, A. Z., Jaafar, N. I., Ali, S. M., Mubarik, M. S., & Shahbaz, M. (2022). Multi-dimensional supply chain flexibility and supply chain resilience: The role of supply chain risks exposure. Operations Management Research, 15 (1–2), 307–325. https://doi.org/10.1007/s12063-021-00232-w

Ponis, S. T., & Koronis, E. (2012). Supply Chain Resilience? Definition of concept and its formative elements. The Journal of Applied Business Research, 28 (5), 921–935. https://doi.org/10.19030/jabr.v28i5.7234

Praharsi, Y., Jamiin, M. A., Suhardjito, G., & Wee, H. M. (2021). The application of Lean Six Sigma and supply chain resilience in maritime industry during the era of COVID-19. International Journal of Lean Six Sigma, 12 (4), 800–834. https://doi.org/10.1108/IJLSS-11-2020-0196

Pu, G., Qiao, W., & Feng, Z. (2023a). Antecedents and outcomes of supply chain resilience: Integrating dynamic capabilities and relational perspective. Journal of Contingencies and Crisis Management, 31 (4), 706–726. https://doi.org/10.1111/1468-5973.12473

Pu, W., Ma, S., & Yan, X. (2023b). Geographical relevance-based multi-period optimization for e-commerce supply chain resilience strategies under disruption risks. International Journal of Production Research, in Press. https://doi.org/10.1080/00207543.2023.2217937

Purvis, L., Spall, S., Naim, M., & Spiegler, V. (2016). Developing a resilient supply chain strategy during ‘boom’ and ‘bust.’ Production Planning and Control, 27 (7–8), 579–590. https://doi.org/10.1080/09537287.2016.1165306

Rahman, T., Paul, S. K., Shukla, N., Agarwal, R., & Taghikhah, F. (2022). Supply chain resilience initiatives and strategies: A systematic review. Computers and Industrial Engineering, 170 , 108317. https://doi.org/10.1016/j.cie.2022.108317

Rajesh, R. (2016). Forecasting supply chain resilience performance using grey prediction. Electronic Commerce Research and Applications, 20 , 42–58. https://doi.org/10.1016/j.elerap.2016.09.006

Rajesh, R. (2021). Flexible business strategies to enhance resilience in manufacturing supply chains: An empirical study. Journal of Manufacturing Systems, 60 , 903–919. https://doi.org/10.1016/j.jmsy.2020.10.010

Ribeiro, J. P., & Barbosa-Povoa, A. (2018). Supply chain resilience: Definitions and quantitative modelling approaches–A literature review. Computers & Industrial Engineering, 115 , 109–122. https://doi.org/10.1016/j.cie.2017.11.006

Ryan, S. M., Roberts, E., Hibbett, E., Bloom, N., Haden, C., Rushforth, R. R., Pfeiffer, K., & Ruddell, B. L. (2021). The FEWSION for community resilience (F4R) process: Building local technical and social capacity for critical supply chain resilience. Frontiers in Environmental Science, 9 , 1–14. https://doi.org/10.3389/fenvs.2021.601220

Sangari, M. S., & Dashtpeyma, M. (2019). An integrated framework of supply chain resilience enablers: A hybrid ISM-FANP approach. International Journal of Business Excellence, 18 (2), 242–268. https://doi.org/10.1504/IJBEX.2019.099558

Scholten, K., Stevenson, M., & van Donk, D. P. (2020). Dealing with the unpredictable: Supply chain resilience. International Journal of Operations & Production Management, 40 (1), 1–10. https://doi.org/10.1108/IJOPM-01-2020-789

Shashi, Centobelli, P., Cerchione, R., & Ertz, M. (2020). Managing supply chain resilience to pursue business and environmental strategies. Business Strategy and the Environment, 29 (3), 1215–1246. https://doi.org/10.1002/bse.2428

Sharma, B., Mittal, M. L., Soni, G., & Ramtiyal, B. (2023). An implementation framework for resiliency assessment in a supply chain. Global Journal of Flexible Systems Management , 24 (4), 591–614

Shishodia, A., Sharma, R., Rajesh, R., & Munim, Z. H. (2023). Supply chain resilience: A review, conceptual framework and future research. International Journal of Logistics Management, 34 (4), 879–908. https://doi.org/10.1108/IJLM-03-2021-0169

Shweta, K., & D., & Chandra, D. (2023). A hybrid framework to model resilience in the generic medicine supply chain of MSMEs. Benchmarking, 30 (6), 2189–2224. https://doi.org/10.1108/BIJ-11-2021-0697

Silva, M. E., Pereira, M. M. O., & Hendry, L. C. (2023). Embracing change in tandem: Resilience and sustainability together transforming supply chains. International Journal of Operations and Production Management, 43 (1), 166–196. https://doi.org/10.1108/IJOPM-09-2022-0625

Singh, N. P., & Singh, S. (2019). Building supply chain risk resilience: Role of big data analytics in supply chain disruption mitigation. Benchmarking, 26 (7), 2318–2342. https://doi.org/10.1108/BIJ-10-2018-0346

Suryadi, A., & Rau, H. (2023). Considering region risks and mitigation strategies in the supplier selection process for improving supply chain resilience. Computers and Industrial Engineering, 181 , 109288. https://doi.org/10.1016/j.cie.2023.109288

Suryawanshi, P., Dutta, P., Varun, L., & Deepak, G. (2021). Sustainable and resilience planning for the supply chain of online hyperlocal grocery services. Sustainable Production and Consumption, 28 , 496–518. https://doi.org/10.1016/j.spc.2021.05.001

Tan, W. J., Cai, W., & Zhang, A. N. (2020). Structural-aware simulation analysis of supply chain resilience. International Journal of Production Research, 58 (17), 5175–5195. https://doi.org/10.1080/00207543.2019.1705421

Tang, C., & Tomlin, B. (2008). The power of flexibility for mitigating supply chain risks. International Journal of Production Economics, 116 (1), 12–27. https://doi.org/10.1016/j.ijpe.2008.07.008

Trabucco, M., & De Giovanni, P. (2021). Achieving resilience and business sustainability during COVID-19: The role of lean supply chain practices and digitalization. Sustainability (Switzerland), 13 (22), 12369. https://doi.org/10.3390/su132212369

Tranfield, D., Denyer, D., & Smart, P. (2003). Towards a methodology for developing evidence-informed management knowledge by means of systematic review. British Journal of Management, 14 (3), 207–222. https://doi.org/10.1111/1467-8551.00375

Tukamuhabwa, B. R., Stevenson, M., Busby, J., & Zorzini, M. (2015). Supply chain resilience: Definition, review and theoretical foundations for further study. International Journal of Production Research, 53 (18), 5592–5623. https://doi.org/10.1080/00207543.2015.1037934

Um, J., & Han, N. (2021). Understanding the relationships between global supply chain risk and supply chain resilience: The role of mitigating strategies. Supply Chain Management, 26 (2), 240–255. https://doi.org/10.1108/SCM-06-2020-0248

Varma, S., Singh, N., & Patra, A. (2024). Supply chain flexibility: Unravelling the research trajectory through citation path analysis. Global Journal of Flexible Systems Management, 25 (2), 199–222

Vimal, K. E. K., Nadeem, S. P., Meledathu Sunil, S., Suresh, G., Sanjeev, N., & Kandasamy, J. (2022a). Modelling the strategies for improving maturity and resilience in medical oxygen supply chain through digital technologies. Journal of Global Operations and Strategic Sourcing, 15 (4), 566–595. https://doi.org/10.1108/JGOSS-10-2021-0088

Vimal, K. E. K., Nadeem, S. P., Ravichandran, M., Ethirajan, M., & Kandasamy, J. (2022b). Resilience strategies to recover from the cascading ripple effect in a copper supply chain through project management. Operations Management Research, 15 , 440–460. https://doi.org/10.1007/s12063-021-00231-x

Wadhwa, S., Saxena, A., & Chan, F. T. S. (2008). Framework for flexibility in dynamic supply chain management. International Journal of Production Research, 46 (6), 1373–1404. https://doi.org/10.1080/00207540600570432

Wang, X., Herty, M., & Zhao, L. (2016). Contingent rerouting for enhancing supply chain resilience from supplier behavior perspective. International Transactions in Operational Research, 23 (4), 775–796. https://doi.org/10.1111/itor.12151

Xu, B., Liu, W., Li, J., Yang, Y., Wen, F., & Song, H. (2023). Resilience measurement and dynamic optimization of container logistics supply chain under adverse events. Computers and Industrial Engineering, 180 , 109202. https://doi.org/10.1016/j.cie.2023.109202

Yi, C. Y., Ngai, E. W. T., & Moon, K. (2011). Supply chain flexibility in an uncertain environment: Exploratory findings from five case studies. Supply Chain Management: An International Journal, 16 (4), 271–283. https://doi.org/10.1108/13598541111139080

Zavala-Alcívar, A., Verdecho, M. J., & Alfaro-Saiz, J. J. (2020). A conceptual framework to manage resilience and increase sustainability in the supply chain. Sustainability (Switzerland), 12 (16), 6300. https://doi.org/10.3390/SU12166300

Zavitsas, K., Zis, T., & Bell, M. G. H. (2018). The impact of flexible environmental policy on maritime supply chain resilience. Transport Policy, 72 , 116–128. https://doi.org/10.1016/j.tranpol.2018.09.020

Zhou, J., Hu, L., Yu, Y., Zhang, J. Z., & Zheng, L. J. (2022). Impacts of IT capability and supply chain collaboration on supply chain resilience: Empirical evidence from China in COVID-19 pandemic. Journal of Enterprise Information Management, in Press. https://doi.org/10.1108/JEIM-03-2022-0091

Zhu, X., & Wu, Y. J. (2022). How does supply chain resilience affect supply chain performance? The Mediating Effect of Sustainability. Sustainability (Switzerland), 14 (21), 1–19. https://doi.org/10.3390/su142114626

Download references

Acknowledgements

This research is supported by an Australian Government Research Training Program Scholarship.

Open Access funding enabled and organized by CAUL and its Member Institutions.

Author information

Authors and affiliations.

Faculty of Engineering and IT, School of Mechanical and Mechatronic Engineering, University of Technology Sydney, Ultimo, NSW, 2007, Australia

Ananna Paul & Suvash C. Saha

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Suvash C. Saha .

Ethics declarations

Conflict of interest.

The authors declare that they have no known competing financial interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Paul, A., Saha, S.C. A Systematic Literature Review on Flexible Strategies and Performance Indicators for Supply Chain Resilience. Glob J Flex Syst Manag (2024). https://doi.org/10.1007/s40171-024-00415-x

Download citation

Received : 16 April 2024

Accepted : 20 August 2024

Published : 10 September 2024

DOI : https://doi.org/10.1007/s40171-024-00415-x

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Flexible strategies
• Performance indicators
• Supply chain resilience
• Systematic literature review
• Find a journal
• Publish with us
• Track your research

Information

• Author Services

Initiatives

You are accessing a machine-readable page. In order to be human-readable, please install an RSS reader.

All articles published by MDPI are made immediately available worldwide under an open access license. No special permission is required to reuse all or part of the article published by MDPI, including figures and tables. For articles published under an open access Creative Common CC BY license, any part of the article may be reused without permission provided that the original article is clearly cited. For more information, please refer to https://www.mdpi.com/openaccess .

Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for future research directions and describes possible research applications.

Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive positive feedback from the reviewers.

Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world. Editors select a small number of articles recently published in the journal that they believe will be particularly interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the most exciting work published in the various research areas of the journal.

Original Submission Date Received: .

• Active Journals
• Find a Journal
• Proceedings Series
• For Authors
• For Reviewers
• For Editors
• For Librarians
• For Publishers
• For Societies
• For Conference Organizers
• Open Access Policy
• Institutional Open Access Program
• Special Issues Guidelines
• Editorial Process
• Research and Publication Ethics
• Article Processing Charges
• Testimonials
• Preprints.org
• SciProfiles
• Encyclopedia

Article Menu

• Subscribe SciFeed
• Recommended Articles
• Google Scholar
• on Google Scholar
• Table of Contents

Find support for a specific problem in the support section of our website.

Please let us know what you think of our products and services.

Visit our dedicated information section to learn more about MDPI.

JSmol Viewer

The state of the art of digital twins in health—a quick review of the literature.

1. Introduction

2. literature review, 2.1. digital twins, 2.2. digital health, 2.3. healthcare, 3. methodology, 5. discussion, 5.1. axis 1: use of digital twins for virtual representation of biological structures, 5.2. axis 2: use of digital twins to improve healthcare processes (personalized care), 5.3. axis 3: use of digital twins to depict healthcare structures and improve operational efficiency, 5.4. axis 4: use of digital twins for the development of medicines and health devices.

• Data Integration: Healthcare management involves a vast amount of information and clinical data. One of the main challenges is the efficient integration of these data into digital twins, ensuring that all relevant information is available in one place. This can be complicated by the diversity of health record systems and data standards.
• Privacy and Security: Maintaining the privacy and security of health data is a critical concern. Digital twins contain highly sensitive information, and it is essential to ensure that they are protected from unauthorized access and data breaches.
• Interoperability: Healthcare systems often use different technologies and standards. For digital twins to be effective, they need to be interoperable, i.e., able to communicate and share information effectively between different systems.

6. Conclusions

Author contributions, data availability statement, conflicts of interest.

• Ahram, T.; Karwowski, W.; Falcão, C. Applications of Systems Engineering for Testing and Evaluation: A Human Systems Integration Perspective. In APA Handbook of Human Systems Integration ; Boehm-Davis, D.A., Durso, F., Lee, J.D., Eds.; APA handbooks in psychology; American Psychological Association: Washington, DC, USA, 2015; pp. 101–111. [ Google Scholar ] [ CrossRef ]
• Gunal, M.M. (Ed.) Simulation for Industry 4.0 ; Springer: Cham, Switzerland, 2019. [ Google Scholar ] [ CrossRef ]
• Lu, Q.; Xie, X.; Parlikad, A.K.; Schooling, J.M.; Konstantinou, E. Moving from building information models to digital twins for operation and maintenance. Proc. Inst. Civ. Eng. 2021 , 174 , 46–56. [ Google Scholar ] [ CrossRef ]
• Pires, F.; Cachada, A.; Barbosa, J.; Moreira, A.P.; Leitão, P. Digital Twin in Industry 4.0: Technologies, Applications and Challenges. In Proceedings of the 2019 IEEE 17th International Conference on Industrial Informatics (INDIN), Helsinki, Finland, 22–25 July 2019; pp. 721–726. [ Google Scholar ] [ CrossRef ]
• Tao, F.; Zhang, M. Digital twin shop-floor: A New shop-floor paradigm towards smart manufacturing. IEEE Access 2017 , 5 , 20418–20427. [ Google Scholar ] [ CrossRef ]
• Mussomeli, A.; Meeker, B.; Shepley, S.; Schatsky, D. Expecting Digital Twins ; Deloitte Insights: Helsinki, Finland, 2018; Available online: https://www2.deloitte.com/content/dam/insights/us/articles/3773_Expecting-digital-twins/DI_Expecting-digital-twins.pdf (accessed on 6 October 2023).
• Barricelli, B.R.; Casiraghi, E.; Fogli, D. A Survey on Digital Twin: Definitions, Characteristics, Applications, and Design Implications. IEEE Access 2019 , 7 , 167653–167671. [ Google Scholar ] [ CrossRef ]
• Semeraro, C.; Lezoche, M.; Panetto, H.; Dassisti, M. Digital twin paradigm: A systematic literature review. Comput. Ind. 2021 , 130 , 103469. [ Google Scholar ] [ CrossRef ]
• Kritzinger, W.; Karner, M.; Traar, G.; Henjes, J.; Sihn, W. Digital Twin in manufacturing: A categorical literature review and classification. IFAC-PapersOnLine 2018 , 51 , 1016–1022. [ Google Scholar ] [ CrossRef ]
• Rivera, L.F.; Jiménez, M.; Angara, P.; Villegas, N.M.; Tamura, G.; Müller, H.A. Towards continuous monitoring in personalized healthcare through digital twins. In Proceedings of the 29th Annual International Conference on Computer Science and Software Engineering, Toronto, ON, Canada, 4–6 November 2019. [ Google Scholar ]
• Scharff, S. From Digital Twin I’m Improved Patient Experience. Siemens Healthineers. 10 September 2010. Available online: https://www.siemens-healthineers.com/perspectives/mso-digital-twin-mater.html (accessed on 16 October 2023).
• Sun, T.; He, X.; Li, Z. Digital twin in healthcare: Recent updates and challenges. Digit. Health 2023 , 9 , 20552076221149651. [ Google Scholar ] [ CrossRef ] [ PubMed ] [ PubMed Central ]
• Fuller, A.; Fan, Z.; Day, C.; Barlow, C. Digital Twin: Enabling Technologies, Challenges and Open Research. IEEE Access 2020 , 8 , 108952–108971. [ Google Scholar ] [ CrossRef ]
• Armentano, R.; Bhadoria, R.S.; Chatterjee, P.; Deka, G.C. The Internet of Things: Foundation for Smart Cities, eHealth, and Ubiquitous Computing ; Chapman and Hall/CRC: Boca Raton, FL, USA, 2017. [ Google Scholar ]
• Lu, V.Q.; Parlikad, A.K.; Woodall, P. Developing a dynamic digital twin at a building level: Using Cambridge campus as case study. In Proceedings of the 2019 International Conference on Smart Infrastructure and Construction 2019 (ICSIC) Driving Data-Informed Decision-Making, Cambridge, UK, 8–10 July 2019. [ Google Scholar ]
• Grieves, M. Digital twin: Manufacturing excellence through virtual factory replication. In White Paper ; NASA: Greenbelt, MA, USA, 2014. [ Google Scholar ]
• Tsaramirsis, G.; Kantaros, A.; Al-Darraji, I.; Piromalis, D.; Apostolopoulos, C.; Pavlopoulou, A.; Alrammal, M.; Ismail, Z.; Buhari, S.M.; Stojmenovic, M.; et al. A Modern Approach towards an Industry 4.0 Model: From Driving Technologies to Management. J. Sens. 2022 , 2022 , 5023011. [ Google Scholar ] [ CrossRef ]
• Guo, J.; Lv, Z. Application of Digital Twins in multiple fields. Multimed Tools Appl. 2022 , 81 , 26941–26967. [ Google Scholar ] [ CrossRef ] [ PubMed ] [ PubMed Central ]
• Schwartz, S.M.; Wildenhaus, K.; Bucher, A.; Byrd, B. Digital Twins and the Emerging Science of Self: Implications for Digital Health Experience Design and “Small” Data. Front. Comput. Sci. 2020 , 2 , 31. [ Google Scholar ] [ CrossRef ]
• Stojanovic, V.; Trapp, M.; Richter, R.; Hagedorn, B.; Döllner, J.; Alamaniotis, M. Towards the generation of digital twins for facility management based on 3D point clouds. In Proceedings of the 34th Annual ARCOM Conference, Belfast, UK, 3–5 September 2018. [ Google Scholar ]
• Karakra, A.; Fontanili, F.; Lamine, E.; Lamothe, J.; Taweel, A. Pervasive computing integrated discrete event simulation for a hospital digital twin. In Proceedings of the 2018 IEEE/ACS 15th International Conference on Computer Systems and Applications (AICCSA), Aqaba, Jordan, 28 October–1 November 2018; pp. 1–6. [ Google Scholar ]
• Aluvalu, R.; Mudrakola, S.; Maheswari, V.U.; Kaladevi, A.C.; Sandhya, M.V.S.; Bhat, C.R. The novel emergency hospital services for patients using digital twins. Microprocess. Microsyst. 2023 , 98 , 104794. [ Google Scholar ] [ CrossRef ]
• Björnsson, B.; Borrebaeck, C.; Elander, N.; Gasslander, T.; Gawel, D.R.; Gustafsson, M.; Jörnsten, R.; Lee, E.J.; Li, X.; Lilja, S.; et al. Digital twins to personalize medicine. Genome Med. 2019 , 12 , 4. [ Google Scholar ] [ CrossRef ]
• Chaudhari, P.; Gangane, C.; Lahe, A. Digital Twin in Industry 4.0 A Real-Time Virtual Replica of Objects Improves Digital Health Monitoring System. In Information Systems and Management Science, Proceedings of the 4th International Conference on Information Systems and Management Science, Msida, Malta, 14–15 December 2021 ; Springer International Publishing: Cham, Switzerland, 2021. [ Google Scholar ]
• Cheng, W.; Lian, W.; Tian, J. Building the intelligent hospital twins for all-scenario intelligence health care. Digit. Health 2022 , 8 , 20552076221107894. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Safa, E.; Asan, O. Digital Twins for Managing Health Care Systems: Rapid Literature Review. J. Med. Internet Res. 2022 , 24 , e37641. [ Google Scholar ]
• Michelfeit, F. Exploring the Possibilities Offered by Digital Twins in Medical Technology. Siemens Healthcare GmbH-Communc. RSNA. 2018. Available online: https://cdn0.scrvt.com/39b415fb07de4d9656c7b516d8e2d907/1800000005899262/d3fa3e835e2d/Exploring-the-possibilities-offered-by-digital-twins-in-medical-technology_1800000005899262.pdf (accessed on 23 March 2024).
• Gabrielli, S.; Piras, E.M.; Ibarra, O.M. Digital Twins in the Future Design of Digital Therapeutics. In Proceedings of the Adjunct, 2023 ACM International Joint Conference on Pervasive and Ubiquitous Computing & the 2023 ACM International Symposium on Wearable Computing, Cancun, Mexico, 8–12 October 2023. [ Google Scholar ]
• Marshall, T. VPH: The Ultimate Stage Before Your Own Medical Digital Twin. 6 September 2016. Available online: https://www.linkedin.com/pulse/vph-ultimate-stage-before-your-own-medical-digital-twin-marchal/ (accessed on 16 October 2019).
• Vallée, A. Digital twin for healthcare systems. Front. Digit. Health 2023 , 5 , 1253050. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Venkatesh, K.P.; Raza, M.M.; Kvedar, J.C. Health digital twins as tools for precision medicine: Considerations for computation, implementation, and regulation. NPJ Digit. Med. 2022 , 5 , 150. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Viceconti, M.; De Vos, M.; Mellone, S.; Geris, L. Position Paper from the Digital Twins in Healthcare to the Virtual Human Twin: A Moon-Shot Project for Digital Health Research. IEEE J. Biomed. Health Informatics 2023 , 28 , 491–501. [ Google Scholar ] [ CrossRef ]
• World Health Organization. Global Strategy on Digital Health 2020–2025 ; Licence: CC BY-NC-SA 3.0 IGO; World Health Organization: Geneva, Switzerland, 2021. [ Google Scholar ]
• Brazil. Ministry of Health. Executive Secretariat. SUS Information Technology Department. Digital Health Strategy for Brazil 2020–2028 [Electronic Resource]/Ministry of Health, Executive Secretariat, SUS Information Technology Department—Brasília: Ministry of Health, 2020; 128 p.: il. ISBN 978-85-334-2841-6. Available online: http://bvsms.saude.gov.br/bvs/publicacoes/estrategia_saude_digital_Brasil.pdf (accessed on 27 March 2024).
• Garcia, L.P.; Duarte, E. Infodemic: Excess quantity to the detriment of quality of information about COVID-19. Epidemiologia Serv. Saude 2020 , 29 , e2020186. [ Google Scholar ] [ CrossRef ]
• Huang, P.-h.; Kim, K.-H.; Schermer, M. Ethical issues of digital twins for personalized health care service: Preliminary mapping study. J. Med. Internet Res. 2022 , 24 , e33081. [ Google Scholar ] [ CrossRef ]
• Millstone, E.; van Zwanenberg, P. The crisis of trust: For science, scientists or for institutions? Nat. Med. 2000 , 6 , 1307–1308. [ Google Scholar ] [ CrossRef ]
• Barros, M.; Lima, P.; Oliveira, D.; Arcanjo, C.; Oliveira, L.; Pereira, S. Digital Inclusion and Education: Equity and access. Int. J. Sci. Studies 2023 , 1 , 124–149. [ Google Scholar ] [ CrossRef ]
• WHO—World Health Organization. WHO Global Strategy on Integrated People-Centred Health Services 2016-2026—Executive Summary: Placing People and Communities at the Centre of Health Services. [Sl]: Draft for Consultation, Version 2015. Available online: https://iris.who.int/bitstream/handle/10665/180984/WHO_HIS_SDS_2015.20_eng.pdf?sequence=1&isAllowed=y (accessed on 21 September 2023).
• Majumder, S.; Deen, M.J. Smartphone Sensors for Health Monitoring and Diagnosis. Sensors 2019 , 19 , 2164. [ Google Scholar ] [ CrossRef ]
• Ricci, A.; Croatti, A.; Montagna, S. Pervasive and connected digital twins—A vision for digital health. IEEE Internet Comput. 2021 , 26 , 26–32. [ Google Scholar ] [ CrossRef ]
• WHO—World Health Organization. Why Are Health Systems Important? WHR 2000. WHO. Available online: https://www.who.int/publications/i/item/924156198X (accessed on 27 March 2024).
• El-Warrak, L.; Nunes, M.; Luna, G.; Barbosa, C.E.; Lyra, A.; Argôlo, M.; Lima, Y.; Salazar, H.; de Souza, J.M. Towards the future of Public Health: Roadmapping Trends and Scenarios in the Post-COVID Healthcare Era. Healthcare 2023 , 11 , 3118. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• US National Science Foundation. Science and Engineering Indicators 2018 ; National Science Foundation: Alexandria, VA, USA, 2018. Available online: https://www.nsf.gov/statistics/2018/nsb20181/assets/nsb20181.pdf (accessed on 10 July 2024).
• Rodrigues, J.L.d.S.d.Q.; Portela, M.C.; Malik, A.M. Agenda for research on patient-centred care in Brazil. Ciência Saúde Coletiva 2019 , 24 , 4263–4275. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• The Health Foundation. Person-Centred Care Made Simple: What Everyone Should Know about Person-Centred Care. Quick Guide ; The Health Foundation: London, UK, 2016; Available online: https://www.health.org.uk/publications/person-centred-care-made-simple (accessed on 20 September 2023).
• Wickramasinghe, N.; Jayaraman, P.P.; Forkan, A.R.M.; Ulapane, N.; Kaul, R.; Vaughan, S.; Zelcer, J. A Vision for Leveraging the Concept of Digital Twins I’m Support the Provision of Personalized Cancer Care. IEEE Internet Comput. 2022 , 26 , 17–24. [ Google Scholar ] [ CrossRef ]

Click here to enlarge figure

Share and Cite

El-Warrak, L.; de Farias, C.M. The State of the Art of Digital Twins in Health—A Quick Review of the Literature. Computers 2024 , 13 , 228. https://doi.org/10.3390/computers13090228

El-Warrak L, de Farias CM. The State of the Art of Digital Twins in Health—A Quick Review of the Literature. Computers . 2024; 13(9):228. https://doi.org/10.3390/computers13090228

El-Warrak, Leonardo, and Claudio M. de Farias. 2024. "The State of the Art of Digital Twins in Health—A Quick Review of the Literature" Computers 13, no. 9: 228. https://doi.org/10.3390/computers13090228

Article Metrics

Article access statistics, further information, mdpi initiatives, follow mdpi.

Subscribe to receive issue release notifications and newsletters from MDPI journals