thematic analysis of literature review

The Guide to Thematic Analysis

• What is Thematic Analysis?
• Advantages of Thematic Analysis
• Disadvantages of Thematic Analysis
• Thematic Analysis Examples
• How to Do Thematic Analysis
• Thematic Coding
• Collaborative Thematic Analysis
• Thematic Analysis Software
• Thematic Analysis in Mixed Methods Approach
• Abductive Thematic Analysis
• Deductive Thematic Analysis
• Inductive Thematic Analysis
• Reflexive Thematic Analysis
• Thematic Analysis in Observations
• Thematic Analysis in Surveys
• Thematic Analysis for Interviews
• Thematic Analysis for Focus Groups
• Thematic Analysis for Case Studies
• Thematic Analysis of Secondary Data
• Introduction

What is a thematic literature review?

Advantages of a thematic literature review, structuring and writing a thematic literature review.

• Thematic Analysis vs. Phenomenology
• Thematic vs. Content Analysis
• Thematic Analysis vs. Grounded Theory
• Thematic Analysis vs. Narrative Analysis
• Thematic Analysis vs. Discourse Analysis
• Thematic Analysis vs. Framework Analysis
• Thematic Analysis in Social Work
• Thematic Analysis in Psychology
• Thematic Analysis in Educational Research
• Thematic Analysis in UX Research
• How to Present Thematic Analysis Results
• Increasing Rigor in Thematic Analysis
• Peer Review in Thematic Analysis

Thematic Analysis Literature Review

A thematic literature review serves as a critical tool for synthesizing research findings within a specific subject area. By categorizing existing literature into themes, this method offers a structured approach to identify and analyze patterns and trends across studies. The primary goal is to provide a clear and concise overview that aids scholars and practitioners in understanding the key discussions and developments within a field. Unlike traditional literature reviews , which may adopt a chronological approach or focus on individual studies, a thematic literature review emphasizes the aggregation of findings through key themes and thematic connections. This introduction sets the stage for a detailed examination of what constitutes a thematic literature review, its benefits, and guidance on effectively structuring and writing one.

A thematic literature review methodically organizes and examines a body of literature by identifying, analyzing, and reporting themes found within texts such as journal articles, conference proceedings, dissertations, and other forms of academic writing. While a particular journal article may offer some specific insight, a synthesis of knowledge through a literature review can provide a comprehensive overview of theories across relevant sources in a particular field.

Unlike other review types that might organize literature chronologically or by methodology , a thematic review focuses on recurring themes or patterns across a collection of works. This approach enables researchers to draw together previous research to synthesize findings from different research contexts and methodologies, highlighting the overarching trends and insights within a field.

At its core, a thematic approach to a literature review research project involves several key steps. Initially, it requires the comprehensive collection of relevant literature that aligns with the review's research question or objectives. Following this, the process entails a meticulous analysis of the texts to identify common themes that emerge across the studies. These themes are not pre-defined but are discovered through a careful reading and synthesis of the literature.

The thematic analysis process is iterative, often involving the refinement of themes as the review progresses. It allows for the integration of a broad range of literature, facilitating a multidimensional understanding of the research topic. By organizing literature thematically, the review illuminates how various studies contribute to each theme, providing insights into the depth and breadth of research in the area.

A thematic literature review thus serves as a foundational element in research, offering a nuanced and comprehensive perspective on a topic. It not only aids in identifying gaps in the existing literature but also guides future research directions by underscoring areas that warrant further investigation. Ultimately, a thematic literature review empowers researchers to construct a coherent narrative that weaves together disparate studies into a unified analysis.

Organize your literature search with ATLAS.ti

Collect and categorize documents to identify gaps and key findings with ATLAS.ti. Download a free trial.

Conducting a literature review thematically provides a comprehensive and nuanced synthesis of research findings, distinguishing it from other types of literature reviews. Its structured approach not only facilitates a deeper understanding of the subject area but also enhances the clarity and relevance of the review. Here are three significant advantages of employing a thematic analysis in literature reviews.

Enhanced understanding of the research field

Thematic literature reviews allow for a detailed exploration of the research landscape, presenting themes that capture the essence of the subject area. By identifying and analyzing these themes, reviewers can construct a narrative that reflects the complexity and multifaceted nature of the field.

This process aids in uncovering underlying patterns and relationships, offering a more profound and insightful examination of the literature. As a result, readers gain an enriched understanding of the key concepts, debates, and evolutionary trajectories within the research area.

Identification of research gaps and trends

One of the pivotal benefits of a thematic literature review is its ability to highlight gaps in the existing body of research. By systematically organizing the literature into themes, reviewers can pinpoint areas that are under-explored or warrant further investigation.

Additionally, this method can reveal emerging trends and shifts in research focus, guiding scholars toward promising areas for future study. The thematic structure thus serves as a roadmap, directing researchers toward uncharted territories and new research questions .

Facilitates comparative analysis and integration of findings

A thematic literature review excels in synthesizing findings from diverse studies, enabling a coherent and integrated overview. By concentrating on themes rather than individual studies, the review can draw comparisons and contrasts across different research contexts and methodologies . This comparative analysis enriches the review, offering a panoramic view of the field that acknowledges both consensus and divergence among researchers.

Moreover, the thematic framework supports the integration of findings, presenting a unified and comprehensive portrayal of the research area. Such integration is invaluable for scholars seeking to navigate the extensive body of literature and extract pertinent insights relevant to their own research questions or objectives.

The process of structuring and writing a thematic literature review is pivotal in presenting research in a clear, coherent, and impactful manner. This review type necessitates a methodical approach to not only unearth and categorize key themes but also to articulate them in a manner that is both accessible and informative to the reader. The following sections outline essential stages in the thematic analysis process for literature reviews , offering a structured pathway from initial planning to the final presentation of findings.

Identifying and categorizing themes

The initial phase in a thematic literature review is the identification of themes within the collected body of literature. This involves a detailed examination of texts to discern patterns, concepts, and ideas that recur across the research landscape. Effective identification hinges on a thorough and nuanced reading of the literature, where the reviewer actively engages with the content to extract and note significant thematic elements. Once identified, these themes must be meticulously categorized, often requiring the reviewer to discern between overarching themes and more nuanced sub-themes, ensuring a logical and hierarchical organization of the review content.

Analyzing and synthesizing themes

After categorizing the themes, the next step involves a deeper analysis and synthesis of the identified themes. This stage is critical for understanding the relationships between themes and for interpreting the broader implications of the thematic findings. Analysis may reveal how themes evolve over time, differ across methodologies or contexts, or converge to highlight predominant trends in the research area. Synthesis involves integrating insights from various studies to construct a comprehensive narrative that encapsulates the thematic essence of the literature, offering new interpretations or revealing gaps in existing research.

Presenting and discussing findings

The final stage of the thematic literature review is the discussion of the thematic findings in a research paper or presentation. This entails not only a descriptive account of identified themes but also a critical examination of their significance within the research field. Each theme should be discussed in detail, elucidating its relevance, the extent of research support, and its implications for future studies. The review should culminate in a coherent and compelling narrative that not only summarizes the key thematic findings but also situates them within the broader research context, offering valuable insights and directions for future inquiry.

Gain a comprehensive understanding of your data with ATLAS.ti

Analyze qualitative data with specific themes that offer insights. See how with a free trial.

Educational resources and simple solutions for your research journey

How to Write a Thematic Literature Review: A Beginner’s Guide

Literature reviews provide a comprehensive understanding of existing knowledge in a particular field, offer insights into gaps and trends, and ultimately lay the foundation for innovative research. However, when tackling complex topics spanning multiple issues, the conventional approach of a standard literature review might not suffice. Many researchers present a literature review without giving any thought to its organization or structure, but this is where a thematic literature review comes into play. In this article, we will explore the significance of thematic reviews, delve into how and when to undertake them, and offer invaluable guidance on structuring and crafting a compelling thematic literature review.

Table of Contents

What is a thematic literature review?

A thematic literature review, also known as a thematic review, involves organizing and synthesizing the existing literature based on recurring themes or topics rather than a chronological or methodological sequence. Typically, when a student or researcher works intensively on their research there are many sub-domains or associated spheres of knowledge that one encounters. While these may not have a direct bearing on the main idea being explored, they provide a much-needed background or context to the discussion. This is where a thematic literature review is useful when dealing with complex research questions that involve multiple facets, as it allows for a more in-depth exploration of specific themes within the broader context.

When to opt for thematic literature review?

It is common practice for early career researchers and students to collate all the literature reviews they have undertaken under one single broad umbrella. However, when working on a literature review that involves multiple themes, lack of organization and structure can slow you down and create confusion. Deciding to embark on a thematic literature review is a strategic choice that should align with your research objectives. Here are some scenarios where opting for a thematic review is advantageous:

• Broad Research Questions: When your research question spans across various dimensions and cannot be adequately addressed through a traditional literature review.
• Interdisciplinary Research: In cases where your research draws from multiple disciplines, a thematic review helps in synthesizing diverse literature cohesively.
• Emerging Research Areas: When exploring emerging fields or topics with limited existing literature, a thematic review can provide valuable insights by focusing on available themes.
• Complex Issues: Thematic reviews are ideal for dissecting complex issues with multiple contributing factors or dimensions.

Advantages of a Thematic Literature Review

With better comprehension and broad insights, thematic literature reviews can help in identifying possible research gaps across themes. A thematic literature review has several advantages over a general or broad-based approach, especially for those working on multiple related themes.

• It provides a comprehensive understanding of specific themes within a broader context, allowing for a deep exploration of relevant literature.
• Thematic reviews offer a structured approach to organizing and synthesizing diverse sources, making it easier to identify trends, patterns, and gaps.
• Researchers can focus on key themes, enabling a more detailed analysis of specific aspects of the research question.
• Thematic reviews facilitate the integration of literature from various disciplines, offering a holistic view of the topic.
• Researchers can provide targeted recommendations or insights related to specific themes, aiding in the formulation of research hypotheses.

Now that we know the benefits of a thematic literature review, what is the best way to arrange reviewed literature in a thematic format?

How to write a thematic literature review

To effectively structure and write a thematic literature review, follow these key steps:

• Define Your Research Question: Clearly define the overarching research question or topic you aim to explore thematically. When writing a thematic literature review, go through different literature review sections of published research work and understand the subtle nuances associated with this approach.
• Identify Themes: Analyze the literature to identify recurring themes or topics relevant to your research question. Categorize the bibliography by dividing them into relevant clusters or units, each dealing with a specific issue. For example, you can divide a topic based on a theoretical approach, methodology, discipline or by epistemology. A theoretical review of related literature for example, may also look to break down geography or issues pertaining to a single country into its different parts or along rural and urban divides.
• Organize the Literature: Group the literature into thematic clusters based on the identified themes. Each cluster represents a different aspect of your research question. It is up to you to define the different narratives of thematic literature reviews depending on the project being undertaken; there is no one formal way of doing this. You can weigh how specific areas stack up against others in terms of existing literature or studies and how many more aspects may need to be added or further looked into.
• Review and Synthesize: Within each thematic cluster, review and synthesize the relevant literature, highlighting key findings and insights. It is recommended to identify any theme-related strengths or weaknesses using an analytical lens.
• Integrate Themes: Analyze how the themes interact with each other, draw linkages between earlier studies and see how they contribute to your own research. A thematic literature review presents readers with a comprehensive overview of the literature available on and around the research topic.
• Provide a Framework: Develop a framework or conceptual model that illustrates the relationships between the themes. Present the most relevant part of the thematic review toward the end and study it in greater detail as it reflects the literature most relevant and directly related to the main research topic.
• Conclusion: Conclude your thematic literature review by summarizing the key findings and their implications for your research question. Be sure to highlight any gaps or areas requiring further investigation in this section.
• Cite and Reference: It is important to remember that a thematic review of literature for a PhD thesis or research paper lends greater credibility to the student or researcher. So ensure that you properly cite and reference all sources according to your chosen citation style.
• Edit and Proofread: Take some time to review your work, ensure proper structure and flow and eliminate any language, grammar, or spelling errors that could deviate reader attention. This will help you deliver a well-structured and elegantly written thematic literature review.

Thematic literature review example

In essence, a thematic literature review allows researchers to dissect complex topics into smaller manageable themes, providing a more focused and structured approach to literature synthesis. This method empowers researchers to gain deeper insights, identify gaps, and generate new knowledge within the context of their research.

To illustrate the process mentioned above, let’s consider an example of a thematic literature review in the context of sustainable development. Imagine the overarching research question is: “What are the key factors influencing sustainable urban planning?” Potential themes could include environmental sustainability, social equity, economic viability, and governance. Each theme would have a dedicated section in the review, summarizing relevant literature and discussing how these factors intersect and impact sustainable urban planning. Close with a strong conclusion that highlights research gaps or areas of investigation. Finally, review and refine the thematic literature review, adding citations and references as required.

In conclusion, when tackling multifaceted research questions, a thematic literature review proves to be an indispensable tool for researchers and students alike. By adopting this approach, scholars can navigate the intricate web of existing literature, unearth meaningful patterns, and contribute to the advancement of knowledge in their respective fields. We hope the information in this article helps you create thematic reviews that illuminate your path to new discoveries and innovative insights.

R Discovery is a literature search and research reading platform that accelerates your research discovery journey by keeping you updated on the latest, most relevant scholarly content. With 250M+ research articles sourced from trusted aggregators like CrossRef, Unpaywall, PubMed, PubMed Central, Open Alex and top publishing houses like Springer Nature, JAMA, IOP, Taylor & Francis, NEJM, BMJ, Karger, SAGE, Emerald Publishing and more, R Discovery puts a world of research at your fingertips.  

Try R Discovery Prime FREE for 1 week or upgrade at just US$72 a year to access premium features that let you listen to research on the go, read in your language, collaborate with peers, auto sync with reference managers, and much more. Choose a simpler, smarter way to find and read research – Download the app and start your free 7-day trial today !  

Related Posts

What is Research Impact: Types and Tips for Academics

Research in Shorts: R Discovery’s New Feature Helps Academics Assess Relevant Papers in 2mins 

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• How to Do Thematic Analysis | Step-by-Step Guide & Examples

How to Do Thematic Analysis | Step-by-Step Guide & Examples

Published on September 6, 2019 by Jack Caulfield . Revised on June 22, 2023.

Thematic analysis is a method of analyzing qualitative data . It is usually applied to a set of texts, such as an interview or transcripts . The researcher closely examines the data to identify common themes – topics, ideas and patterns of meaning that come up repeatedly.

There are various approaches to conducting thematic analysis, but the most common form follows a six-step process: familiarization, coding, generating themes, reviewing themes, defining and naming themes, and writing up. Following this process can also help you avoid confirmation bias when formulating your analysis.

This process was originally developed for psychology research by Virginia Braun and Victoria Clarke . However, thematic analysis is a flexible method that can be adapted to many different kinds of research.

Table of contents

When to use thematic analysis, different approaches to thematic analysis, step 1: familiarization, step 2: coding, step 3: generating themes, step 4: reviewing themes, step 5: defining and naming themes, step 6: writing up, other interesting articles.

Thematic analysis is a good approach to research where you’re trying to find out something about people’s views, opinions, knowledge, experiences or values from a set of qualitative data – for example, interview transcripts , social media profiles, or survey responses .

Some types of research questions you might use thematic analysis to answer:

• How do patients perceive doctors in a hospital setting?
• What are young women’s experiences on dating sites?
• What are non-experts’ ideas and opinions about climate change?
• How is gender constructed in high school history teaching?

To answer any of these questions, you would collect data from a group of relevant participants and then analyze it. Thematic analysis allows you a lot of flexibility in interpreting the data, and allows you to approach large data sets more easily by sorting them into broad themes.

However, it also involves the risk of missing nuances in the data. Thematic analysis is often quite subjective and relies on the researcher’s judgement, so you have to reflect carefully on your own choices and interpretations.

Pay close attention to the data to ensure that you’re not picking up on things that are not there – or obscuring things that are.

Receive feedback on language, structure, and formatting

Professional editors proofread and edit your paper by focusing on:

• Academic style
• Vague sentences
• Style consistency

See an example

Once you’ve decided to use thematic analysis, there are different approaches to consider.

There’s the distinction between inductive and deductive approaches:

• An inductive approach involves allowing the data to determine your themes.
• A deductive approach involves coming to the data with some preconceived themes you expect to find reflected there, based on theory or existing knowledge.

Ask yourself: Does my theoretical framework give me a strong idea of what kind of themes I expect to find in the data (deductive), or am I planning to develop my own framework based on what I find (inductive)?

There’s also the distinction between a semantic and a latent approach:

• A semantic approach involves analyzing the explicit content of the data.
• A latent approach involves reading into the subtext and assumptions underlying the data.

Ask yourself: Am I interested in people’s stated opinions (semantic) or in what their statements reveal about their assumptions and social context (latent)?

After you’ve decided thematic analysis is the right method for analyzing your data, and you’ve thought about the approach you’re going to take, you can follow the six steps developed by Braun and Clarke .

The first step is to get to know our data. It’s important to get a thorough overview of all the data we collected before we start analyzing individual items.

This might involve transcribing audio , reading through the text and taking initial notes, and generally looking through the data to get familiar with it.

Next up, we need to code the data. Coding means highlighting sections of our text – usually phrases or sentences – and coming up with shorthand labels or “codes” to describe their content.

Let’s take a short example text. Say we’re researching perceptions of climate change among conservative voters aged 50 and up, and we have collected data through a series of interviews. An extract from one interview looks like this:

In this extract, we’ve highlighted various phrases in different colors corresponding to different codes. Each code describes the idea or feeling expressed in that part of the text.

At this stage, we want to be thorough: we go through the transcript of every interview and highlight everything that jumps out as relevant or potentially interesting. As well as highlighting all the phrases and sentences that match these codes, we can keep adding new codes as we go through the text.

After we’ve been through the text, we collate together all the data into groups identified by code. These codes allow us to gain a a condensed overview of the main points and common meanings that recur throughout the data.

Here's why students love Scribbr's proofreading services

Discover proofreading & editing

Next, we look over the codes we’ve created, identify patterns among them, and start coming up with themes.

Themes are generally broader than codes. Most of the time, you’ll combine several codes into a single theme. In our example, we might start combining codes into themes like this:

At this stage, we might decide that some of our codes are too vague or not relevant enough (for example, because they don’t appear very often in the data), so they can be discarded.

Other codes might become themes in their own right. In our example, we decided that the code “uncertainty” made sense as a theme, with some other codes incorporated into it.

Again, what we decide will vary according to what we’re trying to find out. We want to create potential themes that tell us something helpful about the data for our purposes.

Now we have to make sure that our themes are useful and accurate representations of the data. Here, we return to the data set and compare our themes against it. Are we missing anything? Are these themes really present in the data? What can we change to make our themes work better?

If we encounter problems with our themes, we might split them up, combine them, discard them or create new ones: whatever makes them more useful and accurate.

For example, we might decide upon looking through the data that “changing terminology” fits better under the “uncertainty” theme than under “distrust of experts,” since the data labelled with this code involves confusion, not necessarily distrust.

Now that you have a final list of themes, it’s time to name and define each of them.

Defining themes involves formulating exactly what we mean by each theme and figuring out how it helps us understand the data.

Naming themes involves coming up with a succinct and easily understandable name for each theme.

For example, we might look at “distrust of experts” and determine exactly who we mean by “experts” in this theme. We might decide that a better name for the theme is “distrust of authority” or “conspiracy thinking”.

Finally, we’ll write up our analysis of the data. Like all academic texts, writing up a thematic analysis requires an introduction to establish our research question, aims and approach.

We should also include a methodology section, describing how we collected the data (e.g. through semi-structured interviews or open-ended survey questions ) and explaining how we conducted the thematic analysis itself.

The results or findings section usually addresses each theme in turn. We describe how often the themes come up and what they mean, including examples from the data as evidence. Finally, our conclusion explains the main takeaways and shows how the analysis has answered our research question.

In our example, we might argue that conspiracy thinking about climate change is widespread among older conservative voters, point out the uncertainty with which many voters view the issue, and discuss the role of misinformation in respondents’ perceptions.

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

• Normal distribution
• Measures of central tendency
• Chi square tests
• Confidence interval
• Quartiles & Quantiles
• Cluster sampling
• Stratified sampling
• Discourse analysis
• Cohort study
• Peer review
• Ethnography

Research bias

• Implicit bias
• Cognitive bias
• Conformity bias
• Hawthorne effect
• Availability heuristic
• Attrition bias
• Social desirability bias

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

Caulfield, J. (2023, June 22). How to Do Thematic Analysis | Step-by-Step Guide & Examples. Scribbr. Retrieved September 13, 2024, from https://www.scribbr.com/methodology/thematic-analysis/

Is this article helpful?

Jack Caulfield

Other students also liked, what is qualitative research | methods & examples, inductive vs. deductive research approach | steps & examples, critical discourse analysis | definition, guide & examples, what is your plagiarism score.

• Research article
• Open access
• Published: 10 July 2008

Methods for the thematic synthesis of qualitative research in systematic reviews

• James Thomas 1 &
• Angela Harden 1  

BMC Medical Research Methodology volume  8 , Article number:  45 ( 2008 ) Cite this article

396k Accesses

4536 Citations

104 Altmetric

Metrics details

There is a growing recognition of the value of synthesising qualitative research in the evidence base in order to facilitate effective and appropriate health care. In response to this, methods for undertaking these syntheses are currently being developed. Thematic analysis is a method that is often used to analyse data in primary qualitative research. This paper reports on the use of this type of analysis in systematic reviews to bring together and integrate the findings of multiple qualitative studies.

We describe thematic synthesis, outline several steps for its conduct and illustrate the process and outcome of this approach using a completed review of health promotion research. Thematic synthesis has three stages: the coding of text 'line-by-line'; the development of 'descriptive themes'; and the generation of 'analytical themes'. While the development of descriptive themes remains 'close' to the primary studies, the analytical themes represent a stage of interpretation whereby the reviewers 'go beyond' the primary studies and generate new interpretive constructs, explanations or hypotheses. The use of computer software can facilitate this method of synthesis; detailed guidance is given on how this can be achieved.

We used thematic synthesis to combine the studies of children's views and identified key themes to explore in the intervention studies. Most interventions were based in school and often combined learning about health benefits with 'hands-on' experience. The studies of children's views suggested that fruit and vegetables should be treated in different ways, and that messages should not focus on health warnings. Interventions that were in line with these suggestions tended to be more effective. Thematic synthesis enabled us to stay 'close' to the results of the primary studies, synthesising them in a transparent way, and facilitating the explicit production of new concepts and hypotheses.

We compare thematic synthesis to other methods for the synthesis of qualitative research, discussing issues of context and rigour. Thematic synthesis is presented as a tried and tested method that preserves an explicit and transparent link between conclusions and the text of primary studies; as such it preserves principles that have traditionally been important to systematic reviewing.

Peer Review reports

The systematic review is an important technology for the evidence-informed policy and practice movement, which aims to bring research closer to decision-making [ 1 , 2 ]. This type of review uses rigorous and explicit methods to bring together the results of primary research in order to provide reliable answers to particular questions [ 3 – 6 ]. The picture that is presented aims to be distorted neither by biases in the review process nor by biases in the primary research which the review contains [ 7 – 10 ]. Systematic review methods are well-developed for certain types of research, such as randomised controlled trials (RCTs). Methods for reviewing qualitative research in a systematic way are still emerging, and there is much ongoing development and debate [ 11 – 14 ].

In this paper we present one approach to the synthesis of findings of qualitative research, which we have called 'thematic synthesis'. We have developed and applied these methods within several systematic reviews that address questions about people's perspectives and experiences [ 15 – 18 ]. The context for this methodological development is a programme of work in health promotion and public health (HP & PH), mostly funded by the English Department of Health, at the EPPI-Centre, in the Social Science Research Unit at the Institute of Education, University of London in the UK. Early systematic reviews at the EPPI-Centre addressed the question 'what works?' and contained research testing the effects of interventions. However, policy makers and other review users also posed questions about intervention need, appropriateness and acceptability, and factors influencing intervention implementation. To address these questions, our reviews began to include a wider range of research, including research often described as 'qualitative'. We began to focus, in particular, on research that aimed to understand the health issue in question from the experiences and point of view of the groups of people targeted by HP&PH interventions (We use the term 'qualitative' research cautiously because it encompasses a multitude of research methods at the same time as an assumed range of epistemological positions. In practice it is often difficult to classify research as being either 'qualitative' or 'quantitative' as much research contains aspects of both [ 19 – 22 ]. Because the term is in common use, however, we will employ it in this paper).

When we started the work for our first series of reviews which included qualitative research in 1999 [ 23 – 26 ], there was very little published material that described methods for synthesising this type of research. We therefore experimented with a variety of techniques borrowed from standard systematic review methods and methods for analysing primary qualitative research [ 15 ]. In later reviews, we were able to refine these methods and began to apply thematic analysis in a more explicit way. The methods for thematic synthesis described in this paper have so far been used explicitly in three systematic reviews [ 16 – 18 ].

The review used as an example in this paper

To illustrate the steps involved in a thematic synthesis we draw on a review of the barriers to, and facilitators of, healthy eating amongst children aged four to 10 years old [ 17 ]. The review was commissioned by the Department of Health, England to inform policy about how to encourage children to eat healthily in the light of recent surveys highlighting that British children are eating less than half the recommended five portions of fruit and vegetables per day. While we focus on the aspects of the review that relate to qualitative studies, the review was broader than this and combined answering traditional questions of effectiveness, through reviewing controlled trials, with questions relating to children's views of healthy eating, which were answered using qualitative studies. The qualitative studies were synthesised using 'thematic synthesis' – the subject of this paper. We compared the effectiveness of interventions which appeared to be in line with recommendations from the thematic synthesis with those that did not. This enabled us to see whether the understandings we had gained from the children's views helped us to explain differences in the effectiveness of different interventions: the thematic synthesis had enabled us to generate hypotheses which could be tested against the findings of the quantitative studies – hypotheses that we could not have generated without the thematic synthesis. The methods of this part of the review are published in Thomas et al . [ 27 ] and are discussed further in Harden and Thomas [ 21 ].

Qualitative research and systematic reviews

The act of seeking to synthesise qualitative research means stepping into more complex and contested territory than is the case when only RCTs are included in a review. First, methods are much less developed in this area, with fewer completed reviews available from which to learn, and second, the whole enterprise of synthesising qualitative research is itself hotly debated. Qualitative research, it is often proposed, is not generalisable and is specific to a particular context, time and group of participants. Thus, in bringing such research together, reviewers are open to the charge that they de-contextualise findings and wrongly assume that these are commensurable [ 11 , 13 ]. These are serious concerns which it is not the purpose of this paper to contest. We note, however, that a strong case has been made for qualitative research to be valued for the potential it has to inform policy and practice [ 11 , 28 – 30 ]. In our experience, users of reviews are interested in the answers that only qualitative research can provide, but are not able to handle the deluge of data that would result if they tried to locate, read and interpret all the relevant research themselves. Thus, if we acknowledge the unique importance of qualitative research, we need also to recognise that methods are required to bring its findings together for a wide audience – at the same time as preserving and respecting its essential context and complexity.

The earliest published work that we know of that deals with methods for synthesising qualitative research was written in 1988 by Noblit and Hare [ 31 ]. This book describes the way that ethnographic research might be synthesised, but the method has been shown to be applicable to qualitative research beyond ethnography [ 32 , 11 ]. As well as meta-ethnography, other methods have been developed more recently, including 'meta-study' [ 33 ], 'critical interpretive synthesis' [ 34 ] and 'metasynthesis' [ 13 ].

Many of the newer methods being developed have much in common with meta-ethnography, as originally described by Noblit and Hare, and often state explicitly that they are drawing on this work. In essence, this method involves identifying key concepts from studies and translating them into one another. The term 'translating' in this context refers to the process of taking concepts from one study and recognising the same concepts in another study, though they may not be expressed using identical words. Explanations or theories associated with these concepts are also extracted and a 'line of argument' may be developed, pulling corroborating concepts together and, crucially, going beyond the content of the original studies (though 'refutational' concepts might not be amenable to this process). Some have claimed that this notion of 'going beyond' the primary studies is a critical component of synthesis, and is what distinguishes it from the types of summaries of findings that typify traditional literature reviews [e.g. [ 32 ], p209]. In the words of Margarete Sandelowski, "metasyntheses are integrations that are more than the sum of parts, in that they offer novel interpretations of findings. These interpretations will not be found in any one research report but, rather, are inferences derived from taking all of the reports in a sample as a whole" [[ 14 ], p1358].

Thematic analysis has been identified as one of a range of potential methods for research synthesis alongside meta-ethnography and 'metasynthesis', though precisely what the method involves is unclear, and there are few examples of it being used for synthesising research [ 35 ]. We have adopted the term 'thematic synthesis', as we translated methods for the analysis of primary research – often termed 'thematic' – for use in systematic reviews [ 36 – 38 ]. As Boyatzis [[ 36 ], p4] has observed, thematic analysis is "not another qualitative method but a process that can be used with most, if not all, qualitative methods..." . Our approach concurs with this conceptualisation of thematic analysis, since the method we employed draws on other established methods but uses techniques commonly described as 'thematic analysis' in order to formalise the identification and development of themes.

We now move to a description of the methods we used in our example systematic review. While this paper has the traditional structure for reporting the results of a research project, the detailed methods (e.g. precise terms we used for searching) and results are available online. This paper identifies the particular issues that relate especially to reviewing qualitative research systematically and then to describing the activity of thematic synthesis in detail.

When searching for studies for inclusion in a 'traditional' statistical meta-analysis, the aim of searching is to locate all relevant studies. Failing to do this can undermine the statistical models that underpin the analysis and bias the results. However, Doyle [[ 39 ], p326] states that, "like meta-analysis, meta-ethnography utilizes multiple empirical studies but, unlike meta-analysis, the sample is purposive rather than exhaustive because the purpose is interpretive explanation and not prediction" . This suggests that it may not be necessary to locate every available study because, for example, the results of a conceptual synthesis will not change if ten rather than five studies contain the same concept, but will depend on the range of concepts found in the studies, their context, and whether they are in agreement or not. Thus, principles such as aiming for 'conceptual saturation' might be more appropriate when planning a search strategy for qualitative research, although it is not yet clear how these principles can be applied in practice. Similarly, other principles from primary qualitative research methods may also be 'borrowed' such as deliberately seeking studies which might act as negative cases, aiming for maximum variability and, in essence, designing the resulting set of studies to be heterogeneous, in some ways, instead of achieving the homogeneity that is often the aim in statistical meta-analyses.

However you look, qualitative research is difficult to find [ 40 – 42 ]. In our review, it was not possible to rely on simple electronic searches of databases. We needed to search extensively in 'grey' literature, ask authors of relevant papers if they knew of more studies, and look especially for book chapters, and we spent a lot of effort screening titles and abstracts by hand and looking through journals manually. In this sense, while we were not driven by the statistical imperative of locating every relevant study, when it actually came down to searching, we found that there was very little difference in the methods we had to use to find qualitative studies compared to the methods we use when searching for studies for inclusion in a meta-analysis.

Quality assessment

Assessing the quality of qualitative research has attracted much debate and there is little consensus regarding how quality should be assessed, who should assess quality, and, indeed, whether quality can or should be assessed in relation to 'qualitative' research at all [ 43 , 22 , 44 , 45 ]. We take the view that the quality of qualitative research should be assessed to avoid drawing unreliable conclusions. However, since there is little empirical evidence on which to base decisions for excluding studies based on quality assessment, we took the approach in this review to use 'sensitivity analyses' (described below) to assess the possible impact of study quality on the review's findings.

In our example review we assessed our studies according to 12 criteria, which were derived from existing sets of criteria proposed for assessing the quality of qualitative research [ 46 – 49 ], principles of good practice for conducting social research with children [ 50 ], and whether studies employed appropriate methods for addressing our review questions. The 12 criteria covered three main quality issues. Five related to the quality of the reporting of a study's aims, context, rationale, methods and findings (e.g. was there an adequate description of the sample used and the methods for how the sample was selected and recruited?). A further four criteria related to the sufficiency of the strategies employed to establish the reliability and validity of data collection tools and methods of analysis, and hence the validity of the findings. The final three criteria related to the assessment of the appropriateness of the study methods for ensuring that findings about the barriers to, and facilitators of, healthy eating were rooted in children's own perspectives (e.g. were data collection methods appropriate for helping children to express their views?).

Extracting data from studies

One issue which is difficult to deal with when synthesising 'qualitative' studies is 'what counts as data' or 'findings'? This problem is easily addressed when a statistical meta-analysis is being conducted: the numeric results of RCTs – for example, the mean difference in outcome between the intervention and control – are taken from published reports and are entered into the software package being used to calculate the pooled effect size [ 3 , 51 ].

Deciding what to abstract from the published report of a 'qualitative' study is much more difficult. Campbell et al . [ 11 ] extracted what they called the 'key concepts' from the qualitative studies they found about patients' experiences of diabetes and diabetes care. However, finding the key concepts in 'qualitative' research is not always straightforward either. As Sandelowski and Barroso [ 52 ] discovered, identifying the findings in qualitative research can be complicated by varied reporting styles or the misrepresentation of data as findings (as for example when data are used to 'let participants speak for themselves'). Sandelowski and Barroso [ 53 ] have argued that the findings of qualitative (and, indeed, all empirical) research are distinct from the data upon which they are based, the methods used to derive them, externally sourced data, and researchers' conclusions and implications.

In our example review, while it was relatively easy to identify 'data' in the studies – usually in the form of quotations from the children themselves – it was often difficult to identify key concepts or succinct summaries of findings, especially for studies that had undertaken relatively simple analyses and had not gone much further than describing and summarising what the children had said. To resolve this problem we took study findings to be all of the text labelled as 'results' or 'findings' in study reports – though we also found 'findings' in the abstracts which were not always reported in the same way in the text. Study reports ranged in size from a few pages to full final project reports. We entered all the results of the studies verbatim into QSR's NVivo software for qualitative data analysis. Where we had the documents in electronic form this process was straightforward even for large amounts of text. When electronic versions were not available, the results sections were either re-typed or scanned in using a flat-bed or pen scanner. (We have since adapted our own reviewing system, 'EPPI-Reviewer' [ 54 ], to handle this type of synthesis and the screenshots below show this software.)

Detailed methods for thematic synthesis

The synthesis took the form of three stages which overlapped to some degree: the free line-by-line coding of the findings of primary studies; the organisation of these 'free codes' into related areas to construct 'descriptive' themes; and the development of 'analytical' themes.

Stages one and two: coding text and developing descriptive themes

In our children and healthy eating review, we originally planned to extract and synthesise study findings according to our review questions regarding the barriers to, and facilitators of, healthy eating amongst children. It soon became apparent, however, that few study findings addressed these questions directly and it appeared that we were in danger of ending up with an empty synthesis. We were also concerned about imposing the a priori framework implied by our review questions onto study findings without allowing for the possibility that a different or modified framework may be a better fit. We therefore temporarily put our review questions to one side and started from the study findings themselves to conduct an thematic analysis.

There were eight relevant qualitative studies examining children's views of healthy eating. We entered the verbatim findings of these studies into our database. Three reviewers then independently coded each line of text according to its meaning and content. Figure 1 illustrates this line-by-line coding using our specialist reviewing software, EPPI-Reviewer, which includes a component designed to support thematic synthesis. The text which was taken from the report of the primary study is on the left and codes were created inductively to capture the meaning and content of each sentence. Codes could be structured, either in a tree form (as shown in the figure) or as 'free' codes – without a hierarchical structure.

line-by-line coding in EPPI-Reviewer.

The use of line-by-line coding enabled us to undertake what has been described as one of the key tasks in the synthesis of qualitative research: the translation of concepts from one study to another [ 32 , 55 ]. However, this process may not be regarded as a simple one of translation. As we coded each new study we added to our 'bank' of codes and developed new ones when necessary. As well as translating concepts between studies, we had already begun the process of synthesis (For another account of this process, see Doyle [[ 39 ], p331]). Every sentence had at least one code applied, and most were categorised using several codes (e.g. 'children prefer fruit to vegetables' or 'why eat healthily?'). Before completing this stage of the synthesis, we also examined all the text which had a given code applied to check consistency of interpretation and to see whether additional levels of coding were needed. (In grounded theory this is termed 'axial' coding; see Fisher [ 55 ] for further discussion of the application of axial coding in research synthesis.) This process created a total of 36 initial codes. For example, some of the text we coded as "bad food = nice, good food = awful" from one study [ 56 ] were:

'All the things that are bad for you are nice and all the things that are good for you are awful.' (Boys, year 6) [[ 56 ], p74]

'All adverts for healthy stuff go on about healthy things. The adverts for unhealthy things tell you how nice they taste.' [[ 56 ], p75]

Some children reported throwing away foods they knew had been put in because they were 'good for you' and only ate the crisps and chocolate . [[ 56 ], p75]

Reviewers looked for similarities and differences between the codes in order to start grouping them into a hierarchical tree structure. New codes were created to capture the meaning of groups of initial codes. This process resulted in a tree structure with several layers to organize a total of 12 descriptive themes (Figure 2 ). For example, the first layer divided the 12 themes into whether they were concerned with children's understandings of healthy eating or influences on children's food choice. The above example, about children's preferences for food, was placed in both areas, since the findings related both to children's reactions to the foods they were given, and to how they behaved when given the choice over what foods they might eat. A draft summary of the findings across the studies organized by the 12 descriptive themes was then written by one of the review authors. Two other review authors commented on this draft and a final version was agreed.

relationships between descriptive themes.

Stage three: generating analytical themes

Up until this point, we had produced a synthesis which kept very close to the original findings of the included studies. The findings of each study had been combined into a whole via a listing of themes which described children's perspectives on healthy eating. However, we did not yet have a synthesis product that addressed directly the concerns of our review – regarding how to promote healthy eating, in particular fruit and vegetable intake, amongst children. Neither had we 'gone beyond' the findings of the primary studies and generated additional concepts, understandings or hypotheses. As noted earlier, the idea or step of 'going beyond' the content of the original studies has been identified by some as the defining characteristic of synthesis [ 32 , 14 ].

This stage of a qualitative synthesis is the most difficult to describe and is, potentially, the most controversial, since it is dependent on the judgement and insights of the reviewers. The equivalent stage in meta-ethnography is the development of 'third order interpretations' which go beyond the content of original studies [ 32 , 11 ]. In our example, the step of 'going beyond' the content of the original studies was achieved by using the descriptive themes that emerged from our inductive analysis of study findings to answer the review questions we had temporarily put to one side. Reviewers inferred barriers and facilitators from the views children were expressing about healthy eating or food in general, captured by the descriptive themes, and then considered the implications of children's views for intervention development. Each reviewer first did this independently and then as a group. Through this discussion more abstract or analytical themes began to emerge. The barriers and facilitators and implications for intervention development were examined again in light of these themes and changes made as necessary. This cyclical process was repeated until the new themes were sufficiently abstract to describe and/or explain all of our initial descriptive themes, our inferred barriers and facilitators and implications for intervention development.

For example, five of the 12 descriptive themes concerned the influences on children's choice of foods (food preferences, perceptions of health benefits, knowledge behaviour gap, roles and responsibilities, non-influencing factors). From these, reviewers inferred several barriers and implications for intervention development. Children identified readily that taste was the major concern for them when selecting food and that health was either a secondary factor or, in some cases, a reason for rejecting food. Children also felt that buying healthy food was not a legitimate use of their pocket money, which they would use to buy sweets that could be enjoyed with friends. These perspectives indicated to us that branding fruit and vegetables as a 'tasty' rather than 'healthy' might be more effective in increasing consumption. As one child noted astutely, 'All adverts for healthy stuff go on about healthy things. The adverts for unhealthy things tell you how nice they taste.' [[ 56 ], p75]. We captured this line of argument in the analytical theme entitled 'Children do not see it as their role to be interested in health'. Altogether, this process resulted in the generation of six analytical themes which were associated with ten recommendations for interventions.

Six main issues emerged from the studies of children's views: (1) children do not see it as their role to be interested in health; (2) children do not see messages about future health as personally relevant or credible; (3) fruit, vegetables and confectionery have very different meanings for children; (4) children actively seek ways to exercise their own choices with regard to food; (5) children value eating as a social occasion; and (6) children see the contradiction between what is promoted in theory and what adults provide in practice. The review found that most interventions were based in school (though frequently with parental involvement) and often combined learning about the health benefits of fruit and vegetables with 'hands-on' experience in the form of food preparation and taste-testing. Interventions targeted at people with particular risk factors worked better than others, and multi-component interventions that combined the promotion of physical activity with healthy eating did not work as well as those that only concentrated on healthy eating. The studies of children's views suggested that fruit and vegetables should be treated in different ways in interventions, and that messages should not focus on health warnings. Interventions that were in line with these suggestions tended to be more effective than those which were not.

Context and rigour in thematic synthesis

The process of translation, through the development of descriptive and analytical themes, can be carried out in a rigorous way that facilitates transparency of reporting. Since we aim to produce a synthesis that both generates 'abstract and formal theories' that are nevertheless 'empirically faithful to the cases from which they were developed' [[ 53 ], p1371], we see the explicit recording of the development of themes as being central to the method. The use of software as described can facilitate this by allowing reviewers to examine the contribution made to their findings by individual studies, groups of studies, or sub-populations within studies.

Some may argue against the synthesis of qualitative research on the grounds that the findings of individual studies are de-contextualised and that concepts identified in one setting are not applicable to others [ 32 ]. However, the act of synthesis could be viewed as similar to the role of a research user when reading a piece of qualitative research and deciding how useful it is to their own situation. In the case of synthesis, reviewers translate themes and concepts from one situation to another and can always be checking that each transfer is valid and whether there are any reasons that understandings gained in one context might not be transferred to another. We attempted to preserve context by providing structured summaries of each study detailing aims, methods and methodological quality, and setting and sample. This meant that readers of our review were able to judge for themselves whether or not the contexts of the studies the review contained were similar to their own. In the synthesis we also checked whether the emerging findings really were transferable across different study contexts. For example, we tried throughout the synthesis to distinguish between participants (e.g. boys and girls) where the primary research had made an appropriate distinction. We then looked to see whether some of our synthesis findings could be attributed to a particular group of children or setting. In the event, we did not find any themes that belonged to a specific group, but another outcome of this process was a realisation that the contextual information given in the reports of studies was very restricted indeed. It was therefore difficult to make the best use of context in our synthesis.

In checking that we were not translating concepts into situations where they did not belong, we were following a principle that others have followed when using synthesis methods to build grounded formal theory: that of grounding a text in the context in which it was constructed. As Margaret Kearney has noted "the conditions under which data were collected, analysis was done, findings were found, and products were written for each contributing report should be taken into consideration in developing a more generalized and abstract model" [[ 14 ], p1353]. Britten et al . [ 32 ] suggest that it may be important to make a deliberate attempt to include studies conducted across diverse settings to achieve the higher level of abstraction that is aimed for in a meta-ethnography.

Study quality and sensitivity analyses

We assessed the 'quality' of our studies with regard to the degree to which they represented the views of their participants. In doing this, we were locating the concept of 'quality' within the context of the purpose of our review – children's views – and not necessarily the context of the primary studies themselves. Our 'hierarchy of evidence', therefore, did not prioritise the research design of studies but emphasised the ability of the studies to answer our review question. A traditional systematic review of controlled trials would contain a quality assessment stage, the purpose of which is to exclude studies that do not provide a reliable answer to the review question. However, given that there were no accepted – or empirically tested – methods for excluding qualitative studies from syntheses on the basis of their quality [ 57 , 12 , 58 ], we included all studies regardless of their quality.

Nevertheless, our studies did differ according to the quality criteria they were assessed against and it was important that we considered this in some way. In systematic reviews of trials, 'sensitivity analyses' – analyses which test the effect on the synthesis of including and excluding findings from studies of differing quality – are often carried out. Dixon-Woods et al . [ 12 ] suggest that assessing the feasibility and worth of conducting sensitivity analyses within syntheses of qualitative research should be an important focus of synthesis methods work. After our thematic synthesis was complete, we examined the relative contributions of studies to our final analytic themes and recommendations for interventions. We found that the poorer quality studies contributed comparatively little to the synthesis and did not contain many unique themes; the better studies, on the other hand, appeared to have more developed analyses and contributed most to the synthesis.

This paper has discussed the rationale for reviewing and synthesising qualitative research in a systematic way and has outlined one specific approach for doing this: thematic synthesis. While it is not the only method which might be used – and we have discussed some of the other options available – we present it here as a tested technique that has worked in the systematic reviews in which it has been employed.

We have observed that one of the key tasks in the synthesis of qualitative research is the translation of concepts between studies. While the activity of translating concepts is usually undertaken in the few syntheses of qualitative research that exist, there are few examples that specify the detail of how this translation is actually carried out. The example above shows how we achieved the translation of concepts across studies through the use of line-by-line coding, the organisation of these codes into descriptive themes, and the generation of analytical themes through the application of a higher level theoretical framework. This paper therefore also demonstrates how the methods and process of a thematic synthesis can be written up in a transparent way.

This paper goes some way to addressing concerns regarding the use of thematic analysis in research synthesis raised by Dixon-Woods and colleagues who argue that the approach can lack transparency due to a failure to distinguish between 'data-driven' or 'theory-driven' approaches. Moreover they suggest that, "if thematic analysis is limited to summarising themes reported in primary studies, it offers little by way of theoretical structure within which to develop higher order thematic categories..." [[ 35 ], p47]. Part of the problem, they observe, is that the precise methods of thematic synthesis are unclear. Our approach contains a clear separation between the 'data-driven' descriptive themes and the 'theory-driven' analytical themes and demonstrates how the review questions provided a theoretical structure within which it became possible to develop higher order thematic categories.

The theme of 'going beyond' the content of the primary studies was discussed earlier. Citing Strike and Posner [ 59 ], Campbell et al . [[ 11 ], p672] also suggest that synthesis "involves some degree of conceptual innovation, or employment of concepts not found in the characterisation of the parts and a means of creating the whole" . This was certainly true of the example given in this paper. We used a series of questions, derived from the main topic of our review, to focus an examination of our descriptive themes and we do not find our recommendations for interventions contained in the findings of the primary studies: these were new propositions generated by the reviewers in the light of the synthesis. The method also demonstrates that it is possible to synthesise without conceptual innovation. The initial synthesis, involving the translation of concepts between studies, was necessary in order for conceptual innovation to begin. One could argue that the conceptual innovation, in this case, was only necessary because the primary studies did not address our review question directly. In situations in which the primary studies are concerned directly with the review question, it may not be necessary to go beyond the contents of the original studies in order to produce a satisfactory synthesis (see, for example, Marston and King, [ 60 ]). Conceptually, our analytical themes are similar to the ultimate product of meta-ethnographies: third order interpretations [ 11 ], since both are explicit mechanisms for going beyond the content of the primary studies and presenting this in a transparent way. The main difference between them lies in their purposes. Third order interpretations bring together the implications of translating studies into one another in their own terms, whereas analytical themes are the result of interrogating a descriptive synthesis by placing it within an external theoretical framework (our review question and sub-questions). It may be, therefore, that analytical themes are more appropriate when a specific review question is being addressed (as often occurs when informing policy and practice), and third order interpretations should be used when a body of literature is being explored in and of itself, with broader, or emergent, review questions.

This paper is a contribution to the current developmental work taking place in understanding how best to bring together the findings of qualitative research to inform policy and practice. It is by no means the only method on offer but, by drawing on methods and principles from qualitative primary research, it benefits from the years of methodological development that underpins the research it seeks to synthesise.

Chalmers I: Trying to do more good than harm in policy and practice: the role of rigorous, transparent and up-to-date evaluations. Ann Am Acad Pol Soc Sci. 2003, 589: 22-40. 10.1177/0002716203254762.

Article   Google Scholar  

Oakley A: Social science and evidence-based everything: the case of education. Educ Rev. 2002, 54: 277-286. 10.1080/0013191022000016329.

Cooper H, Hedges L: The Handbook of Research Synthesis. 1994, New York: Russell Sage Foundation

Google Scholar  

EPPI-Centre: EPPI-Centre Methods for Conducting Systematic Reviews. 2006, London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London, [ http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=89 ]

Higgins J, Green S, (Eds): Cochrane Handbook for Systematic Reviews of Interventions 4.2.6. 2006, Updated September 2006. Accessed 24th January 2007, [ http://www.cochrane.org/resources/handbook/ ]

Petticrew M, Roberts H: Systematic Reviews in the Social Sciences: A practical guide. 2006, Oxford: Blackwell Publishing

Book   Google Scholar  

Chalmers I, Hedges L, Cooper H: A brief history of research synthesis. Eval Health Prof. 2002, 25: 12-37. 10.1177/0163278702025001003.

Article   PubMed   Google Scholar  

Juni P, Altman D, Egger M: Assessing the quality of controlled clinical trials. BMJ. 2001, 323: 42-46. 10.1136/bmj.323.7303.42.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Mulrow C: Systematic reviews: rationale for systematic reviews. BMJ. 1994, 309: 597-599.

White H: Scientific communication and literature retrieval. The Handbook of Research Synthesis. Edited by: Cooper H, Hedges L. 1994, New York: Russell Sage Foundation

Campbell R, Pound P, Pope C, Britten N, Pill R, Morgan M, Donovan J: Evaluating meta-ethnography: a synthesis of qualitative research on lay experiences of diabetes and diabetes care. Soc Sci Med. 2003, 56: 671-684. 10.1016/S0277-9536(02)00064-3.

Dixon-Woods M, Bonas S, Booth A, Jones DR, Miller T, Sutton AJ, Shaw RL, Smith JA, Young B: How can systematic reviews incorporate qualitative research? A critical perspective. Qual Res. 2006, 6: 27-44. 10.1177/1468794106058867.

Sandelowski M, Barroso J: Handbook for Synthesising Qualitative Research. 2007, New York: Springer

Thorne S, Jensen L, Kearney MH, Noblit G, Sandelowski M: Qualitative meta-synthesis: reflections on methodological orientation and ideological agenda. Qual Health Res. 2004, 14: 1342-1365. 10.1177/1049732304269888.

Harden A, Garcia J, Oliver S, Rees R, Shepherd J, Brunton G, Oakley A: Applying systematic review methods to studies of people's views: an example from public health. J Epidemiol Community Health. 2004, 58: 794-800. 10.1136/jech.2003.014829.

Article   PubMed   PubMed Central   Google Scholar  

Harden A, Brunton G, Fletcher A, Oakley A: Young People, Pregnancy and Social Exclusion: A systematic synthesis of research evidence to identify effective, appropriate and promising approaches for prevention and support. 2006, London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London, [ http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=674 ]

Thomas J, Sutcliffe K, Harden A, Oakley A, Oliver S, Rees R, Brunton G, Kavanagh J: Children and Healthy Eating: A systematic review of barriers and facilitators. 2003, London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London, accessed 4 th July 2008, [ http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=246 ]

Thomas J, Kavanagh J, Tucker H, Burchett H, Tripney J, Oakley A: Accidental Injury, Risk-Taking Behaviour and the Social Circumstances in which Young People Live: A systematic review. 2007, London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London, [ http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=1910 ]

Bryman A: Quantity and Quality in Social Research. 1998, London: Unwin

Hammersley M: What's Wrong with Ethnography?. 1992, London: Routledge

Harden A, Thomas J: Methodological issues in combining diverse study types in systematic reviews. Int J Soc Res Meth. 2005, 8: 257-271. 10.1080/13645570500155078.

Oakley A: Experiments in Knowing: Gender and methods in the social sciences. 2000, Cambridge: Polity Press

Harden A, Oakley A, Oliver S: Peer-delivered health promotion for young people: a systematic review of different study designs. Health Educ J. 2001, 60: 339-353. 10.1177/001789690106000406.

Harden A, Rees R, Shepherd J, Brunton G, Oliver S, Oakley A: Young People and Mental Health: A systematic review of barriers and facilitators. 2001, London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London, [ http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=256 ]

Rees R, Harden A, Shepherd J, Brunton G, Oliver S, Oakley A: Young People and Physical Activity: A systematic review of barriers and facilitators. 2001, London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London, [ http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=260 ]

Shepherd J, Harden A, Rees R, Brunton G, Oliver S, Oakley A: Young People and Healthy Eating: A systematic review of barriers and facilitators. 2001, London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London, [ http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=258 ]

Thomas J, Harden A, Oakley A, Oliver S, Sutcliffe K, Rees R, Brunton G, Kavanagh J: Integrating qualitative research with trials in systematic reviews: an example from public health. BMJ. 2004, 328: 1010-1012. 10.1136/bmj.328.7446.1010.

Davies P: What is evidence-based education?. Br J Educ Stud. 1999, 47: 108-121. 10.1111/1467-8527.00106.

Newman M, Thompson C, Roberts AP: Helping practitioners understand the contribution of qualitative research to evidence-based practice. Evid Based Nurs. 2006, 9: 4-7. 10.1136/ebn.9.1.4.

Popay J: Moving Beyond Effectiveness in Evidence Synthesis. 2006, London: National Institute for Health and Clinical Excellence

Noblit GW, Hare RD: Meta-Ethnography: Synthesizing qualitative studies. 1988, Newbury Park: Sage

Britten N, Campbell R, Pope C, Donovan J, Morgan M, Pill R: Using meta-ethnography to synthesise qualitative research: a worked example. J Health Serv Res Policy. 2002, 7: 209-215. 10.1258/135581902320432732.

Paterson B, Thorne S, Canam C, Jillings C: Meta-Study of Qualitative Health Research. 2001, Thousand Oaks, California: Sage

Dixon-Woods M, Cavers D, Agarwal S, Annandale E, Arthur A, Harvey J, Katbamna S, Olsen R, Smith L, Riley R, Sutton AJ: Conducting a critical interpretative synthesis of the literature on access to healthcare by vulnerable groups. BMC Med Res Methodol. 2006, 6: 35-10.1186/1471-2288-6-35.

Dixon-Woods M, Agarwal S, Jones D, Young B, Sutton A: Synthesising qualitative and quantitative evidence: a review of possible methods. J Health Serv Res Policy. 2005, 10: 45-53. 10.1258/1355819052801804.

Boyatzis RE: Transforming Qualitative Information. 1998, Sage: Cleveland

Braun V, Clarke V: Using thematic analysis in psychology. Qual Res Psychol. 2006, 3: 77-101. 10.1191/1478088706qp063oa. [ http://science.uwe.ac.uk/psychology/drvictoriaclarke_files/thematicanalysis%20.pdf ]

Silverman D, Ed: Qualitative Research: Theory, method and practice. 1997, London: Sage

Doyle LH: Synthesis through meta-ethnography: paradoxes, enhancements, and possibilities. Qual Res. 2003, 3: 321-344. 10.1177/1468794103033003.

Barroso J, Gollop C, Sandelowski M, Meynell J, Pearce PF, Collins LJ: The challenges of searching for and retrieving qualitative studies. Western J Nurs Res. 2003, 25: 153-178. 10.1177/0193945902250034.

Walters LA, Wilczynski NL, Haynes RB, Hedges Team: Developing optimal search strategies for retrieving clinically relevant qualitative studies in EMBASE. Qual Health Res. 2006, 16: 162-8. 10.1177/1049732305284027.

Wong SSL, Wilczynski NL, Haynes RB: Developing optimal search strategies for detecting clinically relevant qualitative studies in Medline. Medinfo. 2004, 11: 311-314.

Murphy E, Dingwall R, Greatbatch D, Parker S, Watson P: Qualitative research methods in health technology assessment: a review of the literature. Health Technol Assess. 1998, 2 (16):

Seale C: Quality in qualitative research. Qual Inq. 1999, 5: 465-478.

Spencer L, Ritchie J, Lewis J, Dillon L: Quality in Qualitative Evaluation: A framework for assessing research evidence. 2003, London: Cabinet Office

Boulton M, Fitzpatrick R, Swinburn C: Qualitative research in healthcare II: a structured review and evaluation of studies. J Eval Clin Pract. 1996, 2: 171-179. 10.1111/j.1365-2753.1996.tb00041.x.

Article   CAS   PubMed   Google Scholar  

Cobb A, Hagemaster J: Ten criteria for evaluating qualitative research proposals. J Nurs Educ. 1987, 26: 138-143.

CAS   PubMed   Google Scholar  

Mays N, Pope C: Rigour and qualitative research. BMJ. 1995, 311: 109-12.

Medical Sociology Group: Criteria for the evaluation of qualitative research papers. Med Sociol News. 1996, 22: 68-71.

Alderson P: Listening to Children. 1995, London: Barnardo's

Egger M, Davey-Smith G, Altman D: Systematic Reviews in Health Care: Meta-analysis in context. 2001, London: BMJ Publishing

Sandelowski M, Barroso J: Finding the findings in qualitative studies. J Nurs Scholarsh. 2002, 34: 213-219. 10.1111/j.1547-5069.2002.00213.x.

Sandelowski M: Using qualitative research. Qual Health Res. 2004, 14: 1366-1386. 10.1177/1049732304269672.

Thomas J, Brunton J: EPPI-Reviewer 3.0: Analysis and management of data for research synthesis. EPPI-Centre software. 2006, London: EPPI-Centre, Social Science Research Unit, Institute of Education

Fisher M, Qureshi H, Hardyman W, Homewood J: Using Qualitative Research in Systematic Reviews: Older people's views of hospital discharge. 2006, London: Social Care Institute for Excellence

Dixey R, Sahota P, Atwal S, Turner A: Children talking about healthy eating: data from focus groups with 300 9–11-year-olds. Nutr Bull. 2001, 26: 71-79. 10.1046/j.1467-3010.2001.00078.x.

Daly A, Willis K, Small R, Green J, Welch N, Kealy M, Hughes E: Hierarchy of evidence for assessing qualitative health research. J Clin Epidemiol. 2007, 60: 43-49. 10.1016/j.jclinepi.2006.03.014.

Popay J: Moving beyond floccinaucinihilipilification: enhancing the utility of systematic reviews. J Clin Epidemiol. 2005, 58: 1079-80. 10.1016/j.jclinepi.2005.08.004.

Strike K, Posner G: Types of synthesis and their criteria. Knowledge Structure and Use: Implications for synthesis and interpretation. Edited by: Ward S, Reed L. 1983, Philadelphia: Temple University Press

Marston C, King E: Factors that shape young people's sexual behaviour: a systematic review. The Lancet. 2006, 368: 1581-86. 10.1016/S0140-6736(06)69662-1.

Pre-publication history

The pre-publication history for this paper can be accessed here: http://www.biomedcentral.com/1471-2288/8/45/prepub

Download references

Acknowledgements

The authors would like to thank Elaine Barnett-Page for her assistance in producing the draft paper, and David Gough, Ann Oakley and Sandy Oliver for their helpful comments. The review used an example in this paper was funded by the Department of Health (England). The methodological development was supported by Department of Health (England) and the ESRC through the Methods for Research Synthesis Node of the National Centre for Research Methods. In addition, Angela Harden held a senior research fellowship funded by the Department of Health (England) December 2003 – November 2007. The views expressed in this paper are those of the authors and are not necessarily those of the funding bodies.

Author information

Authors and affiliations.

EPPI-Centre, Social Science Research Unit, Institute of Education, University of London, UK

James Thomas & Angela Harden

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to James Thomas .

Additional information

Competing interests.

The authors declare that they have no competing interests.

Authors' contributions

Both authors contributed equally to the paper and read and approved the final manuscript.

James Thomas and Angela Harden contributed equally to this work.

Authors’ original submitted files for images

Below are the links to the authors’ original submitted files for images.

Authors’ original file for figure 1

Authors’ original file for figure 2, rights and permissions.

This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License ( http://creativecommons.org/licenses/by/2.0 ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and permissions

About this article

Cite this article.

Thomas, J., Harden, A. Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Med Res Methodol 8 , 45 (2008). https://doi.org/10.1186/1471-2288-8-45

Download citation

Received : 17 April 2008

Accepted : 10 July 2008

Published : 10 July 2008

DOI : https://doi.org/10.1186/1471-2288-8-45

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Qualitative Research
• Primary Study
• Analytical Theme
• Healthy Eating
• Review Question

BMC Medical Research Methodology

ISSN: 1471-2288

• General enquiries: [email protected]

• - Google Chrome

Intended for healthcare professionals

• My email alerts
• BMA member login
• Username * Password * Forgot your log in details? Need to activate BMA Member Log In Log in via OpenAthens Log in via your institution

Search form

• Advanced search
• Search responses
• Search blogs
• Practical thematic...

Practical thematic analysis: a guide for multidisciplinary health services research teams engaging in qualitative analysis

• Related content
• Peer review
• on behalf of the Coproduction Laboratory
• 1 Dartmouth Health, Lebanon, NH, USA
• 2 Dartmouth Institute for Health Policy and Clinical Practice, Geisel School of Medicine at Dartmouth College, Lebanon, NH, USA
• 3 Center for Primary Care and Public Health (Unisanté), Lausanne, Switzerland
• 4 Jönköping Academy for Improvement of Health and Welfare, School of Health and Welfare, Jönköping University, Jönköping, Sweden
• 5 Highland Park, NJ, USA
• 6 Division of Public Health Sciences, Department of Surgery, Washington University School of Medicine, St Louis, MO, USA
• Correspondence to: C H Saunders catherine.hylas.saunders{at}dartmouth.edu
• Accepted 26 April 2023

Qualitative research methods explore and provide deep contextual understanding of real world issues, including people’s beliefs, perspectives, and experiences. Whether through analysis of interviews, focus groups, structured observation, or multimedia data, qualitative methods offer unique insights in applied health services research that other approaches cannot deliver. However, many clinicians and researchers hesitate to use these methods, or might not use them effectively, which can leave relevant areas of inquiry inadequately explored. Thematic analysis is one of the most common and flexible methods to examine qualitative data collected in health services research. This article offers practical thematic analysis as a step-by-step approach to qualitative analysis for health services researchers, with a focus on accessibility for patients, care partners, clinicians, and others new to thematic analysis. Along with detailed instructions covering three steps of reading, coding, and theming, the article includes additional novel and practical guidance on how to draft effective codes, conduct a thematic analysis session, and develop meaningful themes. This approach aims to improve consistency and rigor in thematic analysis, while also making this method more accessible for multidisciplinary research teams.

Through qualitative methods, researchers can provide deep contextual understanding of real world issues, and generate new knowledge to inform hypotheses, theories, research, and clinical care. Approaches to data collection are varied, including interviews, focus groups, structured observation, and analysis of multimedia data, with qualitative research questions aimed at understanding the how and why of human experience. 1 2 Qualitative methods produce unique insights in applied health services research that other approaches cannot deliver. In particular, researchers acknowledge that thematic analysis is a flexible and powerful method of systematically generating robust qualitative research findings by identifying, analysing, and reporting patterns (themes) within data. 3 4 5 6 Although qualitative methods are increasingly valued for answering clinical research questions, many researchers are unsure how to apply them or consider them too time consuming to be useful in responding to practical challenges 7 or pressing situations such as public health emergencies. 8 Consequently, researchers might hesitate to use them, or use them improperly. 9 10 11

Although much has been written about how to perform thematic analysis, practical guidance for non-specialists is sparse. 3 5 6 12 13 In the multidisciplinary field of health services research, qualitative data analysis can confound experienced researchers and novices alike, which can stoke concerns about rigor, particularly for those more familiar with quantitative approaches. 14 Since qualitative methods are an area of specialisation, support from experts is beneficial. However, because non-specialist perspectives can enhance data interpretation and enrich findings, there is a case for making thematic analysis easier, more rapid, and more efficient, 8 particularly for patients, care partners, clinicians, and other stakeholders. A practical guide to thematic analysis might encourage those on the ground to use these methods in their work, unearthing insights that would otherwise remain undiscovered.

Given the need for more accessible qualitative analysis approaches, we present a simple, rigorous, and efficient three step guide for practical thematic analysis. We include new guidance on the mechanics of thematic analysis, including developing codes, constructing meaningful themes, and hosting a thematic analysis session. We also discuss common pitfalls in thematic analysis and how to avoid them.

Summary points

Qualitative methods are increasingly valued in applied health services research, but multidisciplinary research teams often lack accessible step-by-step guidance and might struggle to use these approaches

A newly developed approach, practical thematic analysis, uses three simple steps: reading, coding, and theming

Based on Braun and Clarke’s reflexive thematic analysis, our streamlined yet rigorous approach is designed for multidisciplinary health services research teams, including patients, care partners, and clinicians

This article also provides companion materials including a slide presentation for teaching practical thematic analysis to research teams, a sample thematic analysis session agenda, a theme coproduction template for use during the session, and guidance on using standardised reporting criteria for qualitative research

In their seminal work, Braun and Clarke developed a six phase approach to reflexive thematic analysis. 4 12 We built on their method to develop practical thematic analysis ( box 1 , fig 1 ), which is a simplified and instructive approach that retains the substantive elements of their six phases. Braun and Clarke’s phase 1 (familiarising yourself with the dataset) is represented in our first step of reading. Phase 2 (coding) remains as our second step of coding. Phases 3 (generating initial themes), 4 (developing and reviewing themes), and 5 (refining, defining, and naming themes) are represented in our third step of theming. Phase 6 (writing up) also occurs during this third step of theming, but after a thematic analysis session. 4 12

Key features and applications of practical thematic analysis

Step 1: reading.

All manuscript authors read the data

All manuscript authors write summary memos

Step 2: Coding

Coders perform both data management and early data analysis

Codes are complete thoughts or sentences, not categories

Step 3: Theming

Researchers host a thematic analysis session and share different perspectives

Themes are complete thoughts or sentences, not categories

Applications

For use by practicing clinicians, patients and care partners, students, interdisciplinary teams, and those new to qualitative research

When important insights from healthcare professionals are inaccessible because they do not have qualitative methods training

When time and resources are limited

Steps in practical thematic analysis

• Download figure
• Open in new tab
• Download powerpoint

We present linear steps, but as qualitative research is usually iterative, so too is thematic analysis. 15 Qualitative researchers circle back to earlier work to check whether their interpretations still make sense in the light of additional insights, adapting as necessary. While we focus here on the practical application of thematic analysis in health services research, we recognise our approach exists in the context of the broader literature on thematic analysis and the theoretical underpinnings of qualitative methods as a whole. For a more detailed discussion of these theoretical points, as well as other methods widely used in health services research, we recommend reviewing the sources outlined in supplemental material 1. A strong and nuanced understanding of the context and underlying principles of thematic analysis will allow for higher quality research. 16

Practical thematic analysis is a highly flexible approach that can draw out valuable findings and generate new hypotheses, including in cases with a lack of previous research to build on. The approach can also be used with a variety of data, such as transcripts from interviews or focus groups, patient encounter transcripts, professional publications, observational field notes, and online activity logs. Importantly, successful practical thematic analysis is predicated on having high quality data collected with rigorous methods. We do not describe qualitative research design or data collection here. 11 17

In supplemental material 1, we summarise the foundational methods, concepts, and terminology in qualitative research. Along with our guide below, we include a companion slide presentation for teaching practical thematic analysis to research teams in supplemental material 2. We provide a theme coproduction template for teams to use during thematic analysis sessions in supplemental material 3. Our method aligns with the major qualitative reporting frameworks, including the Consolidated Criteria for Reporting Qualitative Research (COREQ). 18 We indicate the corresponding step in practical thematic analysis for each COREQ item in supplemental material 4.

Familiarisation and memoing

We encourage all manuscript authors to review the full dataset (eg, interview transcripts) to familiarise themselves with it. This task is most critical for those who will later be engaged in the coding and theming steps. Although time consuming, it is the best way to involve team members in the intellectual work of data interpretation, so that they can contribute to the analysis and contextualise the results. If this task is not feasible given time limitations or large quantities of data, the data can be divided across team members. In this case, each piece of data should be read by at least two individuals who ideally represent different professional roles or perspectives.

We recommend that researchers reflect on the data and independently write memos, defined as brief notes on thoughts and questions that arise during reading, and a summary of their impressions of the dataset. 2 19 Memoing is an opportunity to gain insights from varying perspectives, particularly from patients, care partners, clinicians, and others. It also gives researchers the opportunity to begin to scope which elements of and concepts in the dataset are relevant to the research question.

Data saturation

The concept of data saturation ( box 2 ) is a foundation of qualitative research. It is defined as the point in analysis at which new data tend to be redundant of data already collected. 21 Qualitative researchers are expected to report their approach to data saturation. 18 Because thematic analysis is iterative, the team should discuss saturation throughout the entire process, beginning with data collection and continuing through all steps of the analysis. 22 During step 1 (reading), team members might discuss data saturation in the context of summary memos. Conversations about saturation continue during step 2 (coding), with confirmation that saturation has been achieved during step 3 (theming). As a rule of thumb, researchers can often achieve saturation in 9-17 interviews or 4-8 focus groups, but this will vary depending on the specific characteristics of the study. 23

Data saturation in context

Braun and Clarke discourage the use of data saturation to determine sample size (eg, number of interviews), because it assumes that there is an objective truth to be captured in the data (sometimes known as a positivist perspective). 20 Qualitative researchers often try to avoid positivist approaches, arguing that there is no one true way of seeing the world, and will instead aim to gather multiple perspectives. 5 Although this theoretical debate with qualitative methods is important, we recognise that a priori estimates of saturation are often needed, particularly for investigators newer to qualitative research who might want a more pragmatic and applied approach. In addition, saturation based, sample size estimation can be particularly helpful in grant proposals. However, researchers should still follow a priori sample size estimation with a discussion to confirm saturation has been achieved.

Definition of coding

We describe codes as labels for concepts in the data that are directly relevant to the study objective. Historically, the purpose of coding was to distil the large amount of data collected into conceptually similar buckets so that researchers could review it in aggregate and identify key themes. 5 24 We advocate for a more analytical approach than is typical with thematic analysis. With our method, coding is both the foundation for and the beginning of thematic analysis—that is, early data analysis, management, and reduction occur simultaneously rather than as different steps. This approach moves the team more efficiently towards being able to describe themes.

Building the coding team

Coders are the research team members who directly assign codes to the data, reading all material and systematically labelling relevant data with appropriate codes. Ideally, at least two researchers would code every discrete data document, such as one interview transcript. 25 If this task is not possible, individual coders can each code a subset of the data that is carefully selected for key characteristics (sometimes known as purposive selection). 26 When using this approach, we recommend that at least 10% of data be coded by two or more coders to ensure consistency in codebook application. We also recommend coding teams of no more than four to five people, for practical reasons concerning maintaining consistency.

Clinicians, patients, and care partners bring unique perspectives to coding and enrich the analytical process. 27 Therefore, we recommend choosing coders with a mix of relevant experiences so that they can challenge and contextualise each other’s interpretations based on their own perspectives and opinions ( box 3 ). We recommend including both coders who collected the data and those who are naive to it, if possible, given their different perspectives. We also recommend all coders review the summary memos from the reading step so that key concepts identified by those not involved in coding can be integrated into the analytical process. In practice, this review means coding the memos themselves and discussing them during the code development process. This approach ensures that the team considers a diversity of perspectives.

Coding teams in context

The recommendation to use multiple coders is a departure from Braun and Clarke. 28 29 When the views, experiences, and training of each coder (sometimes known as positionality) 30 are carefully considered, having multiple coders can enhance interpretation and enrich findings. When these perspectives are combined in a team setting, researchers can create shared meaning from the data. Along with the practical consideration of distributing the workload, 31 inclusion of these multiple perspectives increases the overall quality of the analysis by mitigating the impact of any one coder’s perspective. 30

Coding tools

Qualitative analysis software facilitates coding and managing large datasets but does not perform the analytical work. The researchers must perform the analysis themselves. Most programs support queries and collaborative coding by multiple users. 32 Important factors to consider when choosing software can include accessibility, cost, interoperability, the look and feel of code reports, and the ease of colour coding and merging codes. Coders can also use low tech solutions, including highlighters, word processors, or spreadsheets.

Drafting effective codes

To draft effective codes, we recommend that the coders review each document line by line. 33 As they progress, they can assign codes to segments of data representing passages of interest. 34 Coders can also assign multiple codes to the same passage. Consensus among coders on what constitutes a minimum or maximum amount of text for assigning a code is helpful. As a general rule, meaningful segments of text for coding are shorter than one paragraph, but longer than a few words. Coders should keep the study objective in mind when determining which data are relevant ( box 4 ).

Code types in context

Similar to Braun and Clarke’s approach, practical thematic analysis does not specify whether codes are based on what is evident from the data (sometimes known as semantic) or whether they are based on what can be inferred at a deeper level from the data (sometimes known as latent). 4 12 35 It also does not specify whether they are derived from the data (sometimes known as inductive) or determined ahead of time (sometimes known as deductive). 11 35 Instead, it should be noted that health services researchers conducting qualitative studies often adopt all these approaches to coding (sometimes known as hybrid analysis). 3

In practical thematic analysis, codes should be more descriptive than general categorical labels that simply group data with shared characteristics. At a minimum, codes should form a complete (or full) thought. An easy way to conceptualise full thought codes is as complete sentences with subjects and verbs ( table 1 ), although full sentence coding is not always necessary. With full thought codes, researchers think about the data more deeply and capture this insight in the codes. This coding facilitates the entire analytical process and is especially valuable when moving from codes to broader themes. Experienced qualitative researchers often intuitively use full thought or sentence codes, but this practice has not been explicitly articulated as a path to higher quality coding elsewhere in the literature. 6

Example transcript with codes used in practical thematic analysis 36

• View inline

Depending on the nature of the data, codes might either fall into flat categories or be arranged hierarchically. Flat categories are most common when the data deal with topics on the same conceptual level. In other words, one topic is not a subset of another topic. By contrast, hierarchical codes are more appropriate for concepts that naturally fall above or below each other. Hierarchical coding can also be a useful form of data management and might be necessary when working with a large or complex dataset. 5 Codes grouped into these categories can also make it easier to naturally transition into generating themes from the initial codes. 5 These decisions between flat versus hierarchical coding are part of the work of the coding team. In both cases, coders should ensure that their code structures are guided by their research questions.

Developing the codebook

A codebook is a shared document that lists code labels and comprehensive descriptions for each code, as well as examples observed within the data. Good code descriptions are precise and specific so that coders can consistently assign the same codes to relevant data or articulate why another coder would do so. Codebook development is iterative and involves input from the entire coding team. However, as those closest to the data, coders must resist undue influence, real or perceived, from other team members with conflicting opinions—it is important to mitigate the risk that more senior researchers, like principal investigators, exert undue influence on the coders’ perspectives.

In practical thematic analysis, coders begin codebook development by independently coding a small portion of the data, such as two to three transcripts or other units of analysis. Coders then individually produce their initial codebooks. This task will require them to reflect on, organise, and clarify codes. The coders then meet to reconcile the draft codebooks, which can often be difficult, as some coders tend to lump several concepts together while others will split them into more specific codes. Discussing disagreements and negotiating consensus are necessary parts of early data analysis. Once the codebook is relatively stable, we recommend soliciting input on the codes from all manuscript authors. Yet, coders must ultimately be empowered to finalise the details so that they are comfortable working with the codebook across a large quantity of data.

Assigning codes to the data

After developing the codebook, coders will use it to assign codes to the remaining data. While the codebook’s overall structure should remain constant, coders might continue to add codes corresponding to any new concepts observed in the data. If new codes are added, coders should review the data they have already coded and determine whether the new codes apply. Qualitative data analysis software can be useful for editing or merging codes.

We recommend that coders periodically compare their code occurrences ( box 5 ), with more frequent check-ins if substantial disagreements occur. In the event of large discrepancies in the codes assigned, coders should revise the codebook to ensure that code descriptions are sufficiently clear and comprehensive to support coding alignment going forward. Because coding is an iterative process, the team can adjust the codebook as needed. 5 28 29

Quantitative coding in context

Researchers should generally avoid reporting code counts in thematic analysis. However, counts can be a useful proxy in maintaining alignment between coders on key concepts. 26 In practice, therefore, researchers should make sure that all coders working on the same piece of data assign the same codes with a similar pattern and that their memoing and overall assessment of the data are aligned. 37 However, the frequency of a code alone is not an indicator of its importance. It is more important that coders agree on the most salient points in the data; reviewing and discussing summary memos can be helpful here. 5

Researchers might disagree on whether or not to calculate and report inter-rater reliability. We note that quantitative tests for agreement, such as kappa statistics or intraclass correlation coefficients, can be distracting and might not provide meaningful results in qualitative analyses. Similarly, Braun and Clarke argue that expecting perfect alignment on coding is inconsistent with the goal of co-constructing meaning. 28 29 Overall consensus on codes’ salience and contributions to themes is the most important factor.

Definition of themes

Themes are meta-constructs that rise above codes and unite the dataset ( box 6 , fig 2 ). They should be clearly evident, repeated throughout the dataset, and relevant to the research questions. 38 While codes are often explicit descriptions of the content in the dataset, themes are usually more conceptual and knit the codes together. 39 Some researchers hypothesise that theme development is loosely described in the literature because qualitative researchers simply intuit themes during the analytical process. 39 In practical thematic analysis, we offer a concrete process that should make developing meaningful themes straightforward.

Themes in context

According to Braun and Clarke, a theme “captures something important about the data in relation to the research question and represents some level of patterned response or meaning within the data set.” 4 Similarly, Braun and Clarke advise against themes as domain summaries. While different approaches can draw out themes from codes, the process begins by identifying patterns. 28 35 Like Braun and Clarke and others, we recommend that researchers consider the salience of certain themes, their prevalence in the dataset, and their keyness (ie, how relevant the themes are to the overarching research questions). 4 12 34

Use of themes in practical thematic analysis

Constructing meaningful themes

After coding all the data, each coder should independently reflect on the team’s summary memos (step 1), the codebook (step 2), and the coded data itself to develop draft themes (step 3). It can be illuminating for coders to review all excerpts associated with each code, so that they derive themes directly from the data. Researchers should remain focused on the research question during this step, so that themes have a clear relation with the overall project aim. Use of qualitative analysis software will make it easy to view each segment of data tagged with each code. Themes might neatly correspond to groups of codes. Or—more likely—they will unite codes and data in unexpected ways. A whiteboard or presentation slides might be helpful to organise, craft, and revise themes. We also provide a template for coproducing themes (supplemental material 3). As with codebook justification, team members will ideally produce individual drafts of the themes that they have identified in the data. They can then discuss these with the group and reach alignment or consensus on the final themes.

The team should ensure that all themes are salient, meaning that they are: supported by the data, relevant to the study objectives, and important. Similar to codes, themes are framed as complete thoughts or sentences, not categories. While codes and themes might appear to be similar to each other, the key distinction is that the themes represent a broader concept. Table 2 shows examples of codes and their corresponding themes from a previously published project that used practical thematic analysis. 36 Identifying three to four key themes that comprise a broader overarching theme is a useful approach. Themes can also have subthemes, if appropriate. 40 41 42 43 44

Example codes with themes in practical thematic analysis 36

Thematic analysis session

After each coder has independently produced draft themes, a carefully selected subset of the manuscript team meets for a thematic analysis session ( table 3 ). The purpose of this session is to discuss and reach alignment or consensus on the final themes. We recommend a session of three to five hours, either in-person or virtually.

Example agenda of thematic analysis session

The composition of the thematic analysis session team is important, as each person’s perspectives will shape the results. This group is usually a small subset of the broader research team, with three to seven individuals. We recommend that primary and senior authors work together to include people with diverse experiences related to the research topic. They should aim for a range of personalities and professional identities, particularly those of clinicians, trainees, patients, and care partners. At a minimum, all coders and primary and senior authors should participate in the thematic analysis session.

The session begins with each coder presenting their draft themes with supporting quotes from the data. 5 Through respectful and collaborative deliberation, the group will develop a shared set of final themes.

One team member facilitates the session. A firm, confident, and consistent facilitation style with good listening skills is critical. For practical reasons, this person is not usually one of the primary coders. Hierarchies in teams cannot be entirely flattened, but acknowledging them and appointing an external facilitator can reduce their impact. The facilitator can ensure that all voices are heard. For example, they might ask for perspectives from patient partners or more junior researchers, and follow up on comments from senior researchers to say, “We have heard your perspective and it is important; we want to make sure all perspectives in the room are equally considered.” Or, “I hear [senior person] is offering [x] idea, I’d like to hear other perspectives in the room.” The role of the facilitator is critical in the thematic analysis session. The facilitator might also privately discuss with more senior researchers, such as principal investigators and senior authors, the importance of being aware of their influence over others and respecting and eliciting the perspectives of more junior researchers, such as patients, care partners, and students.

To our knowledge, this discrete thematic analysis session is a novel contribution of practical thematic analysis. It helps efficiently incorporate diverse perspectives using the session agenda and theme coproduction template (supplemental material 3) and makes the process of constructing themes transparent to the entire research team.

Writing the report

We recommend beginning the results narrative with a summary of all relevant themes emerging from the analysis, followed by a subheading for each theme. Each subsection begins with a brief description of the theme and is illustrated with relevant quotes, which are contextualised and explained. The write-up should not simply be a list, but should contain meaningful analysis and insight from the researchers, including descriptions of how different stakeholders might have experienced a particular situation differently or unexpectedly.

In addition to weaving quotes into the results narrative, quotes can be presented in a table. This strategy is a particularly helpful when submitting to clinical journals with tight word count limitations. Quote tables might also be effective in illustrating areas of agreement and disagreement across stakeholder groups, with columns representing different groups and rows representing each theme or subtheme. Quotes should include an anonymous label for each participant and any relevant characteristics, such as role or gender. The aim is to produce rich descriptions. 5 We recommend against repeating quotations across multiple themes in the report, so as to avoid confusion. The template for coproducing themes (supplemental material 3) allows documentation of quotes supporting each theme, which might also be useful during report writing.

Visual illustrations such as a thematic map or figure of the findings can help communicate themes efficiently. 4 36 42 44 If a figure is not possible, a simple list can suffice. 36 Both must clearly present the main themes with subthemes. Thematic figures can facilitate confirmation that the researchers’ interpretations reflect the study populations’ perspectives (sometimes known as member checking), because authors can invite discussions about the figure and descriptions of findings and supporting quotes. 46 This process can enhance the validity of the results. 46

In supplemental material 4, we provide additional guidance on reporting thematic analysis consistent with COREQ. 18 Commonly used in health services research, COREQ outlines a standardised list of items to be included in qualitative research reports ( box 7 ).

Reporting in context

We note that use of COREQ or any other reporting guidelines does not in itself produce high quality work and should not be used as a substitute for general methodological rigor. Rather, researchers must consider rigor throughout the entire research process. As the issue of how to conceptualise and achieve rigorous qualitative research continues to be debated, 47 48 we encourage researchers to explicitly discuss how they have looked at methodological rigor in their reports. Specifically, we point researchers to Braun and Clarke’s 2021 tool for evaluating thematic analysis manuscripts for publication (“Twenty questions to guide assessment of TA [thematic analysis] research quality”). 16

Avoiding common pitfalls

Awareness of common mistakes can help researchers avoid improper use of qualitative methods. Improper use can, for example, prevent researchers from developing meaningful themes and can risk drawing inappropriate conclusions from the data. Braun and Clarke also warn of poor quality in qualitative research, noting that “coherence and integrity of published research does not always hold.” 16

Weak themes

An important distinction between high and low quality themes is that high quality themes are descriptive and complete thoughts. As such, they often contain subjects and verbs, and can be expressed as full sentences ( table 2 ). Themes that are simply descriptive categories or topics could fail to impart meaningful knowledge beyond categorisation. 16 49 50

Researchers will often move from coding directly to writing up themes, without performing the work of theming or hosting a thematic analysis session. Skipping concerted theming often results in themes that look more like categories than unifying threads across the data.

Unfocused analysis

Because data collection for qualitative research is often semi-structured (eg, interviews, focus groups), not all data will be directly relevant to the research question at hand. To avoid unfocused analysis and a correspondingly unfocused manuscript, we recommend that all team members keep the research objective in front of them at every stage, from reading to coding to theming. During the thematic analysis session, we recommend that the research question be written on a whiteboard so that all team members can refer back to it, and so that the facilitator can ensure that conversations about themes occur in the context of this question. Consistently focusing on the research question can help to ensure that the final report directly answers it, as opposed to the many other interesting insights that might emerge during the qualitative research process. Such insights can be picked up in a secondary analysis if desired.

Inappropriate quantification

Presenting findings quantitatively (eg, “We found 18 instances of participants mentioning safety concerns about the vaccines”) is generally undesirable in practical thematic analysis reporting. 51 Descriptive terms are more appropriate (eg, “participants had substantial concerns about the vaccines,” or “several participants were concerned about this”). This descriptive presentation is critical because qualitative data might not be consistently elicited across participants, meaning that some individuals might share certain information while others do not, simply based on how conversations evolve. Additionally, qualitative research does not aim to draw inferences outside its specific sample. Emphasising numbers in thematic analysis can lead to readers incorrectly generalising the findings. Although peer reviewers unfamiliar with thematic analysis often request this type of quantification, practitioners of practical thematic analysis can confidently defend their decision to avoid it. If quantification is methodologically important, we recommend simultaneously conducting a survey or incorporating standardised interview techniques into the interview guide. 11

Neglecting group dynamics

Researchers should concertedly consider group dynamics in the research team. Particular attention should be paid to power relations and the personality of team members, which can include aspects such as who most often speaks, who defines concepts, and who resolves disagreements that might arise within the group. 52

The perspectives of patient and care partners are particularly important to cultivate. Ideally, patient partners are meaningfully embedded in studies from start to finish, not just for practical thematic analysis. 53 Meaningful engagement can build trust, which makes it easier for patient partners to ask questions, request clarification, and share their perspectives. Professional team members should actively encourage patient partners by emphasising that their expertise is critically important and valued. Noting when a patient partner might be best positioned to offer their perspective can be particularly powerful.

Insufficient time allocation

Researchers must allocate enough time to complete thematic analysis. Working with qualitative data takes time, especially because it is often not a linear process. As the strength of thematic analysis lies in its ability to make use of the rich details and complexities of the data, we recommend careful planning for the time required to read and code each document.

Estimating the necessary time can be challenging. For step 1 (reading), researchers can roughly calculate the time required based on the time needed to read and reflect on one piece of data. For step 2 (coding), the total amount of time needed can be extrapolated from the time needed to code one document during codebook development. We also recommend three to five hours for the thematic analysis session itself, although coders will need to independently develop their draft themes beforehand. Although the time required for practical thematic analysis is variable, teams should be able to estimate their own required effort with these guidelines.

Practical thematic analysis builds on the foundational work of Braun and Clarke. 4 16 We have reframed their six phase process into three condensed steps of reading, coding, and theming. While we have maintained important elements of Braun and Clarke’s reflexive thematic analysis, we believe that practical thematic analysis is conceptually simpler and easier to teach to less experienced researchers and non-researcher stakeholders. For teams with different levels of familiarity with qualitative methods, this approach presents a clear roadmap to the reading, coding, and theming of qualitative data. Our practical thematic analysis approach promotes efficient learning by doing—experiential learning. 12 29 Practical thematic analysis avoids the risk of relying on complex descriptions of methods and theory and places more emphasis on obtaining meaningful insights from those close to real world clinical environments. Although practical thematic analysis can be used to perform intensive theory based analyses, it lends itself more readily to accelerated, pragmatic approaches.

Strengths and limitations

Our approach is designed to smooth the qualitative analysis process and yield high quality themes. Yet, researchers should note that poorly performed analyses will still produce low quality results. Practical thematic analysis is a qualitative analytical approach; it does not look at study design, data collection, or other important elements of qualitative research. It also might not be the right choice for every qualitative research project. We recommend it for applied health services research questions, where diverse perspectives and simplicity might be valuable.

We also urge researchers to improve internal validity through triangulation methods, such as member checking (supplemental material 1). 46 Member checking could include soliciting input on high level themes, theme definitions, and quotations from participants. This approach might increase rigor.

Implications

We hope that by providing clear and simple instructions for practical thematic analysis, a broader range of researchers will be more inclined to use these methods. Increased transparency and familiarity with qualitative approaches can enhance researchers’ ability to both interpret qualitative studies and offer up new findings themselves. In addition, it can have usefulness in training and reporting. A major strength of this approach is to facilitate meaningful inclusion of patient and care partner perspectives, because their lived experiences can be particularly valuable in data interpretation and the resulting findings. 11 30 As clinicians are especially pressed for time, they might also appreciate a practical set of instructions that can be immediately used to leverage their insights and access to patients and clinical settings, and increase the impact of qualitative research through timely results. 8

Practical thematic analysis is a simplified approach to performing thematic analysis in health services research, a field where the experiences of patients, care partners, and clinicians are of inherent interest. We hope that it will be accessible to those individuals new to qualitative methods, including patients, care partners, clinicians, and other health services researchers. We intend to empower multidisciplinary research teams to explore unanswered questions and make new, important, and rigorous contributions to our understanding of important clinical and health systems research.

Acknowledgments

All members of the Coproduction Laboratory provided input that shaped this manuscript during laboratory meetings. We acknowledge advice from Elizabeth Carpenter-Song, an expert in qualitative methods.

Coproduction Laboratory group contributors: Stephanie C Acquilano ( http://orcid.org/0000-0002-1215-5531 ), Julie Doherty ( http://orcid.org/0000-0002-5279-6536 ), Rachel C Forcino ( http://orcid.org/0000-0001-9938-4830 ), Tina Foster ( http://orcid.org/0000-0001-6239-4031 ), Megan Holthoff, Christopher R Jacobs ( http://orcid.org/0000-0001-5324-8657 ), Lisa C Johnson ( http://orcid.org/0000-0001-7448-4931 ), Elaine T Kiriakopoulos, Kathryn Kirkland ( http://orcid.org/0000-0002-9851-926X ), Meredith A MacMartin ( http://orcid.org/0000-0002-6614-6091 ), Emily A Morgan, Eugene Nelson, Elizabeth O’Donnell, Brant Oliver ( http://orcid.org/0000-0002-7399-622X ), Danielle Schubbe ( http://orcid.org/0000-0002-9858-1805 ), Gabrielle Stevens ( http://orcid.org/0000-0001-9001-178X ), Rachael P Thomeer ( http://orcid.org/0000-0002-5974-3840 ).

Contributors: Practical thematic analysis, an approach designed for multidisciplinary health services teams new to qualitative research, was based on CHS’s experiences teaching thematic analysis to clinical teams and students. We have drawn heavily from qualitative methods literature. CHS is the guarantor of the article. CHS, AS, CvP, AMK, JRK, and JAP contributed to drafting the manuscript. AS, JG, CMM, JAP, and RWY provided feedback on their experiences using practical thematic analysis. CvP, LCL, SLB, AVC, GE, and JKL advised on qualitative methods in health services research, given extensive experience. All authors meaningfully edited the manuscript content, including AVC and RKS. The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted.

Funding: This manuscript did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Competing interests: All authors have completed the ICMJE uniform disclosure form at https://www.icmje.org/disclosure-of-interest/ and declare: no support from any organisation for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work.

Provenance and peer review: Not commissioned; externally peer reviewed.

• Ziebland S ,
• ↵ A Hybrid Approach to Thematic Analysis in Qualitative Research: Using a Practical Example. 2018. https://methods.sagepub.com/case/hybrid-approach-thematic-analysis-qualitative-research-a-practical-example .
• Maguire M ,
• Vindrola-Padros C ,
• Vindrola-Padros B
• ↵ Vindrola-Padros C. Rapid Ethnographies: A Practical Guide . Cambridge University Press 2021. https://play.google.com/store/books/details?id=n80HEAAAQBAJ
• Schroter S ,
• Merino JG ,
• Barbeau A ,
• ↵ Padgett DK. Qualitative and Mixed Methods in Public Health . SAGE Publications 2011. https://play.google.com/store/books/details?id=LcYgAQAAQBAJ
• Scharp KM ,
• Korstjens I
• Barnett-Page E ,
• ↵ Guest G, Namey EE, Mitchell ML. Collecting Qualitative Data: A Field Manual for Applied Research . SAGE 2013. https://play.google.com/store/books/details?id=-3rmWYKtloC
• Sainsbury P ,
• Emerson RM ,
• Saunders B ,
• Kingstone T ,
• Hennink MM ,
• Kaiser BN ,
• Hennink M ,
• O’Connor C ,
• ↵ Yen RW, Schubbe D, Walling L, et al. Patient engagement in the What Matters Most trial: experiences and future implications for research. Poster presented at International Shared Decision Making conference, Quebec City, Canada. July 2019.
• ↵ Got questions about Thematic Analysis? We have prepared some answers to common ones. https://www.thematicanalysis.net/faqs/ (accessed 9 Nov 2022).
• ↵ Braun V, Clarke V. Thematic Analysis. SAGE Publications. 2022. https://uk.sagepub.com/en-gb/eur/thematic-analysis/book248481 .
• Kalpokas N ,
• Radivojevic I
• Campbell KA ,
• Durepos P ,
• ↵ Understanding Thematic Analysis. https://www.thematicanalysis.net/understanding-ta/ .
• Saunders CH ,
• Stevens G ,
• CONFIDENT Study Long-Term Care Partners
• MacQueen K ,
• Vaismoradi M ,
• Turunen H ,
• Schott SL ,
• Berkowitz J ,
• Carpenter-Song EA ,
• Goldwag JL ,
• Durand MA ,
• Goldwag J ,
• Saunders C ,
• Mishra MK ,
• Rodriguez HP ,
• Shortell SM ,
• Verdinelli S ,
• Scagnoli NI
• Campbell C ,
• Sparkes AC ,
• McGannon KR
• Sandelowski M ,
• Connelly LM ,
• O’Malley AJ ,

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, automatically generate references for free.

• Knowledge Base
• Methodology
• How to Do Thematic Analysis | Guide & Examples

How to Do Thematic Analysis | Guide & Examples

Published on 5 May 2022 by Jack Caulfield . Revised on 7 June 2024.

Thematic analysis is a method of analysing qualitative data . It is usually applied to a set of texts, such as an interview or transcripts . The researcher closely examines the data to identify common themes, topics, ideas and patterns of meaning that come up repeatedly.

There are various approaches to conducting thematic analysis, but the most common form follows a six-step process:

• Familiarisation
• Generating themes
• Reviewing themes
• Defining and naming themes

This process was originally developed for psychology research by Virginia Braun and Victoria Clarke . However, thematic analysis is a flexible method that can be adapted to many different kinds of research.

Table of contents

When to use thematic analysis, different approaches to thematic analysis, step 1: familiarisation, step 2: coding, step 3: generating themes, step 4: reviewing themes, step 5: defining and naming themes, step 6: writing up.

Thematic analysis is a good approach to research where you’re trying to find out something about people’s views, opinions, knowledge, experiences, or values from a set of qualitative data – for example, interview transcripts , social media profiles, or survey responses .

Some types of research questions you might use thematic analysis to answer:

• How do patients perceive doctors in a hospital setting?
• What are young women’s experiences on dating sites?
• What are non-experts’ ideas and opinions about climate change?
• How is gender constructed in secondary school history teaching?

To answer any of these questions, you would collect data from a group of relevant participants and then analyse it. Thematic analysis allows you a lot of flexibility in interpreting the data, and allows you to approach large datasets more easily by sorting them into broad themes.

However, it also involves the risk of missing nuances in the data. Thematic analysis is often quite subjective and relies on the researcher’s judgement, so you have to reflect carefully on your own choices and interpretations.

Pay close attention to the data to ensure that you’re not picking up on things that are not there – or obscuring things that are.

Prevent plagiarism, run a free check.

Once you’ve decided to use thematic analysis, there are different approaches to consider.

There’s the distinction between inductive and deductive approaches:

• An inductive approach involves allowing the data to determine your themes.
• A deductive approach involves coming to the data with some preconceived themes you expect to find reflected there, based on theory or existing knowledge.

There’s also the distinction between a semantic and a latent approach:

• A semantic approach involves analysing the explicit content of the data.
• A latent approach involves reading into the subtext and assumptions underlying the data.

After you’ve decided thematic analysis is the right method for analysing your data, and you’ve thought about the approach you’re going to take, you can follow the six steps developed by Braun and Clarke .

The first step is to get to know our data. It’s important to get a thorough overview of all the data we collected before we start analysing individual items.

This might involve transcribing audio , reading through the text and taking initial notes, and generally looking through the data to get familiar with it.

Next up, we need to code the data. Coding means highlighting sections of our text – usually phrases or sentences – and coming up with shorthand labels or ‘codes’ to describe their content.

Let’s take a short example text. Say we’re researching perceptions of climate change among conservative voters aged 50 and up, and we have collected data through a series of interviews. An extract from one interview looks like this:

In this extract, we’ve highlighted various phrases in different colours corresponding to different codes. Each code describes the idea or feeling expressed in that part of the text.

At this stage, we want to be thorough: we go through the transcript of every interview and highlight everything that jumps out as relevant or potentially interesting. As well as highlighting all the phrases and sentences that match these codes, we can keep adding new codes as we go through the text.

After we’ve been through the text, we collate together all the data into groups identified by code. These codes allow us to gain a condensed overview of the main points and common meanings that recur throughout the data.

Next, we look over the codes we’ve created, identify patterns among them, and start coming up with themes.

Themes are generally broader than codes. Most of the time, you’ll combine several codes into a single theme. In our example, we might start combining codes into themes like this:

At this stage, we might decide that some of our codes are too vague or not relevant enough (for example, because they don’t appear very often in the data), so they can be discarded.

Other codes might become themes in their own right. In our example, we decided that the code ‘uncertainty’ made sense as a theme, with some other codes incorporated into it.

Again, what we decide will vary according to what we’re trying to find out. We want to create potential themes that tell us something helpful about the data for our purposes.

Now we have to make sure that our themes are useful and accurate representations of the data. Here, we return to the dataset and compare our themes against it. Are we missing anything? Are these themes really present in the data? What can we change to make our themes work better?

If we encounter problems with our themes, we might split them up, combine them, discard them, or create new ones: whatever makes them more useful and accurate.

For example, we might decide upon looking through the data that ‘changing terminology’ fits better under the ‘uncertainty’ theme than under ‘distrust of experts’, since the data labelled with this code involves confusion, not necessarily distrust.

Now that you have a final list of themes, it’s time to name and define each of them.

Defining themes involves formulating exactly what we mean by each theme and figuring out how it helps us understand the data.

Naming themes involves coming up with a succinct and easily understandable name for each theme.

For example, we might look at ‘distrust of experts’ and determine exactly who we mean by ‘experts’ in this theme. We might decide that a better name for the theme is ‘distrust of authority’ or ‘conspiracy thinking’.

Finally, we’ll write up our analysis of the data. Like all academic texts, writing up a thematic analysis requires an introduction to establish our research question, aims, and approach.

We should also include a methodology section, describing how we collected the data (e.g., through semi-structured interviews or open-ended survey questions ) and explaining how we conducted the thematic analysis itself.

The results or findings section usually addresses each theme in turn. We describe how often the themes come up and what they mean, including examples from the data as evidence. Finally, our conclusion explains the main takeaways and shows how the analysis has answered our research question.

In our example, we might argue that conspiracy thinking about climate change is widespread among older conservative voters, point out the uncertainty with which many voters view the issue, and discuss the role of misinformation in respondents’ perceptions.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the ‘Cite this Scribbr article’ button to automatically add the citation to our free Reference Generator.

Caulfield, J. (2024, June 07). How to Do Thematic Analysis | Guide & Examples. Scribbr. Retrieved 9 September 2024, from https://www.scribbr.co.uk/research-methods/thematic-analysis-explained/

Is this article helpful?

Jack Caulfield

Other students also liked, qualitative vs quantitative research | examples & methods, inductive reasoning | types, examples, explanation, what is deductive reasoning | explanation & examples.

PY2106: Human Development Across the Lifespan Guide: Writing a Thematic Analysis

• Subject Basics
• Finding Resources
• Writing a Thematic Analysis
• Formatting & Referencing
• Psychology Guide
• Literature review guide This link opens in a new window

What is a thematic analysis?

A thematic analysis is used in qualitative research to focus on examining themes within a topic by identifying, analysing and reporting patterns (themes) within the research topic. It is similar to a literature review, which is a critical survey and assessment of the existing research on your particular topic.

The following links provide more information about the thematic analysis process.

• About Thematic Analysis
• Using thematic analysis in psychology

Thematic Analysis Process

Writing a literature review

You may also find these resources on writing literature reviews useful to help you write your analysis.

• Writing a Literature Review
• Literature Reviews - Synthesise

Writing a literature review (part 1)

Literature review books

Doing a literature review in health and social care: A practical guide

The Literature Review

The Literature Review is a concise step-by-step guide to conducting a literature search and writing up the literature review chapter in graduate dissertations and in professional doctorate theses. 

Succeeding with Your Literature Review

This step-by-step handbook provides comprehensive and practical guidance on the process of researching a range of relevant literature on a subject, as well as planning and writing a literature review.

An interactive approach to writing essays and research reports in psychology

This writing guide offers students an engaging, accessible introduction to the conventions of writing in the psychology discipline.

• << Previous: Finding Resources
• Next: Formatting & Referencing >>
• Last Updated: Apr 26, 2024 9:33 AM
• URL: https://libguides.jcu.edu.au/py2106

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• My Bibliography
• Collections
• Citation manager

Save citation to file

Email citation, add to collections.

• Create a new collection
• Add to an existing collection

Add to My Bibliography

Your saved search, create a file for external citation management software, your rss feed.

• Search in PubMed
• Search in NLM Catalog
• Add to Search

Thematic analysis of qualitative research data: Is it as easy as it sounds?

Affiliations.

• 1 University of Arkansas for Medical Sciences, College of Pharmacy, 4301 West Markham Street, Slot 522-4, Little Rock, AR 72205, United States. Electronic address: [email protected].
• 2 University of Arkansas at Little Rock, School of Education, 2801 S. University, Little Rock, AR 72204, United States. Electronic address: [email protected].
• PMID: 30025784
• DOI: 10.1016/j.cptl.2018.03.019

Issue: We are seeing the use of qualitative research methods more regularly in health professions education as well as pharmacy education. Often, the term "thematic analysis" is used in research studies and subsequently labeled as qualitative research, but saying that one did this type of analysis does not necessarily equate with a rigorous qualitative study. This methodology review will outline how to perform rigorous thematic analyses on qualitative data to draw interpretations from the data.

Methodological literature review: Despite not having an analysis guidebook that fits every research situation, there are general steps that you can take to make sure that your thematic analysis is systematic and thorough. A model of qualitative data analysis can be outlined in five steps: compiling, disassembling, reassembling, interpreting, and concluding.

My recommendations and their applications: Nine practical recommendations are provided to help researchers implement rigorous thematic analyses.

Potential impact: As researchers become comfortable in properly using qualitative research methods, the standards for publication will be elevated. By using these rigorous standards for thematic analysis and making them explicitly known in your data process, your findings will be more valuable.

Keywords: Qualitative; Thematic analysis.

Published by Elsevier Inc.

PubMed Disclaimer

Similar articles

• Thematic analysis of qualitative data: AMEE Guide No. 131. Kiger ME, Varpio L. Kiger ME, et al. Med Teach. 2020 Aug;42(8):846-854. doi: 10.1080/0142159X.2020.1755030. Epub 2020 May 1. Med Teach. 2020. PMID: 32356468
• Conducting and presenting qualitative research in pharmacy education. Bush AA, Amechi MH. Bush AA, et al. Curr Pharm Teach Learn. 2019 Jun;11(6):638-650. doi: 10.1016/j.cptl.2019.02.030. Epub 2019 Mar 27. Curr Pharm Teach Learn. 2019. PMID: 31213322 Review.
• Twelve tips for using Pattern Matching in data analysis for qualitative medical education research. Attard Cortis P, Muir F. Attard Cortis P, et al. Med Teach. 2022 Nov;44(11):1209-1213. doi: 10.1080/0142159X.2021.1937587. Epub 2021 Jun 16. Med Teach. 2022. PMID: 34130598
• Student and educator experiences of maternal-child simulation-based learning: a systematic review of qualitative evidence protocol. MacKinnon K, Marcellus L, Rivers J, Gordon C, Ryan M, Butcher D. MacKinnon K, et al. JBI Database System Rev Implement Rep. 2015 Jan;13(1):14-26. doi: 10.11124/jbisrir-2015-1694. JBI Database System Rev Implement Rep. 2015. PMID: 26447004
• Ensuring rigour and trustworthiness of qualitative research in clinical pharmacy. Hadi MA, José Closs S. Hadi MA, et al. Int J Clin Pharm. 2016 Jun;38(3):641-6. doi: 10.1007/s11096-015-0237-6. Epub 2015 Dec 14. Int J Clin Pharm. 2016. PMID: 26666909 Review.
• 'I've got no PPE to protect my mind': understanding the needs and experiences of first responders exposed to trauma in the workplace. Cogan N, Craig A, Milligan L, McCluskey R, Burns T, Ptak W, Kirk A, Graf C, Goodman J, De Kock J. Cogan N, et al. Eur J Psychotraumatol. 2024;15(1):2395113. doi: 10.1080/20008066.2024.2395113. Epub 2024 Sep 6. Eur J Psychotraumatol. 2024. PMID: 39238472 Free PMC article.
• A qualitative study of negative sociocultural experiences of accessing primary health care services among Africans from refugee backgrounds in Australia: implications for organisational health literacy. Peprah P, Lloyd J, Ajang DA, Harris MF. Peprah P, et al. BMC Prim Care. 2024 Sep 4;25(1):327. doi: 10.1186/s12875-024-02567-2. BMC Prim Care. 2024. PMID: 39232655 Free PMC article.
• The Impact of Climate Change on Road Traffic Crashes in Ghana. Akorli R, Antwi-Agyei P, Davies P, Damsere-Derry J, Baffour-Ata F, Nakua E, Donkor P, Mock C. Akorli R, et al. Res Sq [Preprint]. 2024 Aug 14:rs.3.rs-4654960. doi: 10.21203/rs.3.rs-4654960/v1. Res Sq. 2024. PMID: 39184084 Free PMC article. Preprint.
• Needs and capabilities for improving poultry production and health management in Indonesia. Chapot L, Hibbard R, Ariyanto KB, Maulana KY, Yusuf H, Febriyani W, Cameron A, Paul M, Vergne T, Faverjon C. Chapot L, et al. PLoS One. 2024 Aug 22;19(8):e0308379. doi: 10.1371/journal.pone.0308379. eCollection 2024. PLoS One. 2024. PMID: 39172873 Free PMC article.
• "Psychiatric emergency consultations of minors: a qualitative study of professionals' experiences". So P, Nooteboom LA, Vullings RM, Mulder CL, Vermeiren R. So P, et al. BMC Psychiatry. 2024 Aug 7;24(1):549. doi: 10.1186/s12888-024-05996-6. BMC Psychiatry. 2024. PMID: 39112937 Free PMC article.
• Search in MeSH

Related information

• Cited in Books

LinkOut - more resources

Full text sources.

• Elsevier Science

Other Literature Sources

• scite Smart Citations

Research Materials

• NCI CPTC Antibody Characterization Program
• Citation Manager

NCBI Literature Resources

MeSH PMC Bookshelf Disclaimer

The PubMed wordmark and PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.

Advertisement

Evolution of Ethics and Entrepreneurship: Hybrid Literature Review and Theoretical Propositions

• Review Paper
• Open access
• Published: 11 September 2024

Cite this article

You have full access to this open access article

• Sebastián Uriarte   ORCID: orcid.org/0000-0001-9012-5082 1 ,
• Cristian Geldes 2 &
• Jesús Santorcuato 3  

Entrepreneurship has been highlighted as one of the major forces in addressing significant economic, social, and environmental challenges. These challenges have raised new ethical questions, leading to an explosive growth of research at the intersection of ethics and entrepreneurship. This study provides an overview of the evolution of the scientific literature on the interplay between ethics and entrepreneurship to propose a research proposition with standardized protocols and a broad time limit. Specifically, in a hybrid literature review, 516 articles from peer-reviewed journals indexed in Scopus were analyzed. The review revealed that the field mainly comprises six themes. Through the analysis of each theme, gaps are identified and structured and used to build theoretical proposals for future research agendas applied to current societal challenges. Understanding the link between entrepreneurship and ethics guides practices improves decisions, addresses challenges, promotes sustainability, enhances academia, and builds trust, fostering a responsible, beneficial entrepreneurial environment for society and the economy.

Similar content being viewed by others

Ethics Perspective on Entrepreneurship

The Ethics of Entrepreneurial Shared Value

The ethics of entrepreneurship: a millian approach, explore related subjects.

• Artificial Intelligence
• Medical Ethics

Avoid common mistakes on your manuscript.

Introduction

The role of entrepreneurship in society has been highlighted due to its contributions to economic development, job creation, productivity growth, innovation development, and poverty reduction (Acs et al., 2008 ; Ribeiro-Soriano, 2017 ; Van Praag et al., 2007 ). In recent years, entrepreneurship has even been highlighted as one of the major forces in addressing significant social and environmental challenges, leading to better conditions for people and the planet (Hall et al., 2010 ; Markman et al., 2019 ; Stephan et al., 2016 ). Despite the role of entrepreneurship in society, the relationship between ethics and entrepreneurship is increasingly debated due to certain questionable behaviors by some entrepreneurs. Although entrepreneurs are often seen as value creators, research shows that they do not always act ethically. Entrepreneurs are associated with shrewd maneuvers and creative overcoming of constraints to ensure business success, and these behaviors can lower their ethical standards (Morris et al., 2002 ). Factors such as scarce resources, limited legitimacy, dependence on suppliers, and external pressures can lead to increased risk-taking and unethical actions. This highlights the need for further study of the intersection between ethics and entrepreneurship (EE) (Clarke et al., 2010 ; Hägg et al., 2024 ; Hannafey, 2003 ).

Entrepreneurship is a multidimensional and complex phenomenon that has different definitions, classifications, and approaches (Gedeon, 2010 ; Iversen et al., 2007 ). The Global Entrepreneurship Monitor defines entrepreneurship as “any attempt to create a new business or new venture, such as self-employment, a new business organization or the expansion of an existing business, by an individual, a team of individuals or an established firm” (GEM, 2024 ). In pursuit of new opportunities, entrepreneurship has been challenged by diverse technological, environmental, social, political, and economic changes. These challenges have raised new ethical questions, leading to an explosive growth in business scholars’ research on EE. The discussion of traditional thematic areas such as ethics and entrepreneurs (e.g., Ahsan, 2020 ; Baron et al., 2015 ; Higgins-Desbiolles et al., 2021 ), corporate social responsibility (e.g., Bandyopadhyay et al., 2020 ; Luque et al., 2019 ), and gender (e.g., Althalathini et al., 2022 ; De Clercq et al., 2022 ) has intensified and deepened. Moreover, a broad range of topics has been added, ranging from sustainability (e.g., Markman et al., 2016 ; Rui et al., 2021 ) and artificial intelligence, blockchain or big data (e.g., Obschonka et al., 2020 ; Saheb et al., 2021 ) to social entrepreneurship (Sengupta et al., 2022 ; Slavec Gomezel et al., 2023 ) and even animal ethics (e.g., Notzke, 2019 ). The emergence of new societal challenges requires analysis of the evolution of new EE-related issues.

In this vein, there are three critical reviews. Hannafey ( 2003 ) offers the first specific literature review that highlights the relevance of the topic. Daradkeh ( 2023 ) is a systematic literature review of 75 papers that focus on entrepreneurial and ethical decision-making and behavior, and Vallaster et al. ( 2019 ) are a bibliometric and systematic analysis. The latter is the most comprehensive and cited review of the literature; however, it focuses mainly on a qualitative review of the literature up to 2016 and does not account for thematic evolution. An evolutionary approach is essential for quantitatively understanding mature and emerging topics, including new ethical challenges. In addition, recent literature reviews require standardized protocols to provide a transparent and replicable review and structured frameworks or theoretical propositions to shape future research. Therefore, despite these efforts, there is a gap in the literature from the methodological and theoretical perspectives applied to current societal challenges. These gaps are addressed in our study. From a methodological perspective, this paper presents a comprehensive and timely review of the literature on EE, which has attracted considerable interest among scholars in recent years. The period under review also encompasses recent studies that concern, among other challenges, a pandemic, revolutionary technological changes, and cultural and generational shifts. In addition, this study adopts a hybrid literature review approach that uses qualitative and quantitative methods such as bibliometric analysis, thematic analysis, and structured review and allows for the identification of gaps in research on the main constructs based on the information obtained (Paul et al., 2020 ). From the theoretical perspective, based on this thematic evolution, the main themes that emerge from the entrepreneurship and ethics literature are identified and summarized. Then, through the analysis of papers from each cluster, gaps are identified and structured and used to build theoretical proposals for future research agendas applied to current societal challenges. In this way, this study aims to develop a hybrid critical review of the literature on EE and societal challenges and uses it to develop a research proposition that features standardized protocols and a broad time limit.

Accordingly, this paper responds to calls for additional research on EE (Obschonka et al., 2020 ; Saheb et al., 2023 ; Vallaster et al., 2019 ). In line with the abovementioned research, this study addresses the following research questions (RQs):

RQ1. What is the evolution of publications in ethics and entrepreneurship research? The answer to this research question provides information on the dynamism and relevance that researchers have given to the field.

RQ2. What are the top articles, journals, authors, and collaboration networks in ethics and entrepreneurship research? Responding to this research question enables researchers to identify the foundational theories, methodologies and knowledge that are fundamental for researchers to understand and draw from. It also allows editors to benchmark their journal’s productivity in the field. A collaborative network fosters knowledge sharing and leads to more innovative research and a greater impact on research output.

RQ3. How have the themes in research on ethics and entrepreneurship evolved? Answering this research question provides a comprehensive understanding of the content of the body of knowledge in the field.

RQ4. What opportunities exist for future studies of ethics and entrepreneurship? The answer to this research question is expected to assist researchers in addressing gaps, exploring emerging trends, establishing innovative methodologies and contributing to practical solutions, thereby increasing the relevance and impact of the field.

This study makes significant methodological and theoretical contributions to current societal challenges. First, this study is one of the first hybrid literature reviews of both the ethics and entrepreneurship literature and is certainly the first to address the relationship between ethics and entrepreneurship; it thereby contributes to research at the methodological level. This study broadens the coverage of published research in several key ways: (i) instead of using databases such as Web of Science (WoS), it utilizes a larger database, Scopus, and expands the period under review; (ii) it applies the SPAR-4-SLR protocol, which is explicitly designed for use in the social sciences, thus promoting the transparency and reproducibility of the review results; (iii) it employs modern software, including the Bibliometrix package in R, Rayyan, and Stata 15; and (iv) it cross-references all data with the SCImago Journal & Country Rank database to determine the number of papers published in each quartile and thereby introduces a new measure for assessing the quality of publications in performance analysis. Using this procedure, 516 papers published between 1988 and 2023 were collected from journals indexed in the Scopus database. This procedure made it possible to identify the most relevant and high-impact scientific authors, papers, sources, and coauthorship networks in the EE literature. Consequently, this study responds to calls for enhancing the quality of review studies in business and management (e.g., Alshater et al., 2023 ; Donthu et al., 2021 ; Lim et al., 2022 ; Paul et al., 2021 ; Robledo et al., 2023 ).

Second, by mapping thematic evolution concerning EE, this study presents the state of the field, highlights the theoretical concepts used, and reveals the connections among them. Specifically, six research streams that consolidate previous findings (gender, digital technologies and innovation, sustainability and corporate social responsibility, social entrepreneurship, ethical entrepreneurs, and entrepreneurship ethics) are identified. The in-depth analysis provides theoretical propositions that form a basis for future lines of research highlighting the importance of ethical considerations in all aspects of entrepreneurial practices. Finally, the study offers practical contributions by emphasizing the need for tailored ethical training programs, integrating sustainability into entrepreneurial strategies, promoting gender-inclusive practices, supporting social entrepreneurs, and developing ethical guidelines for digital technologies.

As noted above, ethical entrepreneurial practices contribute to economic growth and can address societal challenges. Understanding the relationship between entrepreneurship and ethics is therefore crucial, as it guides ethical entrepreneurial practices, improves decision-making, addresses emerging challenges, promotes sustainable development, informs policy, enhances academic understanding, and builds public trust. This fosters a responsible, sustainable, and trustworthy entrepreneurial environment that benefits both society and the economy.

Methodology

To improve the understanding of a topic and serve as a basis for future research, review papers identify and critically evaluate scattered literature, adopting approaches ranging from clearly qualitative to overtly quantitative (Pedroletti et al., 2023 ). To answer the specific research questions addressed, this paper adopts two complementary techniques. On the one hand, a bibliographic analysis identifies the research tendencies that articulate the academic debate on entrepreneurship and ethics. On the other hand, a thematic evolution systematizes the focal contributions made by the literature, and an interpretative review elaborates the existing academic knowledge. This mixed approach integrates the principles of the bibliometric review with those of a structured review and thereby allows the identification of gaps in research on the main constructs on the basis of the information obtained; thus, the current work represents a hybrid systematic literature review (Paul et al., 2020 ). In the academic literature, this method has recently been used in areas of the social sciences, such as management, business, and entrepreneurship (Alshater et al., 2023 ; Baier-Fuentes et al., 2020 ; Bhukya et al., 2023 ; Lashitew et al., 2022 ).

Academic bibliometric analysis is an objective quantitative tool that makes it possible to evaluate the development of a specific subject through the application of statistical and mathematical methods that are recognized and validated by the scientific community (Baier-Fuentes et al., 2019 ; Lim et al., 2022 ). Bibliometric analyses identify salient developments and trends as well as the most relevant scientific authors and documents and the sources that have the greatest impact on a specific topic; these analyses are characterized by their accuracy, reliability, and verifiability (Alshater et al., 2023 ). Bibliometric analysis provides a comprehensive analysis of the performance of units of study, such as papers, authors, and journals (Donthu et al., 2021 ). Similar to other research (Baier-Fuentes et al., 2019 ; Lim et al., 2022 ; Magni et al., 2022 ), this study applies two performance analysis techniques: descriptive analysis and citation analysis. In addition, this study develops a coauthorship network map that can be used to understand the collaboration network among authors.

Moreover, this study presents the thematic evolution of entrepreneurship and ethics research. A longitudinal study provides insights into internal dynamics and developmental patterns (Sharma et al., 2023 ). This is achieved by analyzing the relationships among authors’ keywords and displaying them in a thematic cluster map (Pedroletti et al., 2023 ). Themes are ordered according to Callon’s centrality and density metrics (Callon et al., 1991 ). Callon’s centrality quantifies the degree of interaction of a network with other networks. It can be understood as a theme’s relevance within the development of the entire research field analyzed (Cobo et al., 2011 ). The density of callons measures the strength of internal ties among all keywords describing the research theme. A high Callon density indicates that the terms are often used together, implying a well-developed and coherent subfield or topic within the broader research landscape (Cobo et al., 2011 ). On the basis of these two metrics, this study visualizes the themes in a two-dimensional strategic diagram with four quadrants (Sharma et al., 2023 ). Motor themes are conceptually important and internally coherent themes that drive the research literature in a particular period; niche themes are well developed—showing strong internal cohesion—but are of minor importance to the broader field of study; basic themes are central but are not well developed as independent fields of study; finally, emerging or declining themes are not central to the body of work and do not show a strong internal structure, implying less conceptual or theoretical development (Callon et al., 1991 ; Cobo et al., 2011 ). The Louvain clustering algorithm Footnote 1 (Blondel et al., 2008 ) is used to perform the analyses that produce the thematic evolution and coauthorship networks presented above. The Louvain clustering algorithm outperforms other algorithms, such as Walktrap, Fast Greedy and Leiden, in terms of modularity and processing time, providing a fair compromise between the computational complexity and accuracy of the maximum modularity estimation (Lee et al., 2020 ; Zhang et al., 2021 ).

Finally, a reliable literature review must adopt a consistent and replicable protocol. Following Paul et al. ( 2021 ), this study adheres to the Scientific Procedures and Rationale for Systematic Reviews of the Literature (SPAR-4-SLR) guidelines. Unlike the available alternatives, such as the Preferred Reporting Items for Systematic Reviews (PRISMA), this protocol explicitly concerns the social sciences, where entrepreneurial research resides, and provides guidelines for the inclusion and exclusion of publications. More importantly, adopting a review protocol is the best practice for systematic literature reviews because it promotes the transparency and replication of review findings (Lim et al., 2022 ). The SPAR-4-SLR protocol stipulates that a systematic literature review consists of three stages: assembling , arranging , and assessing scholarly literature (Paul et al., 2021 ). Figure  1 shows a summary of the steps, and the following sections explain the most relevant definitions.

The SPAR-4-SLR protocol

This study focuses on journals because journal publications represent completed research that has been submitted to rigorous peer review (Paul et al., 2021 ). Other sources, such as books, book chapters, conference papers, commentaries, and editorials, are not considered, as they do not usually receive the same level of scrutiny to which conceptual, empirical, and review papers in journals are subjected (Baier-Fuentes et al., 2020 ) This study used Scopus as a measure of source quality. Scopus is one of the most important databases in the scientific community. It was designed for use in both searching for bibliographic material and performing content or thematic analysis, and it thereby offers the same analytical tools that are offered by other frequently used databases, such as the Web of Science (WoS) (Baier-Fuentes et al., 2019 ). In fact, this study cross-referenced all the papers in the WoS and Scopus databases that included the word “entrepreneurship” and found 91% of the WoS papers in Scopus. Footnote 2

The search period extended through December 31, 2023, the end of the most recent period at the time of the preparation of this paper. The search keywords related to ethics and entrepreneurship were accompanied by asterisks to account for any potential variations in these terms, and the search in the “title, abstract and keywords” was as follows: (“start-up*” OR “startup*” OR “startup*” OR “entrepren*” OR “venture*” OR “new firm*”) AND (ethic*). The assembly stage yielded 3,823 articles.

This study includes only “papers and reviews,” as other published items such as “editorials” and “notes” may not have undergone peer review; these were written in “English” and published in “journals” in the areas of “business, management and accounting,” the umbrella discipline in Scopus that covers entrepreneurship and business research. This protocol substage resulted in the inclusion of 937 publications and the exclusion of 2,886 publications. The retrieved contributions were screened on the basis of three exclusion criteria: (i) no direct relationship to the study topic (i.e., contributions in areas outside ethics and entrepreneurship); (ii) no direct relationship to the scope of the research (i.e., irrelevant reports concerning ethics or entrepreneurship); and (iii) no direct relationship to the study’s focus (i.e., records that did not involve practices/tools/approaches targeted to ethics and entrepreneurship). The authors independently analyzed the titles, abstracts, and keywords of the retrieved articles and excluded those that fell within one of the three above categories. At the end of the independent selection, the authors agreed to exclude 421 articles. The final dataset consists of 516 Footnote 3 relevant and impactful contributions.

A performance analysis was conducted using Microsoft Excel and Stata 15 to delineate the publication trends and the top journals, articles, authors, countries, and institutions in the field. The Bibliometrix package in R allowed science mapping according to a collaboration network between the authors and countries, analysis of co-occurrence of the keywords, and cocitation analysis. The agenda proposal involved the use of thematic analysis, which was also performed using the Bibliometrix package in R.

Results of the Bibliometric Analysis

Ethics and entrepreneurship performance, publication productivity of ethics and entrepreneurship research (rq1).

Figure  2 shows the productivity in terms of total publications (TPs) and total accumulated citations (TCs) related to the topic of entrepreneurship and ethics (EE) from 1988 to 2023. The first relevant studies were published at the end of the twentieth century. In particular, Longenecker et al. ( 1988 ) focused on the independent and egoistic tendencies that may lead entrepreneurs down different ethical paths to financial gain. Vyakarnam et al. ( 1997 ) identified entrepreneurial activity itself, conflicts of interest, social responsibility, and personal issues as four significant ethical dilemmas.

Evolution of publications and cumulative citations by year

From this starting point until 2013, Footnote 4 which this study labels the “formative phase,” the number of articles increased constantly and significantly, peaking in 2009, when the Journal of Business Venturing published a special issue. The theoretical foundations of the field under study were established during this period. The following five years (2014–2018), which this study calls the “consolidation phase,” began a period during which research remained relatively constant, with an average of 18 papers published per year. The period since 2019 includes approximately half of the publications on the topic and is marked by topics related to contingency, technology, and new business models. Therefore, this study labels this phase the “pragmatic phase.” However, despite the distribution of the total number of publications, the cumulative total number of citations shows a linear trend, with no apparent exponential increase or decrease. Another way to analyze the literature’s growth and influence on this research topic is through the general citation structure of the publications, as presented in Table  1 .

In Table  1 , the publications are ordered according to year, total number, and citations received in the papers published in the same year. The table also includes the number of papers published in each quartile according to the SCimago Journal & Country Rank. During the last few years, the average number of publications more than doubled, but the average number of citations per publication decreased. However, because the data were categorized using quartiles, this study revealed considerable variations and improvements in the quality of the literature. For example, in 2022 and 2023, more than 74% of the publications were in first-quartile journals (Q1). Similarly, in 2002, 2006, and 2008, more than 80% of the publications were in high-impact journals. During the period of topic consolidation (2014–2018), 45% of the publications were in first-quartile journals.

Top Articles for Ethics and Entrepreneurship Research (RQ2)

An exciting aspect to review is the most influential publications on this research topic, i.e., those that have received the most citations. The number of citations each paper received reflects its popularity and influence in the scientific community (Baier-Fuentes et al., 2019 ). Table 2 presents the ten most cited papers on the research topic in each phase defined above.

As mentioned previously, Zahra et al. ( 2009 ) published the most cited paper in the Journal of Business Venturing, with 1,573 citations. This study was published during the formative phase and has been one of the most important papers in research on social entrepreneurship. André et al. ( 2016 ) and Hechavarría et al. ( 2017 ) were the most cited publications during the consolidation phase. In the pragmatic phase, this study highlights Obschonka et al. ( 2020 ); Vallaster et al. ( 2019 ), and Hota et al. ( 2020 ). The latter also have a high average number of citations per year. This reflects the fact that despite their later publication dates, these studies have had a high impact.

Top Journals for Ethics and Entrepreneurship Research (RQ2)

Table 3 shows a classification of the 10 most influential and productive journals in the field of EE. Notably, the journal in which entrepreneurship and ethics research are most concentrated is the Journal of Business Ethics, with 121 articles and 23% of the articles published on this topic. Other journals with renowned reputations are highlighted for their publication of EE papers exceeding a threshold of 100 citations. These include the Business Ethics Quarterly, the Journal of Business Venturing, the Journal of Business Research, the Journal of Small Business and Enterprise Development, Technological Forecasting and Social Change, the International Journal of Entrepreneurship and Behavior Research, and Small Business Economics.

Another important issue that Table  3 reveals is the progress of EE research in journals over time. For this purpose, the papers published in these journals were grouped into the three periods defined above (i.e., 1999 to 2013, 2014 to 2018, and 2019 to 2023). Overall, the results show that the number of publications on EE research in several journals has progressively increased. In fact, the last five years have been very productive, and almost all the focal journals have published at least one document associated with this field of research.

The results also show that almost all these journals began publication during the formative phase. However, this study highlights the Journal of Business Ethics (JBE), the Business Ethics Quarterly (BEQ), and the Journal of Business Venturing (JBV) as pioneering journals in this field. For example, JBE published an article titled “Toward an understanding of ethical behavior in small firms” (Vyakarnam et al., 1997 ), the first paper that dealt with ethics from an entrepreneurial and small business perspective. In 2003, the JBV published “Ethics and Entrepreneurs: An international comparative study,” in which the authors developed a conceptual framework for examining cross-cultural differences in the ethical attitudes of businesspeople on the basis of integrative social contract theory (Bucar et al., 2003 ). The BEQ then published “Managing social-business tensions: A review and research agenda for social enterprise” (Smith et al., 2013 ), one of the first literature reviews to study the tensions between social missions and business ventures.

With respect to the consolidation phase, this study highlights the International Journal of Entrepreneurship and Small Business and the International Journal of Business and Globalization. Both journals presented concentrated publications in the EE field in this phase. Critically, in this phase, the authors began to diversify the scope of their work and deepen the topic in terms of, e.g., social entrepreneurship (e.g., André et al., 2016 ), religion (e.g., Ramadani et al., 2015 ; Tlaiss, 2015 ) and crowdfunding (e.g., Jancenelle et al., 2018 ).

As noted above, the pragmatic phase was the most productive period in the EE field. For example, the Journal of Business Research, Technological Forecasting and Social Change, and Entrepreneurship and Regional Development published all their contributions on ethics and entrepreneurship during this period. In this phase, more contingent topics, such as COVID-19 and digital technologies, were also explored (e.g., Nylund et al., 2022 ).

An important insight is that journals that are primarily interested in business ethics are among the most productive. This presents a challenge for entrepreneurship-focused journals.

Top Authors in the Field of Ethics and Entrepreneurship Research (RQ2)

A total of 1,170 authors contributed to the development of this research topic. Table 4 lists the 30 most productive and influential authors in EE research. In terms of the focal field, the five EE authors with the most citations—the strongest influence—are Neubaum, Zahra, Gedajlovic, Shulman, and Gonin. However, these five authors have concentrated their citations on a single research study and have not contributed to subsequent research. With respect to productivity, Fassin, McVea, and Ratten are the top three authors. Fassin’s work on EE is rooted in conflicts between entrepreneurs and investors, ethics in entrepreneurial finance, and ethical considerations in the innovation industry. In contrast, McVea’s work on EE concerns entrepreneurial decision-making, while Ratten focuses on entrepreneurial activity and ethics as it relates to cloud computing and e-book devices. Importantly, Fassin and McVea have published their studies mainly in first-quartile journals, while Ratten has primarily published works in third-quartile journals, according to the SCimago Journal & Country Rank.

Analyzing the evolution over time of publications per author across the phases defined above, Fassin represents both the development and the consolidation phases. In the pragmatic phase, Hota and Steyaert are the most productive authors. Hota’s work on EE is rooted in social entrepreneurship, and Steyaert’s work is rooted in organizational ethics.

Because authors often collaborate and contribute to other research areas, we present these contributions at a general level. The indicators show that several prominent authors have impacted science in general. These include, for example, Vanessa Ratten, who has more than 8,794 citations and 488 papers.

Collaboration Networks (RQ2)

A collaboration network between authors and a country network can illustrate the social structure of an academic field. Figure  3 shows 10 clusters among the 50 most influential authors, although many authors are presented as isolated nodes. However, this does not mean that these authors do not collaborate. In fact, the average number of authors per publication is 2.4, and only 25% of publications have a single author. Furthermore, there is a high level of collaboration among authors from developed economies such as the United Kingdom, the United States, and European countries. The collaboration rate is represented by more robust lines in Fig.  4 ; in that figure, the countries shown in darker colors are the more productive countries. With the exception of China, collaboration among authors from developing countries is scarce.

Author collaboration network

Collaborations across countries based on author affiliations

These results imply that the topic has low barriers to entry and that few specialized authors have addressed it. Therefore, more scholars should be encouraged to consider contributing to the current debate, especially in collaboration with coauthors from developing countries.

Conceptual Thematic Map (RQ3)

The thematic analysis performed in this study identified the changes and continuities in research related to ethics and entrepreneurship between 1988 and 2023 across the three phases. Figure  5 shows the formative phase in terms of basic themes such as organization, business ethics, and entrepreneurship. The themes identified as motor—high relevance and high development—are sustainability, corporate social responsibility, and social enterprise. Moreover, the themes categorized as niche—low relevance and high development—include entrepreneurs and management. Finally, emerging or declining themes—those with low relevance and low development—encompass entrepreneurial opportunity and developing countries.

Thematic aspects of the topic in the formative phase (1988–2013)

Figure  6 , which shows the consolidation phase (2014–2018), illustrates the evolution of business ethics. This topic has lost relevance but has increased in terms of development. Entrepreneurship also shifted from a basic to a motor theme due to its increased development. New basic themes that emerged during this phase included gender issues and social entrepreneurship. With respect to the niche themes, research linked to innovation and digital technology gained relevance.

Thematic aspects of the topic in the consolidation phase (2014–2018)

Finally, in Fig.  7 , which shows the pragmatic phase (2019–2023), entrepreneurship and social entrepreneurship, along with sustainability, became basic themes, while innovation and digital technology transitioned from niche themes to motor themes. Finally, the emerging themes included aspects related to the organization.

Thematic aspects of the topic in the pragmatic phase (2019–2023)

Figure  8 presents the aggregated thematic map for the entire period under study (1988–2023). From this thematic map, six relevant topics are identified. First, among the motor themes is gender. The emergence of women’s participation in entrepreneurship has raised interest among researchers in gender variations in terms of ethical and social value goals and discrimination. Another important connection with this field concerns the effects of various religions, such as Buddhism, Greek Orthodox Christianity, and, predominantly, Islam, on women’s entrepreneurial activity (e.g., Dissanayake, 2022 ; Gotsis et al., 2009 ; Gümüsay, 2015 ; Ramadani et al., 2015 ). Second, another motor theme is business ethics and entrepreneurs. For example, Hoang et al. ( 2023 ) examined the effects of ethical and entrepreneurial leadership on innovative service behavior in SME hotels in Vietnam, and Belas et al. ( 2022 ) evaluated the perceptions and impact of business ethical factors among entrepreneur-engineers and entrepreneur-nonengineers. Third, among the basic themes are the relationship between sustainability and corporate social responsibility (CSR). Although CSR and sustainability are distinct concepts, the two have overlapped over time in the EE literature. Sustainability is justified and motivated by ethics (moral considerations) and executed by following entrepreneurial principles (e.g., Markman et al., 2016 ). Several studies also link innovative business models that promote sustainable practices or those based on governance that incorporate social and environmental stakeholders (e.g., Cumming et al., 2016 ; Elkington, 2006 ) to sustainability. Fourth, social entrepreneurship is shown in the realm of basic themes. Studies of the ethics of social entrepreneurs have discussed the ethical nature of social entrepreneurs as a factor that differentiates between social and commercial entrepreneurship (Hota et al., 2020 ). Other studies (e.g., Di Lorenzo et al., 2019 ) argue that altruistic motives and values such as freedom or equality have significant elements that contribute to understanding the ethics of social entrepreneurship. Fifth, several studies have addressed the relationship between entrepreneurship and ethics using evidence from SMEs. For example, using small business data, Arend ( 2013 ) studied the effects of dynamic capabilities on ethical and competitive performance and whether these effects depend on a firm’s entrepreneurial characteristics. Despite its importance, this theme can be categorized as a declining topic. Finally, innovation and digital technologies are positioned as emerging themes. Research in this area has focused on several ethical considerations that are relevant at different stages in the life of innovative entrepreneurship, such as intellectual property issues, confidentiality of information, the negotiation process between the entrepreneur and the financier, fundraising, and insider trading, among others (e.g., Fassin, 2000 ; Long et al., 2020 ). Moreover, researchers have recently studied the ethical challenges associated with the use of digital technologies such as artificial intelligence and big data (e.g., Knieps, 2023 ; Obschonka et al., 2020 ; Racine, 2021 ).

Thematic map of the topic (1988–2023)

Theoretical Propositions for Future Avenues for Entrepreneurship and Ethics Research (RQ4)

Ethical considerations in entrepreneurship.

The relevance of this topic is reflected by Hannafey ( 2003 ), who highlighted its importance and proposed a series of issues that should be investigated, such as the role of entrepreneurship in the global economy and intercultural differences in the ethics of entrepreneurs, the evolution, over time, of changes in ethical interests, the perspectives and behaviors of entrepreneurs, the influence of family and academic training on the ethics of entrepreneurs, and the implications of technological changes for people.

Other contributions to the understanding of EE have focused on the formal and informal ethical structures that emerge in entrepreneurial firms over time. In this respect, Morris et al. ( 2002 ) indicated that implementing ethical structures impacts the perceived clarity and appropriateness of a firm’s ethical standards and the firm’s preparedness to address ethical challenges. They identified relevant factors such as the psychological profile of the entrepreneur, the stage of the firm’s life cycle, and the descriptive characteristics of the firm. Another analysis referred to ethics in situations in which entrepreneurs “break the rules,” a situation that is often regarded positively in the business world but can be questioned ethically and morally. In this vein, Brenkert ( 2009 ) has advocated a virtue-based ethics of entrepreneurship in which certain instances of rule-breaking, even if morally wrong, are ethically acceptable and are part of the creative destruction that entrepreneurs bring to the economy and morality. Similarly, Zhang et al. ( 2009 ) reported a positive relationship between “modest rule breaking” in adolescence and entrepreneurship in adulthood.

In these analyses of EE, approaches linked to small and medium-sized enterprises have also emerged. For example, Spence et al. ( 2001 ), in analyzing small business owner-managers’ social and ethical orientation, proposed four “frames” for reviewing and offering policies for small businesses: Profit-maximization priority, subsistence priority, enlightened self-interest, and social priority. Arend ( 2013 ) linked ethics to the dynamic capabilities of small and medium-sized enterprises, showing that these capabilities have positive effects on the ethical performance of these enterprises. In addition, topics such as how Protestant ethics influence entrepreneurship (Carr, 2003 ) have been discussed in relation to analyses of ethical and socially responsible practices in firms in emerging economies such as Malaysia. Ethical practices are positively associated with firm performance, but socially responsible practices are not, as these considerations are not relevant concerns for entrepreneurs (Ahmad et al., 2012 ). Additionally, themes linking ethics with fraud in the reward-based crowdfunding market (Cumming et al., 2023 ) and social value creation have emerged in recent years (Skinner, 2019 ).

Some open research questions may include the following: Do the ethical considerations of companies that exploit natural resources differ from those of other companies? How can organizational structures that encourage ethical decision-making within ventures and their relationships with their stakeholders be developed? How does venture capital address ethical issues in entrepreneurship based on digital technologies such as artificial intelligence? (Arend, 2013 ; Skinner, 2019 ; Spence et al., 2001 ). Accordingly, the following propositions are proposed for consideration in future research:

Proposition 1:

There are different ethical considerations in ventures depending on their nature: Formal or informal, the size of the company, and its relationship with stakeholders.

Gender Perspectives in Entrepreneurial Ethics

The emergence of women’s participation in entrepreneurship has raised interest among researchers regarding how gender variation is related to ethical and social value goals and discrimination. On the one hand, Hechavarría et al. ( 2017 ) found that women entrepreneurs are more likely than male entrepreneurs to emphasize social value goals over economic value creation goals. Other studies have suggested that females perceive ethics and socially responsible behaviors as more important (Ahmad et al., 2010 ) and engage more in social vocations than do their male counterparts (Berings et al., 2012 ). On the other hand, De Clercq et al. ( 2022 ) reported that women entrepreneurs indicate that their sense of work autonomy increases the likelihood that they are satisfied with their ability to balance the demands of their work with their personal lives and that this process is especially salient when they operate in countries characterized by discriminatory socioeconomic and institutional conditions. However, in culturally discriminatory environments, there is a mitigating rather than an invigorating effect.

Additionally, gender as a topic has been linked to religion, especially the Islamic religion (Althalathini et al., 2022 ; Gümüsay, 2015 ; Özkazanç-Pan, 2015 ; Ramadani et al., 2015 ; Tlaiss, 2015 ). Religious practices typically affect individual and societal perceptions of entrepreneurial activities (Gotsis et al., 2009 ). Studies of gender and entrepreneurship in the Arab world have addressed these problems and barriers and linked them to the teachings of Islam (e.g., Itani et al., 2011 ). In particular, Islam has been found to contribute to the systematic subordination of women in patriarchal societies. However, Islam neither forbids nor frowss women’s entrepreneurship; in fact, it is supported and encouraged (Tlaiss, 2015 ). From the Islamic perspective, entrepreneurship is thus based on the pursuit of opportunities, guided by a set of norms, values, and recommendations (socioeconomic/ethical), and has a religious-spiritual pillar that links people to God and to the ultimate goal of pleasing Allah (Gümüsay, 2015 ). Tlaiss ( 2015 ) studied how Islamic values and work ethics are embedded in the entrepreneurial activities of Arab women. Her results show how Muslim women entrepreneurs pursue well-being and excellence in their work while running their businesses. In addition, Muslim women entrepreneurs adhere to work-related Islamic values such as working effectively and hard, honesty and truthfulness, fairness and justice, and benevolence. Through these values and work ethics, Arab Muslim women structure and direct their entrepreneurial careers in ways that lie outside the traditional and doctrinaire interpretations of Islam. Similarly, Althalathini et al. ( 2022 ) explored how Islamic feminism empowers women entrepreneurs and their entrepreneurial activities and behaviors in conflict zones. They found that Islamic feminism shapes the business ethics of Muslim women entrepreneurs who are operating in conflict zones and removes the traditional, patriarchal, colonial, and other cultural restrictions that have veiled Islam. Similarly, Özkazanç-Pan ( 2015 ) proposed that through engagement in entrepreneurship, Islamic feminist positions enable praxis and represent an ethical–political commitment to dismantling neoliberal ideologies that perpetuate gender inequality.

Although gender is one of the most studied topics in the relationship between ethics and entrepreneurship, there are research questions with high potential for future studies. For example, how does work autonomy influence the work-life balance of women entrepreneurs, including factors such as time management, locus of control, and self-efficacy? How can these mechanisms foster ethical values in the work environment? How do female and male entrepreneurs experience and express traits such as resilience and power distance differently? What is the impact of Islam on female entrepreneurs in different countries, given their specific economic, political, and sociocultural contexts? How does the influence of Islam shape the personal lives and motivations of female entrepreneurs, including the challenges and obstacles they face in Middle Eastern countries with Islamic influence (Althalathini et al., 2022 ; De Clercq et al., 2022 ; Tlaiss, 2015 )? Therefore, the following propositions are presented for future research:

Proposition 2:

Different contexts (e.g., cultural, geographic, and religious) influence men’s and women’s personal life balance, motivations, and ethical values related to entrepreneurship.

Social Entrepreneurship

The thematic map of the EE literature highlights “social entrepreneurship” as a basic theme for the period 1998–2023 (Fig.  8 ). Research has shown a growing focus on social entrepreneurship due to its importance in addressing social problems and enriching communities and societies with market-based methods (e.g., Scuotto et al., 2022 ; Tate et al., 2018 ). In this regard, Hota et al. ( 2020 ) performed a bibliometric analysis of works published between 1996 and 2017 and highlighted the importance of ethics in social entrepreneurship, which is itself ethical in nature, differentiating it from commercial entrepreneurship.

Initially, this academic discussion focused on the ethical need to target social entrepreneurship to the social needs of stakeholders to contribute to their development, build legitimacy, and discover new opportunities (e.g., Harmeling et al., 2009 ; Renault, 2006 ). In addition, organizations related to social entrepreneurship, such as nongovernmental organizations and universities, have been analyzed (Easterly et al., 2009 ; Renault, 2006 ). Subsequently, Zahra et al. ( 2009 ) proposed the following definition: “ Social entrepreneurship encompasses the activities and processes undertaken to discover, define, and exploit opportunities to enhance social wealth by creating new ventures or managing existing organizations in an innovative manner” . These authors identified three types of social entrepreneurship: Social bricoleurs, who act on locally discovered opportunities using locally available resources; social constructionists, who build, launch, and operate ventures that address the social needs that existing organizations inadequately address; and social engineers, who identify systemic problems within social systems and structures and address them by bringing about revolutionary change.

More recently, the tension between social missions and values of social entrepreneurship and the allocation of efficiency, innovation, and resources among traditional business ventures has been discussed (Smith et al., 2013 ). Similarly, Pies et al. ( 2010 ) indicated that the purpose of businesses in society is value creation and that companies can help solve global problems through global corporate citizenship if they participate as political and moral actors. The discussion has also turned to the management of social entrepreneurship through its discussion of the following points: The challenge of maintaining social objectives during the scaling-up process (André et al., 2016 ; Smith et al., 2016 ); the analysis of entrepreneurial intention and the ethics of social enterprise business models (Bull et al., 2019 ); the characteristics of ethical leadership, including moral entrepreneurship (Kaptein, 2019 ); the relevance of social entrepreneurship practices to sustaining social contributions expressed in a business’s social mission (Bruder, 2021 ); and the development of social innovation and responsible innovation in ways that create socioethical value for target beneficiaries and achieve sustainable development goals (Lubberink et al., 2019 ).

In recent years, the discussion of social entrepreneurship has continued to deepen our understanding of the ethical and managerial aspects of social enterprises (e.g., Sengupta et al., 2022 ; Slavec Gomezel et al., 2023 ), and new themes have emerged. These themes include the “prosocial” approach, which implies the intent to help or promote the benefit of other persons, groups, or society (e.g., Figueroa-Armijos et al., 2022 ; Yitshaki et al., 2022 ); the role of social media in social enterprises (e.g., Jacobson et al., 2022 ); the use of a religious approach (e.g., Anglin et al., 2023 ; Dissanayake, 2022 ); and an incipient discussion of gender (e.g., Freund et al., 2023 ). Although social entrepreneurship is a basic topic in the literature, there are still open research questions, for example, Are there ethical differences in entrepreneurship and entrepreneurial decisions when gender, religion, and cultural dimensions are considered simultaneously? What are the ethical tensions in companies that have a “prosocial” focus on achieving financial stability? Is there an ethical distinction between social entrepreneurship and sustainable entrepreneurship? What occurs in ventures regarding their ethical decisions to use social media and the types of content available for communicating and promoting such ventures? (Anglin et al., 2023 ; Figueroa-Armijos et al., 2022 ; Yitshaki et al., 2022 ).

Proposition 3:

Social entrepreneurs have different ethical considerations according to the stage of the venture (born, growth, maturity); these factors affect their relationships with stakeholders, the tension between social purpose and financial needs, and the use of social media.

Integrating Sustainability and Corporate Social Responsibility in Entrepreneurial Ventures

Another of the most developed and relevant topics in the EE literature is the balance between profitability and ethics. This review shows that, in the early years, corporate social responsibility (CSR) was the concept used to discuss the topic in both large companies and entrepreneurship. For example, Choi et al. ( 2008 ) studied socially responsible entrepreneurs—who balance the goal of profit with CSR—and discovered that they typically founded their ventures, at least in part, to achieve idealistic goals and pursue both financial and nonfinancial objectives simultaneously. The construct of sustainability has subsequently gained consensus in the literature in reference to this debate. For example, Markman et al. ( 2016 ) presented a special issue on sustainability, ethics, and entrepreneurship, arguing that the balance among social, environmental, and economic responsibility is insufficient. Sustainable, ethical, and entrepreneurial enterprises should thus regenerate the environment and drive positive social change rather than merely minimize damage. The overlap between CSR and sustainability is not surprising, as the two share an ethical basis, the pursuit of responsible business conduct. At their core, and essential for this review, both CSR and sustainability have emerged as responses to the growing recognition of an approach that meets ethical principles and in which the creation of economic value is pursued while social and environmental responsibilities are met in a way that may even achieve social progress and preserve the environment (Choi et al., 2008 ; Elkington, 2006 ; Keijzers, 2002 ; Markman et al., 2016 ).

In the EE literature, several authors have studied the determinants of sustainability. Hechavarría et al. ( 2017 ), for instance, found that entrepreneurs in strong postmaterialist societies are more likely to have social and environmental value creation goals and less likely to have economic value creation goals. They also showed that women entrepreneurs are more likely than men to emphasize social value creation goals than economic value creation goals and that postmaterialists tend to further widen the gender gap in these value creation goals. In another prominent study, Rui et al. ( 2021 ) suggested that entrepreneurs’ responsible leadership and environmental awareness positively impact the relationships among stakeholder pressure, entrepreneurial environmental ethics, and green innovation. Their study thus highlights the importance of improving entrepreneurs’ environmental awareness and leadership in motivating employees’ pro-environmental behavior and appropriately allocating company resources to support green innovation.

In terms of context, two industries have been the most studied, not least because of their high social and environmental impact: Tourism and fashion. On the one hand, the implementation of sustainable and ethical practices in the tourism industry can minimize the negative impact of tourism on the environment, local communities, and cultural heritage (Ryan, 2002 ). Additionally, entrepreneurship can ensure the long-term preservation of the charm of tourist destinations and guarantee to those who invest in tourism that their capital will not be harmed by social and environmental deterioration. Sustainable tourism development allows the management of resources while ensuring that humanity can satisfy its economic, social, and aesthetic needs and, at the same time, preserve basic ecological movements, biological diversity, and life sustenance (Dávid, 2011 ; Power et al., 2017 ). On the other hand, the fashion industry is known for its negative environmental and social impacts, exemplified by pollution, waste, and poor labor practices. Therefore, many companies have attempted to establish themselves as sustainable fashion brands. Hence, Bandyopadhyay et al. ( 2020 ) proposed product authenticity, entrepreneurial processes and local and traditional aspects of products, including their designs and appealing backstories with market relevance, as positioning strategies. However, although transnational textile companies claim to have implemented corporate social responsibility processes to promote ethical behavior, Luque et al. ( 2019 ) noted that this claim is often only an exception. The rule behind the masking of marketing campaigns can be defined as corporate social irresponsibility.

As noted above, sustainability has been a motor theme in entrepreneurship and ethics research; however, the current global trend still presents research questions. For example, what are the local and global benefits and the potential ethical challenges associated with implementing circular economic strategies in different cultural and socioeconomic contexts? How does sustainability impact entrepreneurs’ financial performance in different sectors, such as retail, mining, and agriculture? What is the profile of entrepreneurs who address the United Nations Sustainable Development Goals as part of their business models? How do ethical considerations influence the adoption and effectiveness of green marketing strategies in entrepreneurial ventures? (Hechavarría et al., 2017 ; Pla-Julián et al., 2019 ; Ploum et al., 2019 ; Rui et al., 2021 ; Tzanidis et al., 2024 ). Therefore, the following proposition is presented for future research to consider:

Proposition 4:

The impact of and the return on entrepreneurial activity will depend on the balance between exploiting innovative sustainable development opportunities and the entrepreneurs’ legitimate right to a return on their investments.

Ethical Decision-Making in Entrepreneurship

Initially, it was thought that entrepreneurs have a different ethical orientation than other businesspeople and that their specific differences depend upon the economic, cultural, and religious contexts in which they operated. For example, Confucian entrepreneurs do not follow the traditional rational logic of business management, which can be very costly (Bucar et al., 2003 ; Cheung et al., 2004 ). Understanding the ethical dimension of entrepreneurs’ decision-making is also relevant, especially in contexts that involve uncertainty and ambiguity (McVea, 2009 ). Bryant ( 2009 ) highlighted that entrepreneurs with high self-regulation characteristics are more morally aware and seek to maintain their integrity and interpersonal trust-building. In contrast, entrepreneurs with less self-regulation seem more morally conscious and focus more on failure and loss. Baron et al. ( 2015 ) argued that entrepreneurs’ motivation for financial gain is related to their “moral disengagement” and tendency to make unethical decisions.

The following specific topics have also been addressed: The ethical decisions of small entrepreneurs linked to natural resources (Lahdesmaki, 2005 ); the characteristics of green entrepreneurship, in which different types can be distinguished, such as those that seek to improve environmental conditions through ethics; and other types of green entrepreneurs, such as innovative opportunists, visionary champions, and accidents (Taylor et al., 2004 ). Similarly, Rae ( 2010 ) proposed that university education in entrepreneurship should be directed toward new economic, social, and cultural challenges, whereby ethical and environmental concerns should be reflected in new social ventures.

In addition, studies that adopt specific approaches and have specific contexts have been conducted in recent years. For example, Zhu ( 2015 ) analyzed decision-making processes in Chinese ventures and proposed a vision that integrates Confucianism. Ahsan ( 2020 ) evaluated the ethical dimension of the collaborative economy or “gig economy” ventures by reviewing UBER and concluded that from the point of view of employment (drivers), the neoliberal regime has intensified, producing inequalities. Higgins-Desbiolles et al. ( 2021 ) analyzed the business events industry in Australia from the perspective of feminist ethics. Despite the development of this topic, there are still open research questions. For example, how do volatility, uncertainty, complexity, and ambiguity (VUCA—environment) affect the ethical decisions of entrepreneurs? What is the relationship among the different types of entrepreneurs (e.g., informal, hybrid, habitual, green, social) and ethics (Uriarte et al., 2023 ; Williams et al., 2016 )? What are the effects of teaching ethics on entrepreneurship in preschool, school, and high school? What are the ethical dimensions of specific areas of entrepreneurship, such as events, clothes, and tourism? Can the impact of the failure of entrepreneurial innovation decrease or increase the ethical standards of the entrepreneur (Scuotto et al., 2024 )? What are the ethical choices and characteristics of entrepreneurs who do business with companies and countries that do not respect human rights? What ethical decisions are involved in the exploration and exploitation stages, and what are their implications (Hägg et al., 2024 ; Higgins-Desbiolles et al., 2021 ; Rae, 2010 ; Zhu, 2015 )?

Proposition 5:

Entrepreneurship, the entrepreneurial process, and entrepreneurial responsibility produce different ethical dilemmas and interpretations according to ethical perspectives such as normative ethics, deontology, teleology and metaethics.

Ethical Challenges in the Age of Digital Innovation

The most relevant and emerging theme in the EE literature in recent years has been the emergence of new technologies and the new ethical problems that may arise from them. For example, Hall et al. ( 2006 ) investigated these new possibilities and the ethical dilemmas associated with technological progress. According to them, the emergence of new technologies can “change the rules of the game” (p.245). However, technical feasibility is not the only requirement for success, as the decisive factor in this case is the legitimacy provided by society.

In terms of the formative phase of the EE literature, Vanessa Ratten has been a pioneer in studies of a specific type of digital technology. She has evaluated the relationships among entrepreneurship, ethics, and cloud computing and e-book adoption. Cloud computing Footnote 5 creates ethical issues related to privacy, security, and anonymity (Jaeger et al., 2008 ; Ratten, 2012 ). Ratten concluded that a person’s ethical orientation influences his or her intention to adopt cloud computing (Ratten, 2012 ) and that marketing influences the person’s final decision to adopt it (Ratten, 2012 ). Ethical attitudes toward cloud computing are shaped by the false sense of reality provided by the internet, which allows one to adopt a virtual persona that inhibits the detection of misconduct. Similarly, although there is an ethical dilemma if the material presented is illegal, she found that an individual’s entrepreneurial orientation and the marketing of e-book devices influence his or her intention to adopt them (Ratten et al., 2011 ).

In recent years, the debate on EE has focused on signaling and identifying the ethical risks associated with new technologies such as artificial intelligence, big data, and blockchain. Only a few studies have advanced theories about the relationship between these technologies and entrepreneurial ethics. On the one hand, blockchain Footnote 6 technology is used by firms, startups, governments, and nongovernmental organizations (Kher et al., 2021 ) and presents challenges and ethical considerations. The challenges associated with blockchain include scalability, security, privacy, and energy consumption, while the ethical issues include privacy and the potential for use of the technology for illegal activities (Kher et al., 2021 ). However, blockchain may resolve ethical dilemmas. For example, most debates about the application of science that involves living subjects concern the dilemma of whether to allow research to use living subjects. Several startups (e.g., Nebula Genomics, LunaDNA, and EncrypGen) have proposed a technological solution using blockchain, allowing them to claim that their data collection and sharing procedures will advance scientific progress as well as protect the informational privacy of individuals (Racine, 2021 ). On the other hand, AI-based entrepreneurship presents new ethical challenges. These include concerns about privacy and data protection, bias and fairness, transparency and accountability, the ethical use of AI in decision-making, and safety and trust (Kamishima et al., 2018 ; Obschonka et al., 2020 ; Saheb et al., 2023 ). Despite the importance and explosive rise of artificial intelligence-based entrepreneurship, studies on the ethics of this branch of entrepreneurship remain scarce. Kamishima et al. ( 2018 ), for example, provided a framework, the capability-effectiveness approach, for the ethical development and implementation of AI robotics entrepreneurship that ensures that these technologies align with core human capabilities and ethical considerations. Thus, there is a large potential for new research questions on the topic; for example, how can the disclosure of relevant information to the public be balanced without exposing critical or sensitive data? What are the ethical aspects of governance when implementing blockchain or AI-based entrepreneurship in different sectors? How do human and organizational aspects influence the ethical implementation of emerging technologies in entrepreneurship? Can entrepreneurs be held accountable for the decisions made by AI and big data systems? How can entrepreneurs balance technological innovation with ethical considerations, such as algorithmic bias (Kamishima et al., 2018 ; Kher et al., 2021 ; Obschonka et al., 2020 ; Saheb et al., 2023 )? Therefore, the following proposition is presented for future research to consider:

Proposition 6:

AI-based entrepreneurs must consistently manage ethical dilemmas such as the use and collection of data related to privacy, informed consent, transparency in the use of data, and the ethical responsibility to protect user information at different stages of the entrepreneurship process.

Conclusions

The literature on entrepreneurship and ethics has experienced a boom in recent years, primarily due to new entrepreneurship challenges such as sustainability and social entrepreneurship and the emergence of new technologies such as artificial intelligence and big data. However, the publications related to EE are still fragmented, which justifies the need to systematize this work. Using a mixed approach—bibliographic and thematic evolution analysis—this study provides an overview of the business and social science literature on entrepreneurial ethics. In doing so, this research makes several theoretical, methodological, and managerial contributions.

First, for researchers in ethics and entrepreneurship, the results provide a dynamic research landscape in which the number of publications is increasing but the impact of individual publications is decreasing, possibly due to increased competition and saturation of the field. However, the significant proportion of publications on EE that appear in high-impact journals indicates a positive trajectory in the quality of the field and in its recognition and suggests that despite these challenges, the discipline is evolving toward greater academic rigor and influence. According to the relevant results of the performance analysis and the social structure, there are only ten coauthor clusters among the most influential academics, indicating that they primarily represent selective circles. However, single-authored publications represent approximately 25% of the total number of records in the dataset. This implies that the topic presents few barriers to entry but that few specialized authors have addressed it. Furthermore, with the exception of China, collaboration among authors in developing countries is scarce. Thus, these results should encourage more scholars to consider contributing to the current debate, especially in collaboration with coauthors in developing countries. For this purpose, international conferences are an effective source for networking research.

Second, for editors, as noted above, the Journal of Business Ethics is the most productive journal; it is more than nine times more productive than the next most productive journal. Other journals with a primary interest in business ethics are also among the most productive journals. This provides a challenge for journals focusing on entrepreneurship; those journals should include and encourage additional studies on ethics in entrepreneurship.

Third, this work is one of the first hybrid literature reviews of both the ethics and entrepreneurship literature, and it is certainly the first review that addresses the relationship between ethics and entrepreneurship. It thereby contributes to research at the methodological level in numerous ways. (i) This study broadens the coverage of published research by using a larger database, in this case, Scopus, rather than other databases, such as Web of Science (WoS), and by expanding the period under review. (ii) This study applied a protocol that was explicitly designed for the social sciences (SPAR-4-SLR), thereby promoting the transparency and reproducibility of the review results. (iii) This study uses modern software such as the Bibliometrix package in R, Rayyan and Stata 15. (iv) All the data are cross-referenced with the SCimago Journal & Country Rank database to determine the number of papers published in each quartile. This constitutes a new measure for the quality of publications in performance analysis. On the other hand, this study responds to calls to increase the quality of review studies in business and management (e.g., Alshater et al., 2023 ; Donthu et al., 2021 ; Lim et al., 2022 ; Paul et al., 2021 ; Robledo et al., 2023 ).

Nevertheless, this study has several methodological limitations that should be acknowledged. Although this study attempted to minimize the disadvantages of quantitative and qualitative research designs by bringing them together, qualitative analysis is especially subject to potential researcher bias. Additionally, because inclusion and exclusion criteria for the academic literature were established in this research, the analysis performed and the conclusions reached in this study are limited to papers that met the established criteria. Therefore, the conclusions cannot be applied to the entire body of studies on the role of ethics in entrepreneurship. Moreover, only papers written in English in the areas of business, management, accounting, or the social sciences were examined in this study. Therefore, the analysis did not include research reported in other languages or research that falls within other domains. These are some of the possible limitations of this study.

Fourth, this study makes several theoretical contributions to the field of entrepreneurial ethics. This research traces the field’s evolution through three phases—formative, consolidation, and pragmatic—emphasizing shifts from basic themes such as entrepreneurial ethics to more complex issues such as sustainability, digital technologies and innovation. This evolution highlights the increasing sophistication and integration of ethical considerations into entrepreneurial theory. By categorizing themes into motor, basic, niche and emerging/declining, the analysis provides a clear conceptual structure and offers a systematic approach to understanding the ethical dimensions of entrepreneurship and identifying gaps that should be addressed in future research.

In this regard, the analysis revealed that the relationship between dynamic capabilities and ethical performance in entrepreneurship highlights the role of ethical leadership in enhancing both ethical standards and competitive performance, indicating that ethical considerations vary according to the nature of ventures, including their formal and informal structures, their size, and their stakeholder relationships (Proposition 1 ). In addition, the analysis incorporates the influences of gender and religion and provides a theoretical basis for how these sociocultural factors shape ethical behavior in entrepreneurship, with different contexts influencing the personal life balance, motivations and ethical values of male and female entrepreneurs (Proposition 2 ). Furthermore, the analysis identifies social entrepreneurship as a basic theme, suggesting that ethical considerations change according to the stage of a business and its relationship with its stakeholders in a way that balances social purposes and financial needs (Proposition 3 ). The integration of sustainability and CSR as basic themes reinforces their theoretical importance in promoting ethical entrepreneurial practices and suggests that the impact and performance of entrepreneurial activity depend on the balance between innovative opportunities for sustainable development and the legitimate right to a return on investment (Proposition 4 ). Furthermore, the analysis reveals that the entrepreneurship process presents different ethical dilemmas and interpretations according to various ethical perspectives, such as normative ethics, deontology, teleology, and metaethics (Proposition 5 ). Finally, the emergence of digital technologies such as AI, blockchain, and big data is recognized as an emerging theme, and this requires a theoretical exploration of the ethical implications of these technologies, particularly with respect to data privacy, algorithmic fairness, and transparency in entrepreneurial ventures. The analysis proposes that AI-based entrepreneurs must consistently manage ethical dilemmas related to data privacy, informed consent, transparency, and the ethical responsibility to protect user information at different stages of the entrepreneurship process (Proposition 6 ).

Finally, from a management perspective, the analysis highlights the need for comprehensive ethical training programs that are tailored to the specific challenges faced in each sector and can enable entrepreneurs to improve their ethical decision-making. In addition, it emphasizes the strategic importance of integrating sustainability into entrepreneurial strategies by encouraging entrepreneurs to align ethical practices with long-term sustainability goals. The analysis also highlights the importance of gender-inclusive practices and policies, advocating for environments that support diversity and address gender-specific challenges. Additionally, it suggests that robust support mechanisms for social entrepreneurs, including resources, mentoring, and funding opportunities, be provided to help them navigate ethical challenges while achieving social goals. The focus on ethical leadership highlights the need to cultivate transparency, accountability, and integrity within organizational cultures. Finally, given the importance of digital technologies, the analysis advises managers to develop detailed ethical guidelines for using AI, blockchain, and other innovations, ensuring their responsible use in a way that sustains ethical standards in modern entrepreneurial businesses.

The Louvain algorithm is a method used to identify groups or communities within large networks. It starts with each node (or point) as its own group. Then, it gradually combines these nodes into larger groups based on how closely they are connected, aiming to maximize the number of connections within each group compared to the connections between different groups. This process continues until no further improvements can be made, resulting in a clear picture of the clusters within the network.

To conduct this cross-check, this study downloaded, exported, and removed all duplicate and incomplete references from all historical data in both the WoS (35,747) and Scopus (53,063) databases that included the word “entrepreneurship” in the abstract, title, or list of keywords in all areas up to June 2023. This study then cross-referenced the databases using the DOI indicator and the titles of the papers (24,447). Subsequently, using the DOI, this study queried the Scopus database for all papers that were not found in the previous search (8,105). The total number of papers found in the two databases was 32,552.

All documents in this review are available in the online supplement (Appendix 1).

The data series describing the total number of papers per year was used to determine the breakpoints. The procedure was as follows: (a) The series was reviewed graphically; three cutoffs (2009, 2013, and 2018) were visually observed. (b) Statistical tests were performed on structural breaks to determine the number of cutoffs. This test yielded only two statistically significant cutoffs. (c) The next test was adjusted to two breaks. This test yielded the proposed results for the years 2013 and 2018. (d) Statistical tests were performed using these two years as structural cutoffs ( p  < 0.01). (e) Tests were performed with other breaks. These tests were also positive for 2000 and 2009. (f) The researchers evaluated the tests performed; the breaks were determined to have occurred in 2013 and 2018.

“computing platform that is able to dynamically provide, configure and reconfigure servers to address a wide range of needs, ranging from scientific research to e-commerce” (Jaeger et al., 2008 , p. 269).

“an open, distributed ledger that can record transactions between two parties efficiently and in a verifiable and permanent way” (Iansiti et al., 2017 , p. 4).

Acs, Z. J., Desai, S., & Hessels, J. (2008). Entrepreneurship, economic development and institutions. Small business economics (pp. 219–234). NY: Springer.

Google Scholar  

Ahmad, N. H., & Ramayah, T. (2012). Does the notion of ‘doing well by doing good’ prevail among entrepreneurial ventures in a developing nation? Journal of Business Ethics, 106 (4), 479–490.

Article   Google Scholar  

Ahmad, N. H., & Seet, P. S. (2010). Gender variations in ethical and socially responsible considerations among SME entrepreneurs in Malaysia. International Journal of Business and Society, 11 (1), 77–88.

Ahsan, M. (2020). Entrepreneurship and ethics in the sharing economy: A critical perspective. Journal of Business Ethics, 161 (1), 19–33.

Alshater, M. M., Joshipura, M., Khoury, R. E., & Nasrallah, N. (2023). Initial coin offerings: A hybrid empirical review. Small business economics (pp. 891–903). NY: Springer.

Althalathini, D., Al-Dajani, H., & Apostolopoulos, N. (2022). The impact of islamic feminism in empowering women’s entrepreneurship in conflict zones: Evidence from Afghanistan, Iraq and Palestine. Journal of Business Ethics, 178 (1), 39–55.

André, K., & Pache, A. C. (2016). From caring entrepreneur to caring enterprise: Addressing the ethical challenges of scaling up social enterprises. Journal of Business Ethics, 133 (4), 659–675.

Anglin, A. H., Milanov, H., & Short, J. C. (2023). Religious expression and crowdfunded microfinance success: Insights from role congruity theory. Journal of Business Ethics, 185 (2), 397–426.

Arend, R. J. (2013). Ethics-focused dynamic capabilities: A small business perspective. Small Business Economics, 41 (1), 1–24.

Baier-Fuentes, H., Merigó, J. M., Amorós, J. E., & Gaviria-Marín, M. (2019). International entrepreneurship: A bibliometric overview. The International Entrepreneurship and Management Journal, 15 (2), 385–429.

Baier-Fuentes, H., González-Serrano, M. H., Alonso-Dos Santos, M., Inzunza-Mendoza, W., & Pozo-Estrada, V. (2020). Emotions and sport management: A bibliometric overview. Frontiers in Psychology, 11 , 1512.

Bandyopadhyay, C., & Ray, S. (2020). Finding the sweet spot between ethics and aesthetics: A social entrepreneurial perspective to sustainable fashion brand (Juxta)positioning. Journal of Global Marketing, 33 (5), 377–395.

Baron, R. A., Zhao, H., & Miao, Q. (2015). Personal motives, moral disengagement, and unethical decisions by entrepreneurs: Cognitive mechanisms on the “slippery slope.” Journal of Business Ethics, 128 (1), 107–118.

Belas, J., Škare, M., Gavurova, B., Dvorsky, J., & Kotaskova, A. (2022). The impact of ethical and CSR factors on engineers’ attitudes towards SMEs sustainability. Journal of Business Research, 149 , 589–598.

Berings, D., & Adriaenssens, S. (2012). The role of business ethics, personality, work values and gender in vocational interests from adolescents. Journal of Business Ethics, 106 (3), 325–335.

Bhukya, R., & Paul, J. (2023). Social influence research in consumer behavior: What we learned and what we need to learn?–A hybrid systematic literature review. Journal of Business Research, 162 , 113870.

Blondel, V. D., Guillaume, J.-L., Lambiotte, R., & Lefebvre, E. (2008). Fast unfolding of communities in large networks. Journal of Statistical Mechanics: Theory and Experiment, 2008 (10), P10008.

Brenkert, G. G. (2009). Innovation, rule breaking and the ethics of entrepreneurship. Journal of Business Venturing, 24 (5), 448–464.

Bruder, I. (2021). A social mission is not enough: reflecting the normative foundations of social entrepreneurship. Journal of Business Ethics, 174 (3), 487–505.

Bryant, P. (2009). Self-regulation and moral awareness among entrepreneurs. Journal of Business Venturing, 24 (5), 505–518.

Bucar, B., Glas, M., & Hisrich, R. D. (2003). Ethics and entrepreneurs: An international comparative study. Journal of Business Venturing, 18 (2), 261–281.

Bull, M., & Ridley-Duff, R. (2019). Towards an appreciation of ethics in social enterprise business models. Journal of Business Ethics, 159 (3), 619–634.

Callon, M., Courtial, J. P., & Laville, F. (1991). Co-word analysis as a tool for describing the network of interactions between basic and technological research: The case of polymer chemsitry. Scientometrics, 22 , 155–205.

Carr, P. (2003). Revisiting the protestant ethic and the spirit of capitalism: Understanding the relationship between ethics and enterprise. Journal of Business Ethics, 47 (1), 7–16.

Cheung, T. S., & King, A. Y. C. (2004). Righteousness and profitableness: The moral choices of contemporary Confucian entrepreneurs. Journal of Business Ethics, 54 (3), 245–260.

Choi, D. Y., & Gray, E. R. (2008). Socially responsible entrepreneurs: What do they do to create and build their companies? Business Horizons, 51 (4), 341–352.

Clarke, J., & Holt, R. (2010). Reflective judgement: Understanding entrepreneurship as ethical practice. Journal of Business Ethics, 94 (3), 317–331.

Cobo, M. J., López-Herrera, A. G., Herrera-Viedma, E., & Herrera, F. (2011). An approach for detecting, quantifying, and visualizing the evolution of a research field: A practical application to the Fuzzy sets theory field. Journal of Informetrics, 5 (1), 146–166.

Cumming, D., Hou, W., & Lee, E. (2016). Sustainable and ethical entrepreneurship, corporate finance and governance, and institutional reform in China. Journal of Business Ethics, 134 (4), 505–508.

Cumming, D., Hornuf, L., Karami, M., & Schweizer, D. (2023). Disentangling crowdfunding from fraudfunding. Journal of Business Ethics, 182 (4), 1103–1128.

Daradkeh, M. (2023). Navigating the complexity of entrepreneurial ethics: A systematic review and future research agenda. Sustainability, 15 (14), 11099.

Dávid, L. (2011). Tourism ecology: Towards the responsible, sustainable tourism future. Worldwide Hospitality and Tourism Themes, 3 (3), 210–216.

De Clercq, D., & Brieger, S. A. (2022). When discrimination is worse, autonomy is key: How women entrepreneurs leverage job autonomy resources to find work-life balance. Journal of Business Ethics, 177 (3), 665–682.

Di Lorenzo, F., & Scarlata, M. (2019). Social enterprises, venture philanthropy and the alleviation of income inequality. Journal of Business Ethics, 159 (2), 307–323.

Dissanayake, S. (2022). The Impact of Buddhist religion on entrepreneurial intention. J. Manage. Spirit. Relig., 19 (3), 302–319.

Donthu, N., Kumar, S., Mukherjee, D., Pandey, N., & Lim, W. M. (2021). How to conduct a bibliometric analysis: An overview and guidelines. Journal of Business Research, 133 , 285–296.

Easterly, L., & Miesing, P. (2009). Ngos, social venturing, and community citizenship behavior. Business and Society, 48 (4), 538–564.

Elkington, J. (2006). Governance for sustainability. Corporate Governance: An International Review, 14 (6), 522–529.

Fassin, Y. (2000). Innovation and ethics ethical considerations in the innovation business. Journal of Business Ethics, 27 (1), 193–203.

Figueroa-Armijos, M., & Berns, J. P. (2022). Vulnerable populations and individual social responsibility in prosocial crowdfunding: Does the framing matter for female and rural entrepreneurs? Journal of Business Ethics, 177 (2), 377–394.

Freund, D., RamírezGarcía, I., Boluk, K. A., Canut-Cascalló, M., & López-Planas, M. (2023). Exploring the gendered tourism entrepreneurial ecosystem in Barcelona and responses required: A feminist ethic of care. Journal of Sustainable Tourism . https://doi.org/10.1080/09669582.2023.2207780

Gedeon, S. (2010). What is entrepreneurship. Entrepreneurial Practice Review, 1 (3), 16–35.

GEM. (2024). Global Entrepreneurship Monitor, How GEM defines entrepreneurship . https://gemconsortium.org/

Gotsis, G., & Kortezi, Z. (2009). The impact of Greek Orthodoxy on entrepreneurship: A theoretical framework. Journal of Enterprising Communities, 3 (2), 152–175.

Gümüsay, A. A. (2015). Entrepreneurship from an Islamic perspective. Journal of Business Ethics, 130 (1), 199–208.

Hägg, G., Haataja, V., Kurczewska, A., & McKelvie, A. (2024). Entrepreneurial responsibility: A conceptual framework to understand ethical dualism throughout the entrepreneurial process. Entrepreneurship Theory and Practice . https://doi.org/10.1177/10422587241227072

Hall, J., & Rosson, P. (2006). The impact of technological turbulence on entrepreneurial behavior, social norms and ethics: Three internet-based cases. Journal of Business Ethics, 64 (3), 231–248.

Hall, J. K., Daneke, G. A., & Lenox, M. J. (2010). Sustainable development and entrepreneurship: Past contributions and future directions. Journal of Business Venturing, 25 (5), 439–448.

Hannafey, F. T. (2003). Entrepreneurship and ethics: A literature review. Journal of Business Ethics, 46 (2), 99–110.

Harmeling, S. S., Sarasvathy, S. D., & Freeman, R. E. (2009). Related debates in ethics and entrepreneurship: Values, opportunities, and contingency. Journal of Business Ethics, 84 (3), 341–365.

Hechavarría, D. M., Terjesen, S. A., Ingram, A. E., Renko, M., Justo, R., & Elam, A. (2017). Taking care of business: The impact of culture and gender on entrepreneurs’ blended value creation goals. Small Business Economics, 48 (1), 225–257.

Higgins-Desbiolles, F., & Monga, M. (2021). Transformative change through events business: A feminist ethic of care analysis of building the purpose economy. Events and sustainable (pp. 1989–2007). UK: Routledge.

Hoang, G., Luu, T. T., Du, T., & Nguyen, T. T. (2023). Can both entrepreneurial and ethical leadership shape employees’ service innovative behavior? Journal of Services Marketing, 37 (4), 446–463.

Hota, P. K., Subramanian, B., & Narayanamurthy, G. (2020). Mapping the intellectual structure of social entrepreneurship research: A citation/co-citation analysis. Journal of Business Ethics, 166 (1), 89–114.

Iansiti, M., & Lakhani, K. R. (2017). The truth about blockchain. Harvard Business Review, 95 (1), 118–127.

Itani, H., Sidani, Y. M., & Baalbaki, I. (2011). United Arab Emirates female entrepreneurs: Motivations and frustrations. Equality, Diversity and Inclusion: An International Journal, 30 (5), 409–424.

Iversen, J., Jørgensen, R., & Malchow-Møller, N. (2007). Defining and measuring entrepreneurship. Foundations and Trends® in Entrepreneurship, 4 (1), 1–63.

Jacobson, J., & Harrison, B. (2022). Sustainable fashion social media influencers and content creation calibration. International Journal of Advertising, 41 (1), 150–177.

Jaeger, P. T., Lin, J., & Grimes, J. M. (2008). Cloud computing and information policy: Computing in a policy cloud? Journal of Information Technology and Politics, 5 (3), 269–283.

Jancenelle, V. E., & Javalgi, R. G. (2018). The effect of moral foundations in prosocial crowdfunding. International Small Business Journal: Researching Entrepreneurship, 36 (8), 932–951.

Kamishima, Y., Gremmen, B., & Akizawa, H. (2018). Can merging a capability approach with effectual processes help us define a permissible action range for AI robotics entrepreneurship? Philosophy of Management, 17 (1), 97–113.

Kaptein, M. (2019). The moral entrepreneur: A new component of ethical leadership. Journal of Business Ethics, 156 (4), 1135–1150.

Keijzers, G. (2002). The transition to the sustainable enterprise. Journal of Cleaner Production, 10 (4), 349–359.

Kher, R., Terjesen, S., & Liu, C. (2021). Blockchain, Bitcoin, and ICOs: A review and research agenda. Small Business Economics, 56 (4), 1699–1720.

Knieps, G. (2023). The governance of big data and artificial intelligence in network industries. Competition and Regulation in Network Industries, 24 , 57–71.

Lahdesmaki, M. (2005). When ethics matters—Interpreting the ethical discourse of small nature-based entrepreneurs. Journal of Business Ethics, 61 (1), 55–68.

Lashitew, A. A., Narayan, S., Rosca, E., & Bals, L. (2022). Creating social value for the ‘base of the pyramid’: An integrative review and research agenda. Journal of Business Ethics, 178 (2), 445–466.

Lee, Y., Lee, Y., Seong, J., Stanescu, A., & Hwang, C. S. (2020). A comparison of network clustering algorithms in keyword network analysis: A case study with geography conference presentations. International Journal of Geospatial and Environmental Research, 7 (3), 1.

Lim, W. M., Rasul, T., Kumar, S., & Ala, M. (2022). Past, present, and future of customer engagement. Journal of Business Research, 140 , 439–458.

Long, T. B., Blok, V., Dorrestijn, S., & Macnaghten, P. (2020). The design and testing of a tool for developing responsible innovation in start-up enterprises. Journal Responsible Innovation, 7 (1), 45–75.

Longenecker, J. G., McKinney, J. A., & Moore, C. W. (1988). Egoism and independence: Entrepreneurial ethics. Organizational Dynamics, 16 (3), 64–72.

Lubberink, R., Blok, V., van Ophem, J., & Omta, O. (2019). Responsible innovation by social entrepreneurs: An exploratory study of values integration in innovations. J. Responsible Innov., 6 (2), 179–210.

Luque, A., & Herrero-García, N. (2019). How corporate social (ir)responsibility in the textile sector is defined, and its impact on ethical sustainability: An analysis of 133 concepts. Corp Social Responsibility Environmental Management, 26 (6), 1285–1306.

Magni, D., Palladino, R., Papa, A., & Cailleba, P. (2022). Exploring the journey of responsible business model innovation in Asian companies: A review and future research agenda . NY: Springer.

Markman, G. D., Russo, M., Lumpkin, G. T., Jennings, P. D., & Mair, J. (2016). Entrepreneurship as a platform for pursuing multiple goals: A special issue on sustainability, ethics, and entrepreneurship. Journal of Management Studies, 53 (5), 673–694.

Markman, G. D., Waldron, T. L., Gianiodis, P. T., & Espina, M. I. (2019). E pluribus unum: Impact entrepreneurship as a solution to grand challenges. Academy of Management Perspectives, 33 (4), 371–382.

McVea, J. F. (2009). A field study of entrepreneurial decision-making and moral imagination. Journal of Business Venturing, 24 (5), 491–504.

Morris, M. H., Schindehutte, M., Walton, J., & Allen, J. (2002). The ethical context of entrepreneurship: Proposing and testing a developmental framework. Journal of Business Ethics, 40 (4), 331–361.

Notzke, C. (2019). Equestrian tourism: Animal agency observed. Current Issues Tourism, 22 (8), 948–966.

Nylund, P. A., Ferràs-Hernández, X., Pareras, L., & Brem, A. (2022). The emergence of entrepreneurial ecosystems based on enabling technologies: Evidence from synthetic biology. Journal of Business Research, 149 , 728–735.

Obschonka, M., & Audretsch, D. B. (2020). Artificial intelligence and big data in entrepreneurship: A new era has begun. Small Business Economics, 55 (3), 529–539.

Özkazanç-Pan, B. (2015). Secular and Islamic feminist entrepreneurship in Turkey. International Journal of Gender and Entrepreneurship, 7 (1), 45–65.

Paul, J., & Criado, A. R. (2020). The art of writing literature review: What do we know and what do we need to know? International Business Review, 29 (4), 101717.

Paul, J., Lim, W. M., O’Cass, A., Hao, A. W., & Bresciani, S. (2021). Scientific procedures and rationales for systematic literature reviews (SPAR-4-SLR). International Journal of Consumer Studies, 45 (4), O1-6.

Pedroletti, D., & Ciabuschi, F. (2023). Reshoring: A review and research agenda. Journal of Business Research, 164 , 114005.

Pies, I., Beckmann, M., & Hielscher, S. (2010). Value creation, management competencies, and global corporate citizenship: An ordonomic approach to business ethics in the age of globalization. Journal of Business Ethics, 94 (2), 265–278.

Pla-Julián, I., & Guevara, S. (2019). Is circular economy the key to transitioning towards sustainable development? Challenges from the perspective of care ethics. Futures, 105 , 67–77.

Ploum, L., Blok, V., Lans, T., & Omta, O. (2019). Educating for self-interest or -transcendence? An empirical approach to investigating the role of moral competencies in opportunity recognition for sustainable development. Business Ethics, 28 (2), 243–260.

Power, S., Di Domenico, M., & Miller, G. (2017). The nature of ethical entrepreneurship in tourism. Annals of Tourism Research, 65 , 36–48.

Racine, V. (2021). Can blockchain solve the dilemma in the ethics of genomic biobanks? Science and Engineering Ethics, 27 (3), 35.

Rae, D. (2010). Universities and enterprise education: Responding to the challenges of the new era. Journal of Small Business and Enterprise Development, 17 (4), 591–606.

Ramadani, V., Dana, L. P., Ratten, V., & Tahiri, S. (2015). The context of Islamic entrepreneurship and business: Concept, principles and perspectives. International Journal of Business and Globalisation, 15 (3), 244–261.

Ratten, V. (2012). Entrepreneurial and ethical adoption behaviour of cloud computing. The Journal of High Technology Management Research, 23 (2), 155–164.

Ratten, V., Palumbo, A. J., & Donahue, J. F. (2011). Ethics, entrepreneurship and the adoption of e-book devices. International Journal of Innovation Learning, 10 (3), 310–325.

Renault, C. S. (2006). Academic capitalism and university incentives for faculty entrepreneurship. The Journal of Technology Transfer, 31 (2), 227–239.

Ribeiro-Soriano, D. (2017). Small business and entrepreneurship: Their role in economic and social development (pp. 1–3). NY: Taylor and Francis.

Robledo, S., Grisales Aguirre, A. M., Hughes, M., & Eggers, F. (2023). “Hasta la vista, baby”–Will machine learning terminate human literature reviews in entrepreneurship? Journal of Small Business Management, 61 (3), 1314–1343.

Rui, Z., & Lu, Y. (2021). Stakeholder pressure, corporate environmental ethics and green innovation. Asian Journal of Technology Innovation, 29 (1), 70–86.

Ryan, C. (2002). Equity, management, power sharing and sustainability-Issues of the ‘new tourism.’ Tourism Management, 23 (1), 17–26.

Saheb, T., & Mamaghani, F. H. (2021). Exploring the digital business ecosystem of the internet of things in emerging economies with a focus on the role of pseudo-private companies (pp. 1–21). Australia: Australasian Computer Society.

Saheb, T., & Saheb, T. (2023). Topical review of artificial intelligence national policies: A mixed method analysis . Amsterdam: Elsevier.

Scuotto, V., Lemaire, S. L. L., Magni, D., & Maalaoui, A. (2022). Extending knowledge-based view: Future trends of corporate social entrepreneurship to fight the gig economy challenges. Journal of Business Research, 139 , 1111–1122.

Scuotto, V., Magni, D., Garcia-Perez, A., & Pironti, M. (2024). The impact of innovation failure: Entrepreneurship adversity or opportunity? Technovation, 131 , 102944.

Sengupta, S., & Lehtimäki, H. (2022). Contextual understanding of care ethics in social entrepreneurship. Entrepreneurship and Regional Development, 34 (5–6), 402–433.

Sharma, G. D., Verma, M., Taheri, B., Chopra, R., & Parihar, J. S. (2023). Socio-economic aspects of hydrogen energy: An integrative review. Technological Forecasting and Social Change, 192 , 122574.

Skinner, H. (2019). The impact of cultural values and economic constraints on tourism businesses’ ethical practices. International Journal of Tour Cities, 5 (2), 169–187.

SlavecGomezel, A., & Stritar, R. (2023). Does it pay to be an ethical leader in entrepreneurship? An investigation of the relationships between entrepreneurs’ regulatory focus, ethical leadership, and small firm growth. Review of Managerial Science, 17 (1), 155–173.

Smith, W. K., Gonin, M., & Besharov, M. L. (2013). Managing social-business tensions: A review and research agenda for social enterprise. Business Ethics Quarterly, 23 (3), 407–442.

Smith, B. R., Kistruck, G. M., & Cannatelli, B. (2016). The impact of moral intensity and desire for control on scaling decisions in social entrepreneurship. Journal of Business Ethics, 133 (4), 677–689.

Spence, L. J., & Rutherfoord, R. (2001). Social responsibility, profit maximisation and the small firm owner-manager. Journal of Small Business and Enterprise Development, 8 (2), 126–139.

Stephan, U., Patterson, M., Kelly, C., & Mair, J. (2016). Organizations driving positive social change: A review and an integrative framework of change processes. Journal of Management, 42 (5), 1250–1281.

Tate, W. L., & Bals, L. (2018). Achieving shared triple bottom line (TBL) value creation: Toward a social resource-based view (SRBV) of the firm. Journal of Business Ethics, 152 (3), 803–826.

Taylor, D. W., & Walley, E. E. (2004). The green entrepreneur: Opportunist, Maverick or Visionary? International Journal of Entrepreneurship Small Business, 1 (1–2), 56–69.

Tlaiss, H. A. (2015). How islamic business ethics impact women entrepreneurs: Insights from four arab middle eastern countries. Journal of Business Ethics, 129 (4), 859–877.

Tzanidis, T., Magni, D., Scuotto, V., & Maalaoui, A. (2024). B2B green marketing strategies for European firms: Implications for people, planet and profit. Industrial Marketing Management, 117 , 481–492.

Uriarte, S., Mandakovic, V., & Amorós, J. E. (2023). Hybrid entrepreneurs: The value of experience. Journal of Small Business Management . https://doi.org/10.1080/00472778.2023.2284911

Vallaster, C., Kraus, S., Lindahl, J. M. M., & Nielsen, A. (2019). Ethics and entrepreneurship: A bibliometric study and literature review. Journal of Business Research, 99 , 226–237.

Van Praag, C. M., & Versloot, P. H. (2007). What is the value of entrepreneurship? A review of recent research. Small Business Economics, 29 (4), 351–382.

Vyakarnam, S., Bailey, A., Myers, A., & Burnett, D. (1997). Towards an understanding of ethical behaviour in small firms. Journal of Business Ethics, 16 (15), 1625–1636.

Williams, C. C., & Shahid, M. S. (2016). Informal entrepreneurship and institutional theory: Explaining the varying degrees of (in)formalization of entrepreneurs in Pakistan. Entrepreneurship and Regional Development, 28 (1–2), 1–25.

Yitshaki, R., Kropp, F., & Honig, B. (2022). The role of compassion in shaping social entrepreneurs’ prosocial opportunity recognition. Journal of Business Ethics, 179 (2), 617–647.

Zahra, S. A., Gedajlovic, E., Neubaum, D. O., & Shulman, J. M. (2009). A typology of social entrepreneurs: Motives, search processes and ethical challenges. Journal of Business Venturing, 24 (5), 519–532.

Zhang, Z., & Arvey, R. D. (2009). Rule breaking in adolescence and entrepreneurial status: An empirical investigation. Journal of Business Venturing, 24 (5), 436–447.

Zhang, J., Fei, J., Song, X., & Feng, J. (2021). An improved Louvain algorithm for community detection. Mathematical Problems in Engineering, 2021 , 1–14.

Zhu, Y. (2015). The role of Qing (Positive Emotions) and Li 1 (Rationality) in Chinese entrepreneurial decision making: A Confucian Ren-Yi wisdom perspective. Journal of Business Ethics, 126 (4), 613–630.

Download references

Author information

Authors and affiliations.

School of Business, Universidad Adolfo Ibañez, Diagonal Las Torres 2640, Peñalolen, Santiago, Chile

Sebastián Uriarte

Faculty of Economics and Business, Universidad Alberto Hurtado, Erasmo Escala 1835, Santiago, Chile

Cristian Geldes

Universidad de Chile, Av. Libertador Bernardo O’Higgins 1058, Santiago, Chile

Jesús Santorcuato

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Sebastián Uriarte .

Ethics declarations

Conflicts of interest.

The authors declare that they have no conflicts of interest.

Ethical Approval

This article does not contain any studies involving human or animals performed by any of the authors.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 71 KB)

Rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/ .

Reprints and permissions

About this article

Uriarte, S., Geldes, C. & Santorcuato, J. Evolution of Ethics and Entrepreneurship: Hybrid Literature Review and Theoretical Propositions. J Bus Ethics (2024). https://doi.org/10.1007/s10551-024-05815-8

Download citation

Received : 08 March 2024

Accepted : 28 August 2024

Published : 11 September 2024

DOI : https://doi.org/10.1007/s10551-024-05815-8

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Entrepreneurship
• Hybrid literature review
• Thematic analysis
• Research agenda
• Find a journal
• Publish with us
• Track your research

• Open access
• Published: 09 September 2024

Clinician voices on ethics of LLM integration in healthcare: a thematic analysis of ethical concerns and implications

• Tala Mirzaei 1 ,
• Leila Amini 1 &
• Pouyan Esmaeilzadeh 1  

BMC Medical Informatics and Decision Making volume  24 , Article number:  250 ( 2024 ) Cite this article

138 Accesses

Metrics details

This study aimed to explain and categorize key ethical concerns about integrating large language models (LLMs) in healthcare, drawing particularly from the perspectives of clinicians in online discussions.

Materials and methods

We analyzed 3049 posts and comments extracted from a self-identified clinician subreddit using unsupervised machine learning via Latent Dirichlet Allocation and a structured qualitative analysis methodology.

Analysis uncovered 14 salient themes of ethical implications, which we further consolidated into 4 overarching domains reflecting ethical issues around various clinical applications of LLM in healthcare, LLM coding, algorithm, and data governance, LLM’s role in health equity and the distribution of public health services, and the relationship between users (human) and LLM systems (machine).

Mapping themes to ethical frameworks in literature illustrated multifaceted issues covering transparent LLM decisions, fairness, privacy, access disparities, user experiences, and reliability.

This study emphasizes the need for ongoing ethical review from stakeholders to ensure responsible innovation and advocates for tailored governance to enhance LLM use in healthcare, aiming to improve clinical outcomes ethically and effectively.

Peer Review reports

Introduction

The development of Large Language Models (LLMs) marks a significant advancement in integrating deep learning techniques within natural language processing (NLP), enhancing the field’s capabilities far beyond traditional methods. This represents a deepening of the synergy between cutting-edge artificial intelligence (AI) technologies and foundational NLP approaches.These generative AI models, trained on vast datasets, have demonstrated remarkable proficiency in generating text virtually indistinguishable from human-authored content [ 1 ]. This transformative potential extends across diverse domains, including healthcare, where they offer the capacity to process, analyze, and generate insights from extensive textual healthcare data. Notably, recent studies have shown that technologies like ChatGPT (an LLM developed by OpenAI) outperform physician-patient communication in terms of both quality and empathy [ 2 ].

However, integrating LLMs into sensitive areas like healthcare has several challenges [ 3 ]. These models may occasionally produce inaccurate or biased responses [ 4 ]. In medical informatics, such inaccuracies can have far-reaching consequences, potentially resulting in physical and psychological harm, as well as inappropriate changes in treatment or patient adherence [ 5 , 6 ]. It is essential to recognize that using LLM-driven recommendations in healthcare differs significantly from other sectors due to the highly sensitive nature of health information and the heightened vulnerability of consumers to potential medical errors [ 3 ]. LLMs, designed for plausibility rather than absolute accuracy, do not inherently verify the truthfulness of their output [ 7 , 8 ]. Additionally, while valuable, their tuning through human feedback is not infallible [ 9 , 10 ].

The primary objective of this research is to investigate and elucidate the ethical complexities inherent in integrating LLMs within healthcare settings, with a specific focus on the perspectives of clinical practitioners. Our objective is to categorize and critically analyze the ethical challenges that emerge from the deployment of LLMs in healthcare, thereby contributing to a more informed understanding within the field. Overlooking these ethical concerns could result in missed opportunities to harness LLMs effectively, including optimizing processes like triage, screening, and treatment administration [ 11 , 12 , 13 ], while also offering the potential to democratize healthcare access through the direct provision of AI-driven healthcare services to patients’ homes. [ 14 , 15 , 16 ] Ethical lapses or misinterpretations may result in societal resistance or the formation of skewed regulations and policies, thereby hindering the advancement and acceptance of vital data science applications in healthcare [ 8 ]. This study underscores the transformative potential of LLMs in healthcare, contingent upon their implementation being guided by a thorough understanding of the ethical implications.

Material and method

This study aimed to elucidate emergent themes within the “medicine” subreddit ( https://www.reddit.com/r/medicine/ ), a digital platform specifically for medical professionals. The subreddit comprises more than 465,000 members, including self-identified physicians and other healthcare professionals from across the globe. This subreddit mandates that all users must set a flair accurately reflecting their role in the healthcare system. All posts and comments are in the English language. We considered this subreddit more suitable for academic research than other social media platforms, such as Twitter/X, due to its community structure and the nature of interactions within the platform. The specialized community in this subreddit enables us to easily find and engage with content that is highly relevant to the topic of the study. The format of Reddit supports longer, more in-depth discussions. In addition, this subreddit has a dedicated moderator and established rules for posting, so off-topic posts and spam are often quickly removed.

We adopted the Sequential Explanatory Method [ 17 ], which represents a mixed methods research design characterized by an initial phase of quantitative data collection and analysis, followed by a subsequent phase of qualitative analysis. The research design pipeline is shown in Fig.  1 .

We initially applied an unsupervised machine learning technique to uncover prominent latent topics within the large-scale unstructured data. Subsequently, we employed thematic analysis to further refine these topics. This involved a detailed examination of the data, including the inductive development of themes based on the latent topics identified by the machine learning algorithms [ 18 , 19 ]. Our primary focus was on individual posts, each typically hosting multiple user comments pertaining to the thread’s subject matter (e.g., “ChatGPT in medicine” posted on February 21, 2023).

Method overview

Data collection

Data for this study were collected over 12 months, from November 2022 to November 2023. This timeframe corresponds with the release of ChatGPT by OpenAI in November 2022, a refined version of the GPT-3 LLM optimized for conversational responses. The period is characterized by rapid user adoption of LLMs, providing a relevant context for examining its impact and usage [ 20 ]. We employed a selection of keywords to specifically extract the viewpoints of the subreddit members regarding ethical issues of integration of LLM and AI in healthcare. To select the keywords, we included broad terms such as “artificial intelligence”, “Large Language Model” and “ethical AI”, ensuring we cover variations and abbreviations of these terms to consider different ways of expressing the same idea. We included the specific model names such as “ChatGPT” and “GPT-3”. We also considered related concepts such as “privacy” and “transparency” to gather the relevant information about the topic. The complete list of keywords is provided in the supplemental material . For data extraction, we utilized the Python Reddit API Wrapper (PRAW) to interface with Reddit’s API, capturing post URLs, timestamps, the textual content of the posts, and the text from associated threads.

Data processing techniques

We employed the Natural Language Toolkit (NLTK) library (version 3.8.1) for processing textual data [ 21 ]. This process involved eliminating stop words, breaking down paragraphs into sentences, and further decomposing sentences into individual words or tokens as well as lemmatization to reduce words to their base form [ 18 , 22 ]. Additionally, our preprocessing included generating sequential word combinations, namely bigrams (e.g., mental disorder), as part of our feature set to capture more nuanced linguistic structures.

Unsupervised machine learning

We utilized the Latent Dirichlet Allocation (LDA) [ 23 ] approach, a well-established technique in NLP, social media analytics, and information retrieval [ 19 ]. LDA, an unsupervised probabilistic model, identifies topics by detecting underlying semantic patterns in a substantial text corpus [ 24 ]. Based on the data itself, the algorithm produces frequently mentioned pairs of words, the pairs of words that co-occur, and latent topics and their distributions over topics in the document. We calculated coherence scores to assess the validity of our topic model and identify the optimum number of topics. These scores measured the semantic similarity among words within a topic, indicating our model’s interpretability and thematic consistency. This similarity was determined by representing words as vectors based on their co-occurrence relationships. The coherence score was then calculated as the arithmetic mean of these similarities [ 25 ]. High coherence scores suggested meaningful thematic groupings, while lower scores may point to topics formed by statistical inference rather than actual thematic coherence. We employed Gensim (version 4.3.2; RARE Technologies Ltd) [ 26 ], an open-source Python library dedicated to topic modeling [ 19 ], for practical implementation.

Qualitative analysis

We employed a qualitative thematic analysis to complement and contextualize the LDA model findings. Our interpretative analysis adhered to a thematic analysis model [ 27 ]. The research team comprised three subject matter experts in the field of health informatics. Each researcher conducted a thorough review of a selection of at least five posts and their corresponding comments to identify and familiarize themselves with the emerging topics. A priori themes were utilized as the initial coding template based on the Governance Model for AI in Healthcare (GMAIH) [ 14 ]. Additional themes were generated for topics that did not readily adhere to a priori themes. Working independently, we assigned thematic names to the topics, ensuring they accurately represented the post content. We then critically assessed the initial codes for their alignment with the identified topics. In this assessment, we compared the theme names that each of us individually assigned to the labeled topics. This process continued until a unanimous consensus was achieved among all three researchers.

After processing all raw data, our final dataset included 3049 relevant posts and comments. We identified the most popular unigrams and bigrams regardless of the grammar structure of the words. Figure  2 shows a visualization of the most popular unigrams.

The word cloud of the most popular unigrams

In alignment with prior studies [ 25 , 28 , 29 ], we calculated the C V coherence score [ 25 , 30 ] to ascertain the optimal number of topics tailored to our dataset. Through this analytical process, the LDA model suggested that a configuration of 20 topics would yield a high coherence score for our data. We present the variation in coherence scores as a function of the total number of topics in Fig.  3 . Our analysis demonstrates that coherence scores range from 0.36 to 0.45, with the highest coherence for the model that includes 20 topics.

Highest coherence score achieved for the model with 20 topics

Next, three researchers completed the thematic analysis on the initial 20 topics generated as the results of the LDA analysis. This initial comparison resulted in an inter-annotator agreement rate, calculated using Fleiss’s Kappa, which equaled 0.66. Discrepancies primarily arose from interpretational nuances, exemplified by one researcher favoring a more abstract label (e.g., “Trustworthiness of AI”) while another leaned towards a more specific label (e.g., “Trust in use of AI for health-related information-seeking”). Upon closer examination, we observed conceptual similarities among several topics, leading us to converge them into broader themes with the agreement from all three researchers to ensure the topics corresponded meaningfully under one theme. An iterative process of comparison and consensus-building was employed to resolve discrepancies and achieve a unanimous agreement on thematic categorization. This method is in accordance with previous studies [ 25 , 28 , 29 ] that highlight the necessity of human intervention to clarify themes and reduce overlaps beyond what statistical measures alone can achieve. This process resulted in consolidating the initial topics into 14 distinct themes, as detailed in Table  1 . For example, three separate topics about LLM role in “communications about home visit and insurance”, “patient_provider interactions” and “communications about Narcan and relapse case” were merged into one broader theme as “LLM-enhanced healthcare communications”. An extended version of Table  1 is provided in the supplemental material that provides more details about the topics, extended definition of themes, potential ethical concerns, and direct quotes from the subreddit posts. This approach provides a framework for situating our findings within the ongoing discourse on LLM ethics in healthcare.

Next, we grouped these 14 themes into four higher-order ethical domains. Consolidating specific themes into the four ethical domains involved an iterative qualitative process examining the themes, identifying relationships between them, grouping conceptually related themes, and reaching full consensus amongst the three annotators on the final broader categories and allocation of themes. To develop the four ethical domains, we used the following frameworks: Principles of Biomedical Ethics framework emphasizes four central bioethical principles of autonomy, beneficence, non-maleficence, and justice. This framework can be applied to the ethical evaluation of LLM applications in healthcare, particularly concerning performance, communication, and diagnostics, ensuring that these technologies benefit patients without causing harm and respecting patient autonomy and justice [ 31 ]. Therefore, themes 1, 2, 5, 10 and 14 are grouped in domain (1) The FAIR data principles (Findability, Accessibility, Interoperability, and Reusability) are best practices for handling sensitive health data that could apply to mental care and rural contexts [ 32 ]. Themes 3, 4, 6 and 9, directly related to coding and data governance for special case such as rural healthcare applications and mental health are grouped in domain (2) The Social Determinants of Health framework promoted by the World Health Organization (WHO) underscores the importance of addressing the conditions in which people are born, grow, live, work, and age. This framework supports the domain of health equity by highlighting the need for LLM applications to promote fair access to healthcare services and to address disparities in health outcomes [ 33 ]. Themes 11 and 12 directly related to fairness and public health accessibility are grouped in doain (3) The Trustworthy AI Framework outlines seven key requirements for AI systems, including human agency and oversight, diversity, non-discrimination, and fairness. This framework supports the domain focusing on education, user experience, and trust, emphasizing the need for LLM systems to be designed and deployed in a trustworthy manner that respects human rights [ 34 ]. Themes 7, 8 and 13 directly discuss items related to user experience,, education and trust in LLM for healthcare applications, are grouped under domain (4) Table  2 shows the four core ethical domains.

Key ethical themes identified

This study identified key ethical themes concerning the integration of LLMs in healthcare, drawing particularly from the perspectives of clinicians in online discussions. Our findings found 14 distinct themes further categorized into four higher-order ethical domains: ethical implications in clinical applications, ethical coding and data governance, health equity, and the relationship between users and LLM systems. These themes mainly cover concerns about transparent and fair LLM decisions, privacy issues, access disparities, user experiences, and the reliability of LLMs in clinical settings. These themes highlight multifaceted ethical challenges that must be addressed to ensure the responsible deployment of LLMs in healthcare.

Ethical implications in clinical LLM applications

The identified themes and domains suggest that clinicians are generally concerned about the ethical implications of LLM integration in various aspects of healthcare, including direct patient care and communications. These concerns emphasize the need for robust ethical guidelines and frameworks that alleviate these diverse concerns to ensure LLMs’ responsible and effective use in clinical practice. Identifying specific themes provides a structured understanding of the ethical implications, guiding the development of targeted policies and practices. Moreover, the identified themes, such as LLM-enhanced healthcare communication, LLM in nursing and care quality improvement, and ethical aspects of LLM application in diagnostics, are in line with previous studies that have discussed the potential of AI to improve patient outcomes, optimize processes, and democratize healthcare access [ 15 , 16 ].

What sets our study apart is the specific focus on clinicians’ voices, which are often underrepresented in discussions about AI ethics. While studies by He et al. [ 6 ] and Tian et al. [ 7 ] broadly discuss LLMs’ technical and ethical challenges, our research dives into the ethical concerns perceived by practicing clinicians. This clinician-centric approach provides practical insights into real-world implications and the ethical considerations that directly impact patient care, and complements the broader discussions in existing studies by Esmaeilzadeh [ 35 ] on the ethical challenges of AI (such as LLMs) in healthcare.

A prominent theme centered on the potential for miscommunication and misunderstanding when LLMs are used to facilitate healthcare communication. Previous studies highlight the possibility of miscommunication due to using LLMs for healthcare purposes [ 36 , 37 ]. Concerns were raised regarding the ability of LLMs to navigate the touches of human language and effectively convey sensitive medical information. Similarly, the potential for overreliance on LLMs in emergency and outpatient settings was a recurring theme. Participants expressed anxieties about compromising patient safety and treatment efficacy by relying on imperfect algorithmic assessments, particularly in critical situations. Key areas of LLM application include healthcare communication through tools like chatbots and virtual assistants, which transform patient engagement by providing personalized information and support. For example, studies have shown that LLM chatbots effectively deliver health education and foster more interactive patient communication [ 38 ].

In nursing, LLM’s predictive analytics can be used to detect patients at risk and enhance care quality [ 39 ]. In emergency care, LLM can aid in swift and accurate decision-making, which is essential to prioritizing treatment and diagnosing conditions quickly, exemplifying how LLM integration can optimize care delivery and patient outcomes efficiently [ 40 ]. Moreover, clinicians found it essential that the diagnostic process involving LLMs is transparent and that the reasoning behind their recommendations is explainable. Transparency and explainability are emphasized in the literature as vital factors that can build trust and accountability in clinical decision-making [ 41 ].

Ethical coding and data governance

The issue of bias in the coding and training data of large language models (LLMs) emerged as a significant concern among clinicians. They underscored the necessity for stringent ethical oversight and robust standards to prevent LLMs from perpetuating existing biases within healthcare systems. Aligned with prior studies, our findings also highlight the potential for privacy breaches and underscore the critical need for rigorous data security measures when handling sensitive medical data for LLM training [ 35 ].

Despite these concerns, participants acknowledged several potential benefits of LLMs in specific areas. Notably, themes emerged around the potential of LLMs to enhance the quality of mental healthcare and improve healthcare accessibility in rural regions. Clinicians also noted the capability of LLMs to optimize processes such as triage and screening. These insights provide valuable guidance for researchers and developers focusing on LLM applications in healthcare.

A key takeaway is the imperative for transparency, interpretability, and explainability in the design of AI-powered tools, including LLMs, in healthcare settings. Healthcare professionals must comprehend the rationale behind LLM outputs to ensure the technology is utilized effectively and ethically [ 42 ]. Furthermore, robust data governance practices are essential to mitigate the risks of bias and privacy violations, thereby ensuring the integrity and trustworthiness of LLM applications.

Our thematic analysis not only confirms but also expands upon concerns highlighted in recent studies, particularly regarding data privacy as a critical ethical challenge in deploying generative AI and LLMs in healthcare [ 43 ]. Echoing the findings of Chang et al. [ 4 ] and Thirunavukarasu et al. [ 5 ], our study underscores the pressing need for comprehensive ethical guidelines in the development and deployment of LLMs.

The themes related to ethical coding and data governance, especially within the contexts of mental healthcare and rural healthcare applications, resonate with existing literature on privacy, data security, and the necessity for rigorous oversight in AI development [ 32 , 33 ]. Our study adds to this discourse by elucidating the ethical challenges specific to LLM applications in these sensitive healthcare domains.

In the realm of mental healthcare, clinicians expressed profound concerns about the confidentiality of patient-LLM interactions. The use of chatbots in mental health, for instance, could pose serious privacy issues, particularly regarding the retention and use of deeply personal information shared during therapy sessions. This highlights the unique challenges of implementing LLMs in mental health contexts, where patient trust and data sensitivity are of paramount importance.

Our study also revealed significant ethical dilemmas in rural healthcare applications related to data representation and model fairness. LLMs primarily trained on urban patient data may fail to accurately represent the health conditions and socioeconomic factors prevalent in rural communities. This potential bias underscores the need for diverse and representative training data to ensure equitable healthcare delivery across different geographic settings.

Furthermore, our findings emphasize the ethical implications of data governance in the telemedicine applications of LLMs. For example, when LLMs are employed to analyze video consultations or patient messages, concerns arise regarding the ethical management of patient consent and data ownership across state or national boundaries. This example illustrates the intricate interplay between LLM technology, data protection regulations, and the increasingly global nature of healthcare delivery.

Health equity and access disparities

The themes around LLM in health equity and the distribution of public health services align with the growing recognition of the potential for AI to exacerbate or mitigate health disparities [ 33 , 34 ]. Our findings underscore the importance of addressing these ethical dimensions to ensure that LLM applications promote equitable access to healthcare services. The integration of LLMs in public health has the potential to significantly impact various aspects of health equity, from improving access to information and resources in underserved communities to tailoring healthcare interventions that meet the specific needs of diverse populations. LLMs can be programmed to identify and highlight disparities in healthcare delivery, enabling public health officials and policymakers to make more informed decisions that target and reduce these inequities.

However, there is also a risk that if not carefully managed, LLMs could reinforce existing biases and disparities. For instance, if the training data for LLMs predominantly represents the experiences and needs of more privileged groups, the resulting applications may not adequately serve or may even disadvantage marginalized populations. Therefore, it is crucial to ensure that the development and deployment of LLMs in healthcare are guided by principles of fairness and inclusivity, and involve continuous monitoring and evaluation to prevent unintended consequences.

By addressing these ethical considerations, our study contributes to a deeper understanding of how LLMs can be leveraged to support health equity and improve the distribution of public health services. This includes advocating for the involvement of diverse communities in the development process, ensuring transparency in AI decision-making, and fostering collaborations between technologists, healthcare providers, and patients to create AI systems that truly serve the needs of all populations.

Education, user experience and trust in healthcare

The themes associated with education, user experience, and trust in healthcare LLM systems resonate with the emphasis in the literature on the need for explainable, transparent, and trustworthy AI systems in healthcare [ 44 ]. Our study contributes to this discussion by highlighting the perspectives of clinicians on the ethical aspects of their relationship with LLM systems.

Several studies emphasize the importance of educating clinical staff and professionals about LLM applications in medical practices, ensuring they are aware of both these technologies’ potential and limitations [ 45 ]. Training can cover how LLM tools work, their potential benefits, potential biases, errors, and ethical considerations [ 46 ]. This education is crucial for practitioners to use LLM tools effectively, responsibly, and ethically. Our research supports the growing call for LLM literacy among healthcare professionals, highlighting the importance of continuous education for users in clinical settings to keep pace with rapidly evolving LLM technologies [ 47 ].

Participants reported ambivalence about AI-assisted decision-making, reflecting concerns about maintaining clinical autonomy and the potential for deskilling. This links to broader discussions in the literature about preserving the art of clinical reasoning in an increasingly technology-driven healthcare environment [ 48 ]. Thus, physicians must ensure that reliance on LLMs and AI does not erode our ability to think critically and independently.

The introduction of LLM can augment, not replace, the essential elements of healthcare delivery [ 49 ]. This entails carefully integrating LLM tools to support healthcare professionals, allowing them to focus more on the interpersonal aspects of patient care. AI’s role, primarily in data analysis, diagnostics, and treatment recommendations, should complement the healthcare provider’s expertise [ 50 ]. Maintaining the human touch in patient-provider interactions is crucial, as is preserving empathy and understanding, which are fundamental to patient care. As LLM systems become more integrated into healthcare settings, they should be designed and implemented to bolster these human elements rather than overshadow them. LLMs could transform the way healthcare providers communicate with patients. It is necessary to ensure that this technology supports and enhances this relationship rather than undermining it with impersonal or inaccurate communication. Lastly, integrating LLM into clinical workflows is a significant shift in care delivery. By automating routine tasks, LLM may allow clinicians to focus more on direct patient care, enhancing the overall quality of healthcare services. This is evident in applications like LLM-driven medication management, which not only reduces the risk of human error but also improves efficiency in patient care [ 51 ].

Theoretical contributions

Theoretically, our study contributes a structured, empirically grounded framework for understanding key ethical implications of LLM integration in healthcare from the perspective of online clinician communities. The identified themes and domains provide a structured foundation for further research and theory development in this emerging field. Also, by focusing on the voices of clinicians, our research offers a unique theoretical lens that emphasizes the practical ethical concerns faced by healthcare providers. This perspective is crucial for developing ethically sound AI systems sensitive to the realities of clinical practice.

Practical implications

Practically, our findings can inform the development of ethical guidelines, standards, and policies for the responsible integration of LLMs in healthcare. The insights from clinicians can guide the design and deployment of LLM systems that are not only technically robust but also ethically aligned with the values and concerns of healthcare professionals and patients. Policymakers can use these insights to create regulations that ensure LLMs are used responsibly, minimizing risks to patients. Developers of LLMs can also use the insights from this study to design systems that are more transparent, fair, and aligned with the ethical expectations of clinicians. This alignment can enhance the trust and acceptance of AI tools in healthcare settings. Moreover, Our findings highlight the need for comprehensive training programs for clinicians on the ethical use of LLMs. By understanding the potential ethical pitfalls, clinicians can better navigate the complexities of integrating these technologies into patient care.

Limitation and future study

While our study provides valuable insights, it also has limitations that suggest avenues for future research. This study, focusing on the ethical aspects of LLM in healthcare, reveals several avenues for future research due to its limitations. We utilized a dataset exclusively sourced from a subreddit frequented by self-identified physicians and healthcare professionals, who signal their roles within the healthcare ecosystem through specific flair tags next to their usernames. This approach, while innovative, suggests the potential for more robust data collection in future research by directly engaging with clinicians. Expanding the scope to include diverse forums could offer a richer, more varied perspective of clinician viewpoints from different regions and medical specialties. Future investigations should also consider a comparative analysis across various clinician communities to deepen the understanding of these issues.

Moreover, future studies may expand the scope to include the perspectives of patients, technology developers, policymakers, and ethicists, providing a more holistic view. Incorporating perspectives from patients, tech developers, policymakers, and ethicists will give a more holistic view of the ethical landscape. Further research could explore potential solutions in regulation and policy, extending beyond identifying ethical implications. We also acknowledge that the keywords used for data extraction may not have been exhaustive. Future research should aim to address these gaps by incorporating a more extensive set of keywords to capture a broader range of discussions and ethical considerations pertaining to a variety of LLMs tailored to medical contexts .

In addition, while our team reached a consensus on qualitative coding, engaging a broader range of experts could diversify thematic insights, emphasizing the subjectivity in qualitative research and the potential for biases inherent in human interpretation. While topic modeling is well-supported in literature for applications such as corpus exploration and information retrieval, it is crucial to prioritize evaluations based on real-world task performance over merely optimizing traditional likelihood-based measures. To bridge the gap between automated evaluations and human interpretability, future developments in topic modeling should consider incorporating human judgments directly into the model learning process. Alternatively, developing computational methods that simulate human evaluations could further enhance the relevance and usability of the topics generated, making them both qualitatively rich and practically useful. Lastly, given the rapid advancements in LLM technology and its healthcare applications, ongoing reassessment of the ethical landscape is essential. Longitudinal studies are recommended to observe evolving clinician perspectives as technology integration progresses, ensuring that ethical considerations remain aligned with technological developments.

This study develops a framework from self-identified clinician insights to categorize the ethical challenges of integrating LLM in healthcare, identifying 14 key themes. These themes cover issues spanning transparent and fair LLM decisions, privacy, access disparities, user experiences, and reliability concerns that must be proactively addressed to harness LLM’s immense potential while respecting patient rights. As LLM capabilities rapidly progress, sustained ethical inquiries focusing on real-world integration complexities from stakeholders’ viewpoints remain imperative to responsible innovation. Our thematic mapping notably synthesizes, reinforces, and expands current discourse at the intersection of medicine and LLM domain, advocating for tailored governance rather than broad regulations. This research enriches the ethical groundwork to guide policy and practice, promoting the use of LLM in healthcare to improve clinical outcomes ethically and effectively.

Data availability

The data underlying this article will be shared on reasonable request to the corresponding author.

Preiksaitis C, Rose C. Opportunities, challenges, and future directions of generative artificial intelligence in medical education: scoping review. JMIR Med Educ. 2023;9:e48785.

Article   PubMed   PubMed Central   Google Scholar  

Ayers JW, Poliak A, Dredze M, Leas EC, Zhu Z, Kelley JB et al. Comparing physician and artificial intelligence chatbot responses to patient questions posted to a public social media forum. JAMA Intern Med [Internet]. 2023;183(6):589–96. https://doi.org/10.1001/jamainternmed.2023.1838

Yang R, Tan TF, Lu W, Thirunavukarasu AJ, Ting DS, Liu N. Large language models in health care: development, applications, and challenges. Health Care Sci. 2023;2(4):255–63.

Chang Y, Wang X, Wang J, Wu Y, Yang L, Zhu K, Chen H, Yi X, Wang C, Wang Y, Ye W. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology; 2023.

Thirunavukarasu AJ, Ting DS, Elangovan K, Gutierrez L, Tan TF, Ting DS. Large language models in medicine. Nat Med. 2023;29(8):1930–40.

Article   CAS   PubMed   Google Scholar  

He K, Mao R, Lin Q, Ruan Y, Lan X, Feng M, Cambria E. A survey of large language models for healthcare: from data, technology, and applications to accountability and ethics. arXiv preprint arXiv:2310.05694. 2023 Oct 9.

Tian S, Jin Q, Yeganova L, Lai PT, Zhu Q, Chen X, Yang Y, Chen Q, Kim W, Comeau DC, Islamaj R. Opportunities and challenges for ChatGPT and large language models in biomedicine and health. Briefings in Bioinformatics. 2024;25(Abadir :bbad493.

Reddy S, Fox J, Purohit MP. Artificial intelligence-enabled healthcare delivery. J R Soc Med. 2019;112(Abadir :22 – 8.

Hadi MU, Qureshi R, Shah A, Irfan M, Zafar A, Shaikh MB, Akhtar N, Wu J, Mirjalili S. A survey on large language models: applications, challenges, limitations, and practical usage. Authorea Preprints. 2023 Oct 31.

Garg RK, Urs VL, Agarwal AA, Chaudhary SK, Paliwal V, Kar SK. Exploring the role of ChatGPT in patient care (diagnosis and treatment) and medical research: a systematic review. Health Promotion Perspect. 2023;13(3):183.

Article   Google Scholar  

Lee S, Lee J, Park J, Park J, Kim D, Lee J, Oh J. Deep learning-based natural language processing for detecting medical symptoms and histories in emergency patient triage. Am J Emerg Med. 2024;77:29–38.

Article   PubMed   Google Scholar  

Wang X, Sanders HM, Liu Y, Seang K, Tran BX, Atanasov AG, Qiu Y, Tang S, Car J, Wang YX, Wong TY. ChatGPT: promise and challenges for deployment in low-and middle-income countries. Lancet Reg Health–Western Pac. 2023;41.

Tripathi S, Sukumaran R, Cook TS. Efficient healthcare with large language models: optimizing clinical workflow and enhancing patient care. J Am Med Inf Association 2024 Jan 25:ocad258.

Reddy S, Allan S, Coghlan S, Cooper P. A governance model for the application of LLM in health care. J Am Med Inform Assoc. 2020;27(3):491–7.

Ogundare O, Sofolahan S. Large Language models in Ambulatory Devices for Home Health Diagnostics: a case study of Sickle Cell Anemia Management. arXiv preprint arXiv:2305.03715. 2023 May 5.

Montagna S, Ferretti S, Klopfenstein LC, Florio A, Pengo MF. Data decentralisation of LLM-based chatbot systems in chronic disease self-management. InProceedings of the 2023 ACM Conference on Information Technology for Social Good 2023 Sep 6 (pp. 205–212).

Ancker JS, Benda NC, Reddy M, Unertl KM, Veinot T. Guidance for publishing qualitative research in informatics. J Am Med Inform Assoc. 2021;28(12):2743–8.

Feldman R, Sanger J. The text mining handbook: advanced approaches in analyzing unstructured data. Cambridge University Press; 2007.

Jelodar H, Wang Y, Yuan C, Feng X, Jiang X, Li Y et al. Latent Dirichllocation (LDA) and topic modeling: models, applications, a survey. Multimed Tools Appl [Internet]. 2019;78(Abadir :15169–211. https://doi.org/10.1007/s11042-018-6894-4

Yan M, Cerri GG, Moraes FY. ChatGPT and medicine: how AI language models are shaping the future and health related careers. Nature Biotechnology. 2023;41(Abadir :1657-8.

Loper E, Bird SNLTK. The Natural Language Toolkit [Internet]. arXiv [cs.CL]. 2002. http://arxiv.org/abs/cs/0205028

Vijayarani S, Ilamathi MJ, Nithya M. Preprocessing techniques for text mining-an overview. Int J Comput Sci Communication Networks. 2015;5(Abadir et al):7–16.

Google Scholar  

Blei DM, Ng AY, Jordan MI. Latent dirichlet allocation. J Mach Learn Res. 2003;3(Jan):993–1022.

Chang J, Gerrish S, Wang C, Boyd-Graber J, Blei D. Reading tea leaves: How humans interpret topic models. In: Advances in neural information processing systems. 2009.

Röder M, Both A, Hinneburg A. Exploring the space of topic coherence measures. InProceedings of the eighth ACM international conference on Web search and data mining 2015 Feb 2 (pp. 399–408).

Srinivasa-Desikan B. Natural Language Processing and Computational Linguistics: a practical guide to text analysis with Python, Gensim, spaCy, and Keras. Packt Publishing Ltd; 2018.

Brooks J, McCluskey S, Turley E, King N. The utility of Template Analysis in qualitative psychology research. Qual Res Psychol [Internet]. 2015;12(Abadir :202–22. https://doi.org/10.1080/14780887.2014.955224

Xue J, Chen J, Hu R, Chen C, Zheng C, Su Y, Zhu T. Twitter discussions and emotions about the COVID-19 pandemic: Machine learning approach. Journal of medical Internet research. 2020;22(Abadir :e20550.

Liu Q, Zheng Z, Zheng J, Chen Q, Liu G, Chen S, Chu B, Zhu H, Akinwunmi B, Huang J, Zhang CJ. Health communication through news media during the early stage of the COVID-19 outbreak in China: digital topic modeling approach. J Med Internet Res. 2020;22(4):e19118.

Syed S, Spruit M. Full-text or abstract? examining topic coherence scores using latent dirichllocation. In2017 IEEE International conference on data science and advanced analytics (DSAA) 2017 Oct 19 (pp. 165–174). IEEE.

Beauchamp T, Childress J. Principles of biomedical ethics: marking its fortieth anniversary. Am J Bioeth. 2019;19(Abadir et al):9–12.

Wilkinson, M. D., Dumontier, M., Aalbersberg, I. J., Appleton, G., Axton, M., Baak, A., ... & Mons, B. (2016). The FAIR Guiding Principles for scientific data management and stewardship. Scientific data, 3(Abadir et al.), 1–9

Heyman JC, Kelly PL, Reback GM, Blumenstock KH. (2018). Social determinants of health. Health and social work: practice, policy, and research.

AI H. High-level expert group on artificial intelligence. 6 ed. Ethics guidelines for trustworthy AI; 2019.

Esmaeilzadeh P. Challenges and strategies for wide-scale artificial intelligence (AI) deployment in healthcare practices: a perspective for healthcare organizations. Artif Intell Med. 2024;151:102861.

Nasseri M, Brandtner P, Zimmermann R, Falatouri T, Darbanian F, Obinwanne T. (2023, July). Applications of large language models (LLMs) in business analytics–exemplary use cases in data preparation tasks. In International Conference on Human-Computer Interaction (pp. 182–198). Cham: Springer Nature Switzerland.

Denecke K, May R, LLMHealthGroup, Rivera Romero O. (2024). Potential of large Language models in Health Care: Delphi Study. J Med Internet Res, 26, e52399.

Zhang J, Oh YJ, Lange P, Yu Z, Fukuoka Y. Artificial intelligence chatbot behavior change model for designing artificial intelligence chatbots to promote physical activity and a healthy diet: Viewpoint. J Med Internet Res [Internet]. 2020;22(9):e22845. https://doi.org/10.2196/22845

Choudhury A, Asan O. Role of artificial intelligence in patient safety outcomes: Systematic literature review. JMIR Med Inform [Internet]. 2020;8(Abadir :e18599. https://doi.org/10.2196/18599

Abedi V, Khan A, Chaudhary D, Misra D, Avula V, Mathrawala D et al. Using artificial intelligence for improving stroke diagnosis in emergency departments: a practical framework. Ther Adv Neurol Disord [Internet]. 2020;13:175628642093896. https://doi.org/10.1177/1756286420938962

Subramanian HV, Canfield C, Shank DB. (2024). Designing explainable AI to improve human-AI team performance: a medical stakeholder-driven scoping review. Artif Intell Med, 102780.

Choudhury A, Chaudhry Z. (2024). Large Language models and user trust: consequence of Self-Referential Learning Loop and the deskilling of Health Care professionals. J Med Internet Res, 26, e56764.

Chen Y, Esmaeilzadeh P. (2024). Generative AI in medical practice: in-depth exploration of privacy and security challenges. J Med Internet Res, 26, e53008.

Murphy K, Di Ruggiero E, Upshur R, Willison DJ, Malhotra N, Cai JC et al. Artificial intelligence for good health: a scoping review of the ethics literature. BMC Med Ethics [Internet]. 2021;22(Abadir :14. https://doi.org/10.1186/s12910-021-00577-8

Charow R, Jeyakumar T, Younus S, Dolatabadi E, Salhia M, Al-Mouaswas D et al. Artificial intelligence education programs for health care professionals: Scoping review. JMIR Med Educ [Internet]. 2021;7(4):e31043. https://doi.org/10.2196/31043

Vollmer S, Mateen BA, Bohner G, Király FJ, Ghani R, Jonsson P et al. Machine learning and artificial intelligence research for patient benefit: 20 critical questions on transparency, replicability, ethics, and effectiveness. BMJ [Internet]. 2020;368:l6927. https://doi.org/10.1136/bmj.l6927

Gama F, Tyskbo D, Nygren J, Barlow J, Reed J, Svedberg P. Implementation frameworks for artificial intelligence translation into health care practice: Scoping review. J Med Internet Res [Internet]. 2022;24(Abadir :e32215. https://doi.org/10.2196/32215

Wartman SA, Combs CD. Medical education must move from the information age to the age of artificial intelligence. Acad Med. 2018;93(8):1107–9.

Kelly CJ, Karthikesalingam A, Suleyman M, Corrado G, King D. Key challenges for delivering clinical impact with artificial intelligence. BMC Med [Internet]. 2019;17(Abadir :195. https://doi.org/10.1186/s12916-019-1426-2

Markus AF, Kors JA, Rijnbeek PR. The role of explainability in creating trustworthy artificial intelligence for health care: A comprehensive survey of the terminology, design choices, and evaluation strategies. J Biomed Inform [Internet]. 2021;113(103655):103655. https://doi.org/10.1016/j.jbi.2020.103655

Johnson KB, Wei WQ, Weeraratne D, Frisse ME, Misulis K, Rhee K et al. Precision medicine, LLM, and the future of personalized health care. Clin Transl Sci. 2021;14(Abadir :86–93.

Download references

Acknowledgements

Not applicable.

This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.

Author information

Authors and affiliations.

Information Systems & Business Analytics, College of Business, Florida International University, 11200 S.W. 8th St., Room RB 250, Miami, FL, 33199, USA

Tala Mirzaei, Leila Amini & Pouyan Esmaeilzadeh

You can also search for this author in PubMed   Google Scholar

Contributions

T.M. and P.E. conceived and designed this study. T.M. and L.A. collected and processed the data and built the machine learning models. T.M., P.E. and L.A. completed the qualitative analysis. T.M. and P.E. drafted the manuscript. All authors have read and agreed to the published version of the manuscript.

Corresponding author

Correspondence to Tala Mirzaei .

Ethics declarations

Ethics approval and consent to participate.

The Florida International University Office of Research Integrity – Research Compliance, MARC 414 – has reviewed the study and has determined that it is Not Human Subject Research (NHSR). Therefore, Consent to participate was deemed not required according to IRB protocol. IRB Protocol NHSR number IRB-24-0054, Reference number 114022.

Consent for publication

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Mirzaei, T., Amini, L. & Esmaeilzadeh, P. Clinician voices on ethics of LLM integration in healthcare: a thematic analysis of ethical concerns and implications. BMC Med Inform Decis Mak 24 , 250 (2024). https://doi.org/10.1186/s12911-024-02656-3

Download citation

Received : 25 March 2024

Accepted : 27 August 2024

Published : 09 September 2024

DOI : https://doi.org/10.1186/s12911-024-02656-3

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Artificial Intelligence
• Thematic analysis

BMC Medical Informatics and Decision Making

ISSN: 1472-6947

• General enquiries: [email protected]

Warning: The NCBI web site requires JavaScript to function. more...

An official website of the United States government

The .gov means it's official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you're on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• Browse Titles

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Lau F, Kuziemsky C, editors. Handbook of eHealth Evaluation: An Evidence-based Approach [Internet]. Victoria (BC): University of Victoria; 2017 Feb 27.

Handbook of eHealth Evaluation: An Evidence-based Approach [Internet].

Chapter 9 methods for literature reviews.

Guy Paré and Spyros Kitsiou .

9.1. Introduction

Literature reviews play a critical role in scholarship because science remains, first and foremost, a cumulative endeavour ( vom Brocke et al., 2009 ). As in any academic discipline, rigorous knowledge syntheses are becoming indispensable in keeping up with an exponentially growing eHealth literature, assisting practitioners, academics, and graduate students in finding, evaluating, and synthesizing the contents of many empirical and conceptual papers. Among other methods, literature reviews are essential for: (a) identifying what has been written on a subject or topic; (b) determining the extent to which a specific research area reveals any interpretable trends or patterns; (c) aggregating empirical findings related to a narrow research question to support evidence-based practice; (d) generating new frameworks and theories; and (e) identifying topics or questions requiring more investigation ( Paré, Trudel, Jaana, & Kitsiou, 2015 ).

Literature reviews can take two major forms. The most prevalent one is the “literature review” or “background” section within a journal paper or a chapter in a graduate thesis. This section synthesizes the extant literature and usually identifies the gaps in knowledge that the empirical study addresses ( Sylvester, Tate, & Johnstone, 2013 ). It may also provide a theoretical foundation for the proposed study, substantiate the presence of the research problem, justify the research as one that contributes something new to the cumulated knowledge, or validate the methods and approaches for the proposed study ( Hart, 1998 ; Levy & Ellis, 2006 ).

The second form of literature review, which is the focus of this chapter, constitutes an original and valuable work of research in and of itself ( Paré et al., 2015 ). Rather than providing a base for a researcher’s own work, it creates a solid starting point for all members of the community interested in a particular area or topic ( Mulrow, 1987 ). The so-called “review article” is a journal-length paper which has an overarching purpose to synthesize the literature in a field, without collecting or analyzing any primary data ( Green, Johnson, & Adams, 2006 ).

When appropriately conducted, review articles represent powerful information sources for practitioners looking for state-of-the art evidence to guide their decision-making and work practices ( Paré et al., 2015 ). Further, high-quality reviews become frequently cited pieces of work which researchers seek out as a first clear outline of the literature when undertaking empirical studies ( Cooper, 1988 ; Rowe, 2014 ). Scholars who track and gauge the impact of articles have found that review papers are cited and downloaded more often than any other type of published article ( Cronin, Ryan, & Coughlan, 2008 ; Montori, Wilczynski, Morgan, Haynes, & Hedges, 2003 ; Patsopoulos, Analatos, & Ioannidis, 2005 ). The reason for their popularity may be the fact that reading the review enables one to have an overview, if not a detailed knowledge of the area in question, as well as references to the most useful primary sources ( Cronin et al., 2008 ). Although they are not easy to conduct, the commitment to complete a review article provides a tremendous service to one’s academic community ( Paré et al., 2015 ; Petticrew & Roberts, 2006 ). Most, if not all, peer-reviewed journals in the fields of medical informatics publish review articles of some type.

The main objectives of this chapter are fourfold: (a) to provide an overview of the major steps and activities involved in conducting a stand-alone literature review; (b) to describe and contrast the different types of review articles that can contribute to the eHealth knowledge base; (c) to illustrate each review type with one or two examples from the eHealth literature; and (d) to provide a series of recommendations for prospective authors of review articles in this domain.

9.2. Overview of the Literature Review Process and Steps

As explained in Templier and Paré (2015) , there are six generic steps involved in conducting a review article:

• formulating the research question(s) and objective(s),
• searching the extant literature,
• screening for inclusion,
• assessing the quality of primary studies,
• extracting data, and
• analyzing data.

Although these steps are presented here in sequential order, one must keep in mind that the review process can be iterative and that many activities can be initiated during the planning stage and later refined during subsequent phases ( Finfgeld-Connett & Johnson, 2013 ; Kitchenham & Charters, 2007 ).

Formulating the research question(s) and objective(s): As a first step, members of the review team must appropriately justify the need for the review itself ( Petticrew & Roberts, 2006 ), identify the review’s main objective(s) ( Okoli & Schabram, 2010 ), and define the concepts or variables at the heart of their synthesis ( Cooper & Hedges, 2009 ; Webster & Watson, 2002 ). Importantly, they also need to articulate the research question(s) they propose to investigate ( Kitchenham & Charters, 2007 ). In this regard, we concur with Jesson, Matheson, and Lacey (2011) that clearly articulated research questions are key ingredients that guide the entire review methodology; they underscore the type of information that is needed, inform the search for and selection of relevant literature, and guide or orient the subsequent analysis. Searching the extant literature: The next step consists of searching the literature and making decisions about the suitability of material to be considered in the review ( Cooper, 1988 ). There exist three main coverage strategies. First, exhaustive coverage means an effort is made to be as comprehensive as possible in order to ensure that all relevant studies, published and unpublished, are included in the review and, thus, conclusions are based on this all-inclusive knowledge base. The second type of coverage consists of presenting materials that are representative of most other works in a given field or area. Often authors who adopt this strategy will search for relevant articles in a small number of top-tier journals in a field ( Paré et al., 2015 ). In the third strategy, the review team concentrates on prior works that have been central or pivotal to a particular topic. This may include empirical studies or conceptual papers that initiated a line of investigation, changed how problems or questions were framed, introduced new methods or concepts, or engendered important debate ( Cooper, 1988 ). Screening for inclusion: The following step consists of evaluating the applicability of the material identified in the preceding step ( Levy & Ellis, 2006 ; vom Brocke et al., 2009 ). Once a group of potential studies has been identified, members of the review team must screen them to determine their relevance ( Petticrew & Roberts, 2006 ). A set of predetermined rules provides a basis for including or excluding certain studies. This exercise requires a significant investment on the part of researchers, who must ensure enhanced objectivity and avoid biases or mistakes. As discussed later in this chapter, for certain types of reviews there must be at least two independent reviewers involved in the screening process and a procedure to resolve disagreements must also be in place ( Liberati et al., 2009 ; Shea et al., 2009 ). Assessing the quality of primary studies: In addition to screening material for inclusion, members of the review team may need to assess the scientific quality of the selected studies, that is, appraise the rigour of the research design and methods. Such formal assessment, which is usually conducted independently by at least two coders, helps members of the review team refine which studies to include in the final sample, determine whether or not the differences in quality may affect their conclusions, or guide how they analyze the data and interpret the findings ( Petticrew & Roberts, 2006 ). Ascribing quality scores to each primary study or considering through domain-based evaluations which study components have or have not been designed and executed appropriately makes it possible to reflect on the extent to which the selected study addresses possible biases and maximizes validity ( Shea et al., 2009 ). Extracting data: The following step involves gathering or extracting applicable information from each primary study included in the sample and deciding what is relevant to the problem of interest ( Cooper & Hedges, 2009 ). Indeed, the type of data that should be recorded mainly depends on the initial research questions ( Okoli & Schabram, 2010 ). However, important information may also be gathered about how, when, where and by whom the primary study was conducted, the research design and methods, or qualitative/quantitative results ( Cooper & Hedges, 2009 ). Analyzing and synthesizing data : As a final step, members of the review team must collate, summarize, aggregate, organize, and compare the evidence extracted from the included studies. The extracted data must be presented in a meaningful way that suggests a new contribution to the extant literature ( Jesson et al., 2011 ). Webster and Watson (2002) warn researchers that literature reviews should be much more than lists of papers and should provide a coherent lens to make sense of extant knowledge on a given topic. There exist several methods and techniques for synthesizing quantitative (e.g., frequency analysis, meta-analysis) and qualitative (e.g., grounded theory, narrative analysis, meta-ethnography) evidence ( Dixon-Woods, Agarwal, Jones, Young, & Sutton, 2005 ; Thomas & Harden, 2008 ).

9.3. Types of Review Articles and Brief Illustrations

EHealth researchers have at their disposal a number of approaches and methods for making sense out of existing literature, all with the purpose of casting current research findings into historical contexts or explaining contradictions that might exist among a set of primary research studies conducted on a particular topic. Our classification scheme is largely inspired from Paré and colleagues’ (2015) typology. Below we present and illustrate those review types that we feel are central to the growth and development of the eHealth domain.

9.3.1. Narrative Reviews

The narrative review is the “traditional” way of reviewing the extant literature and is skewed towards a qualitative interpretation of prior knowledge ( Sylvester et al., 2013 ). Put simply, a narrative review attempts to summarize or synthesize what has been written on a particular topic but does not seek generalization or cumulative knowledge from what is reviewed ( Davies, 2000 ; Green et al., 2006 ). Instead, the review team often undertakes the task of accumulating and synthesizing the literature to demonstrate the value of a particular point of view ( Baumeister & Leary, 1997 ). As such, reviewers may selectively ignore or limit the attention paid to certain studies in order to make a point. In this rather unsystematic approach, the selection of information from primary articles is subjective, lacks explicit criteria for inclusion and can lead to biased interpretations or inferences ( Green et al., 2006 ). There are several narrative reviews in the particular eHealth domain, as in all fields, which follow such an unstructured approach ( Silva et al., 2015 ; Paul et al., 2015 ).

Despite these criticisms, this type of review can be very useful in gathering together a volume of literature in a specific subject area and synthesizing it. As mentioned above, its primary purpose is to provide the reader with a comprehensive background for understanding current knowledge and highlighting the significance of new research ( Cronin et al., 2008 ). Faculty like to use narrative reviews in the classroom because they are often more up to date than textbooks, provide a single source for students to reference, and expose students to peer-reviewed literature ( Green et al., 2006 ). For researchers, narrative reviews can inspire research ideas by identifying gaps or inconsistencies in a body of knowledge, thus helping researchers to determine research questions or formulate hypotheses. Importantly, narrative reviews can also be used as educational articles to bring practitioners up to date with certain topics of issues ( Green et al., 2006 ).

Recently, there have been several efforts to introduce more rigour in narrative reviews that will elucidate common pitfalls and bring changes into their publication standards. Information systems researchers, among others, have contributed to advancing knowledge on how to structure a “traditional” review. For instance, Levy and Ellis (2006) proposed a generic framework for conducting such reviews. Their model follows the systematic data processing approach comprised of three steps, namely: (a) literature search and screening; (b) data extraction and analysis; and (c) writing the literature review. They provide detailed and very helpful instructions on how to conduct each step of the review process. As another methodological contribution, vom Brocke et al. (2009) offered a series of guidelines for conducting literature reviews, with a particular focus on how to search and extract the relevant body of knowledge. Last, Bandara, Miskon, and Fielt (2011) proposed a structured, predefined and tool-supported method to identify primary studies within a feasible scope, extract relevant content from identified articles, synthesize and analyze the findings, and effectively write and present the results of the literature review. We highly recommend that prospective authors of narrative reviews consult these useful sources before embarking on their work.

Darlow and Wen (2015) provide a good example of a highly structured narrative review in the eHealth field. These authors synthesized published articles that describe the development process of mobile health (m-health) interventions for patients’ cancer care self-management. As in most narrative reviews, the scope of the research questions being investigated is broad: (a) how development of these systems are carried out; (b) which methods are used to investigate these systems; and (c) what conclusions can be drawn as a result of the development of these systems. To provide clear answers to these questions, a literature search was conducted on six electronic databases and Google Scholar . The search was performed using several terms and free text words, combining them in an appropriate manner. Four inclusion and three exclusion criteria were utilized during the screening process. Both authors independently reviewed each of the identified articles to determine eligibility and extract study information. A flow diagram shows the number of studies identified, screened, and included or excluded at each stage of study selection. In terms of contributions, this review provides a series of practical recommendations for m-health intervention development.

9.3.2. Descriptive or Mapping Reviews

The primary goal of a descriptive review is to determine the extent to which a body of knowledge in a particular research topic reveals any interpretable pattern or trend with respect to pre-existing propositions, theories, methodologies or findings ( King & He, 2005 ; Paré et al., 2015 ). In contrast with narrative reviews, descriptive reviews follow a systematic and transparent procedure, including searching, screening and classifying studies ( Petersen, Vakkalanka, & Kuzniarz, 2015 ). Indeed, structured search methods are used to form a representative sample of a larger group of published works ( Paré et al., 2015 ). Further, authors of descriptive reviews extract from each study certain characteristics of interest, such as publication year, research methods, data collection techniques, and direction or strength of research outcomes (e.g., positive, negative, or non-significant) in the form of frequency analysis to produce quantitative results ( Sylvester et al., 2013 ). In essence, each study included in a descriptive review is treated as the unit of analysis and the published literature as a whole provides a database from which the authors attempt to identify any interpretable trends or draw overall conclusions about the merits of existing conceptualizations, propositions, methods or findings ( Paré et al., 2015 ). In doing so, a descriptive review may claim that its findings represent the state of the art in a particular domain ( King & He, 2005 ).

In the fields of health sciences and medical informatics, reviews that focus on examining the range, nature and evolution of a topic area are described by Anderson, Allen, Peckham, and Goodwin (2008) as mapping reviews . Like descriptive reviews, the research questions are generic and usually relate to publication patterns and trends. There is no preconceived plan to systematically review all of the literature although this can be done. Instead, researchers often present studies that are representative of most works published in a particular area and they consider a specific time frame to be mapped.

An example of this approach in the eHealth domain is offered by DeShazo, Lavallie, and Wolf (2009). The purpose of this descriptive or mapping review was to characterize publication trends in the medical informatics literature over a 20-year period (1987 to 2006). To achieve this ambitious objective, the authors performed a bibliometric analysis of medical informatics citations indexed in medline using publication trends, journal frequencies, impact factors, Medical Subject Headings (MeSH) term frequencies, and characteristics of citations. Findings revealed that there were over 77,000 medical informatics articles published during the covered period in numerous journals and that the average annual growth rate was 12%. The MeSH term analysis also suggested a strong interdisciplinary trend. Finally, average impact scores increased over time with two notable growth periods. Overall, patterns in research outputs that seem to characterize the historic trends and current components of the field of medical informatics suggest it may be a maturing discipline (DeShazo et al., 2009).

9.3.3. Scoping Reviews

Scoping reviews attempt to provide an initial indication of the potential size and nature of the extant literature on an emergent topic (Arksey & O’Malley, 2005; Daudt, van Mossel, & Scott, 2013 ; Levac, Colquhoun, & O’Brien, 2010). A scoping review may be conducted to examine the extent, range and nature of research activities in a particular area, determine the value of undertaking a full systematic review (discussed next), or identify research gaps in the extant literature ( Paré et al., 2015 ). In line with their main objective, scoping reviews usually conclude with the presentation of a detailed research agenda for future works along with potential implications for both practice and research.

Unlike narrative and descriptive reviews, the whole point of scoping the field is to be as comprehensive as possible, including grey literature (Arksey & O’Malley, 2005). Inclusion and exclusion criteria must be established to help researchers eliminate studies that are not aligned with the research questions. It is also recommended that at least two independent coders review abstracts yielded from the search strategy and then the full articles for study selection ( Daudt et al., 2013 ). The synthesized evidence from content or thematic analysis is relatively easy to present in tabular form (Arksey & O’Malley, 2005; Thomas & Harden, 2008 ).

One of the most highly cited scoping reviews in the eHealth domain was published by Archer, Fevrier-Thomas, Lokker, McKibbon, and Straus (2011) . These authors reviewed the existing literature on personal health record ( phr ) systems including design, functionality, implementation, applications, outcomes, and benefits. Seven databases were searched from 1985 to March 2010. Several search terms relating to phr s were used during this process. Two authors independently screened titles and abstracts to determine inclusion status. A second screen of full-text articles, again by two independent members of the research team, ensured that the studies described phr s. All in all, 130 articles met the criteria and their data were extracted manually into a database. The authors concluded that although there is a large amount of survey, observational, cohort/panel, and anecdotal evidence of phr benefits and satisfaction for patients, more research is needed to evaluate the results of phr implementations. Their in-depth analysis of the literature signalled that there is little solid evidence from randomized controlled trials or other studies through the use of phr s. Hence, they suggested that more research is needed that addresses the current lack of understanding of optimal functionality and usability of these systems, and how they can play a beneficial role in supporting patient self-management ( Archer et al., 2011 ).

9.3.4. Forms of Aggregative Reviews

Healthcare providers, practitioners, and policy-makers are nowadays overwhelmed with large volumes of information, including research-based evidence from numerous clinical trials and evaluation studies, assessing the effectiveness of health information technologies and interventions ( Ammenwerth & de Keizer, 2004 ; Deshazo et al., 2009 ). It is unrealistic to expect that all these disparate actors will have the time, skills, and necessary resources to identify the available evidence in the area of their expertise and consider it when making decisions. Systematic reviews that involve the rigorous application of scientific strategies aimed at limiting subjectivity and bias (i.e., systematic and random errors) can respond to this challenge.

Systematic reviews attempt to aggregate, appraise, and synthesize in a single source all empirical evidence that meet a set of previously specified eligibility criteria in order to answer a clearly formulated and often narrow research question on a particular topic of interest to support evidence-based practice ( Liberati et al., 2009 ). They adhere closely to explicit scientific principles ( Liberati et al., 2009 ) and rigorous methodological guidelines (Higgins & Green, 2008) aimed at reducing random and systematic errors that can lead to deviations from the truth in results or inferences. The use of explicit methods allows systematic reviews to aggregate a large body of research evidence, assess whether effects or relationships are in the same direction and of the same general magnitude, explain possible inconsistencies between study results, and determine the strength of the overall evidence for every outcome of interest based on the quality of included studies and the general consistency among them ( Cook, Mulrow, & Haynes, 1997 ). The main procedures of a systematic review involve:

• Formulating a review question and developing a search strategy based on explicit inclusion criteria for the identification of eligible studies (usually described in the context of a detailed review protocol).
• Searching for eligible studies using multiple databases and information sources, including grey literature sources, without any language restrictions.
• Selecting studies, extracting data, and assessing risk of bias in a duplicate manner using two independent reviewers to avoid random or systematic errors in the process.
• Analyzing data using quantitative or qualitative methods.
• Presenting results in summary of findings tables.
• Interpreting results and drawing conclusions.

Many systematic reviews, but not all, use statistical methods to combine the results of independent studies into a single quantitative estimate or summary effect size. Known as meta-analyses , these reviews use specific data extraction and statistical techniques (e.g., network, frequentist, or Bayesian meta-analyses) to calculate from each study by outcome of interest an effect size along with a confidence interval that reflects the degree of uncertainty behind the point estimate of effect ( Borenstein, Hedges, Higgins, & Rothstein, 2009 ; Deeks, Higgins, & Altman, 2008 ). Subsequently, they use fixed or random-effects analysis models to combine the results of the included studies, assess statistical heterogeneity, and calculate a weighted average of the effect estimates from the different studies, taking into account their sample sizes. The summary effect size is a value that reflects the average magnitude of the intervention effect for a particular outcome of interest or, more generally, the strength of a relationship between two variables across all studies included in the systematic review. By statistically combining data from multiple studies, meta-analyses can create more precise and reliable estimates of intervention effects than those derived from individual studies alone, when these are examined independently as discrete sources of information.

The review by Gurol-Urganci, de Jongh, Vodopivec-Jamsek, Atun, and Car (2013) on the effects of mobile phone messaging reminders for attendance at healthcare appointments is an illustrative example of a high-quality systematic review with meta-analysis. Missed appointments are a major cause of inefficiency in healthcare delivery with substantial monetary costs to health systems. These authors sought to assess whether mobile phone-based appointment reminders delivered through Short Message Service ( sms ) or Multimedia Messaging Service ( mms ) are effective in improving rates of patient attendance and reducing overall costs. To this end, they conducted a comprehensive search on multiple databases using highly sensitive search strategies without language or publication-type restrictions to identify all rct s that are eligible for inclusion. In order to minimize the risk of omitting eligible studies not captured by the original search, they supplemented all electronic searches with manual screening of trial registers and references contained in the included studies. Study selection, data extraction, and risk of bias assessments were performed inde­­pen­dently by two coders using standardized methods to ensure consistency and to eliminate potential errors. Findings from eight rct s involving 6,615 participants were pooled into meta-analyses to calculate the magnitude of effects that mobile text message reminders have on the rate of attendance at healthcare appointments compared to no reminders and phone call reminders.

Meta-analyses are regarded as powerful tools for deriving meaningful conclusions. However, there are situations in which it is neither reasonable nor appropriate to pool studies together using meta-analytic methods simply because there is extensive clinical heterogeneity between the included studies or variation in measurement tools, comparisons, or outcomes of interest. In these cases, systematic reviews can use qualitative synthesis methods such as vote counting, content analysis, classification schemes and tabulations, as an alternative approach to narratively synthesize the results of the independent studies included in the review. This form of review is known as qualitative systematic review.

A rigorous example of one such review in the eHealth domain is presented by Mickan, Atherton, Roberts, Heneghan, and Tilson (2014) on the use of handheld computers by healthcare professionals and their impact on access to information and clinical decision-making. In line with the methodological guide­lines for systematic reviews, these authors: (a) developed and registered with prospero ( www.crd.york.ac.uk/ prospero / ) an a priori review protocol; (b) conducted comprehensive searches for eligible studies using multiple databases and other supplementary strategies (e.g., forward searches); and (c) subsequently carried out study selection, data extraction, and risk of bias assessments in a duplicate manner to eliminate potential errors in the review process. Heterogeneity between the included studies in terms of reported outcomes and measures precluded the use of meta-analytic methods. To this end, the authors resorted to using narrative analysis and synthesis to describe the effectiveness of handheld computers on accessing information for clinical knowledge, adherence to safety and clinical quality guidelines, and diagnostic decision-making.

In recent years, the number of systematic reviews in the field of health informatics has increased considerably. Systematic reviews with discordant findings can cause great confusion and make it difficult for decision-makers to interpret the review-level evidence ( Moher, 2013 ). Therefore, there is a growing need for appraisal and synthesis of prior systematic reviews to ensure that decision-making is constantly informed by the best available accumulated evidence. Umbrella reviews , also known as overviews of systematic reviews, are tertiary types of evidence synthesis that aim to accomplish this; that is, they aim to compare and contrast findings from multiple systematic reviews and meta-analyses ( Becker & Oxman, 2008 ). Umbrella reviews generally adhere to the same principles and rigorous methodological guidelines used in systematic reviews. However, the unit of analysis in umbrella reviews is the systematic review rather than the primary study ( Becker & Oxman, 2008 ). Unlike systematic reviews that have a narrow focus of inquiry, umbrella reviews focus on broader research topics for which there are several potential interventions ( Smith, Devane, Begley, & Clarke, 2011 ). A recent umbrella review on the effects of home telemonitoring interventions for patients with heart failure critically appraised, compared, and synthesized evidence from 15 systematic reviews to investigate which types of home telemonitoring technologies and forms of interventions are more effective in reducing mortality and hospital admissions ( Kitsiou, Paré, & Jaana, 2015 ).

9.3.5. Realist Reviews

Realist reviews are theory-driven interpretative reviews developed to inform, enhance, or supplement conventional systematic reviews by making sense of heterogeneous evidence about complex interventions applied in diverse contexts in a way that informs policy decision-making ( Greenhalgh, Wong, Westhorp, & Pawson, 2011 ). They originated from criticisms of positivist systematic reviews which centre on their “simplistic” underlying assumptions ( Oates, 2011 ). As explained above, systematic reviews seek to identify causation. Such logic is appropriate for fields like medicine and education where findings of randomized controlled trials can be aggregated to see whether a new treatment or intervention does improve outcomes. However, many argue that it is not possible to establish such direct causal links between interventions and outcomes in fields such as social policy, management, and information systems where for any intervention there is unlikely to be a regular or consistent outcome ( Oates, 2011 ; Pawson, 2006 ; Rousseau, Manning, & Denyer, 2008 ).

To circumvent these limitations, Pawson, Greenhalgh, Harvey, and Walshe (2005) have proposed a new approach for synthesizing knowledge that seeks to unpack the mechanism of how “complex interventions” work in particular contexts. The basic research question — what works? — which is usually associated with systematic reviews changes to: what is it about this intervention that works, for whom, in what circumstances, in what respects and why? Realist reviews have no particular preference for either quantitative or qualitative evidence. As a theory-building approach, a realist review usually starts by articulating likely underlying mechanisms and then scrutinizes available evidence to find out whether and where these mechanisms are applicable ( Shepperd et al., 2009 ). Primary studies found in the extant literature are viewed as case studies which can test and modify the initial theories ( Rousseau et al., 2008 ).

The main objective pursued in the realist review conducted by Otte-Trojel, de Bont, Rundall, and van de Klundert (2014) was to examine how patient portals contribute to health service delivery and patient outcomes. The specific goals were to investigate how outcomes are produced and, most importantly, how variations in outcomes can be explained. The research team started with an exploratory review of background documents and research studies to identify ways in which patient portals may contribute to health service delivery and patient outcomes. The authors identified six main ways which represent “educated guesses” to be tested against the data in the evaluation studies. These studies were identified through a formal and systematic search in four databases between 2003 and 2013. Two members of the research team selected the articles using a pre-established list of inclusion and exclusion criteria and following a two-step procedure. The authors then extracted data from the selected articles and created several tables, one for each outcome category. They organized information to bring forward those mechanisms where patient portals contribute to outcomes and the variation in outcomes across different contexts.

9.3.6. Critical Reviews

Lastly, critical reviews aim to provide a critical evaluation and interpretive analysis of existing literature on a particular topic of interest to reveal strengths, weaknesses, contradictions, controversies, inconsistencies, and/or other important issues with respect to theories, hypotheses, research methods or results ( Baumeister & Leary, 1997 ; Kirkevold, 1997 ). Unlike other review types, critical reviews attempt to take a reflective account of the research that has been done in a particular area of interest, and assess its credibility by using appraisal instruments or critical interpretive methods. In this way, critical reviews attempt to constructively inform other scholars about the weaknesses of prior research and strengthen knowledge development by giving focus and direction to studies for further improvement ( Kirkevold, 1997 ).

Kitsiou, Paré, and Jaana (2013) provide an example of a critical review that assessed the methodological quality of prior systematic reviews of home telemonitoring studies for chronic patients. The authors conducted a comprehensive search on multiple databases to identify eligible reviews and subsequently used a validated instrument to conduct an in-depth quality appraisal. Results indicate that the majority of systematic reviews in this particular area suffer from important methodological flaws and biases that impair their internal validity and limit their usefulness for clinical and decision-making purposes. To this end, they provide a number of recommendations to strengthen knowledge development towards improving the design and execution of future reviews on home telemonitoring.

9.4. Summary

Table 9.1 outlines the main types of literature reviews that were described in the previous sub-sections and summarizes the main characteristics that distinguish one review type from another. It also includes key references to methodological guidelines and useful sources that can be used by eHealth scholars and researchers for planning and developing reviews.

Typology of Literature Reviews (adapted from Paré et al., 2015).

As shown in Table 9.1 , each review type addresses different kinds of research questions or objectives, which subsequently define and dictate the methods and approaches that need to be used to achieve the overarching goal(s) of the review. For example, in the case of narrative reviews, there is greater flexibility in searching and synthesizing articles ( Green et al., 2006 ). Researchers are often relatively free to use a diversity of approaches to search, identify, and select relevant scientific articles, describe their operational characteristics, present how the individual studies fit together, and formulate conclusions. On the other hand, systematic reviews are characterized by their high level of systematicity, rigour, and use of explicit methods, based on an “a priori” review plan that aims to minimize bias in the analysis and synthesis process (Higgins & Green, 2008). Some reviews are exploratory in nature (e.g., scoping/mapping reviews), whereas others may be conducted to discover patterns (e.g., descriptive reviews) or involve a synthesis approach that may include the critical analysis of prior research ( Paré et al., 2015 ). Hence, in order to select the most appropriate type of review, it is critical to know before embarking on a review project, why the research synthesis is conducted and what type of methods are best aligned with the pursued goals.

9.5. Concluding Remarks

In light of the increased use of evidence-based practice and research generating stronger evidence ( Grady et al., 2011 ; Lyden et al., 2013 ), review articles have become essential tools for summarizing, synthesizing, integrating or critically appraising prior knowledge in the eHealth field. As mentioned earlier, when rigorously conducted review articles represent powerful information sources for eHealth scholars and practitioners looking for state-of-the-art evidence. The typology of literature reviews we used herein will allow eHealth researchers, graduate students and practitioners to gain a better understanding of the similarities and differences between review types.

We must stress that this classification scheme does not privilege any specific type of review as being of higher quality than another ( Paré et al., 2015 ). As explained above, each type of review has its own strengths and limitations. Having said that, we realize that the methodological rigour of any review — be it qualitative, quantitative or mixed — is a critical aspect that should be considered seriously by prospective authors. In the present context, the notion of rigour refers to the reliability and validity of the review process described in section 9.2. For one thing, reliability is related to the reproducibility of the review process and steps, which is facilitated by a comprehensive documentation of the literature search process, extraction, coding and analysis performed in the review. Whether the search is comprehensive or not, whether it involves a methodical approach for data extraction and synthesis or not, it is important that the review documents in an explicit and transparent manner the steps and approach that were used in the process of its development. Next, validity characterizes the degree to which the review process was conducted appropriately. It goes beyond documentation and reflects decisions related to the selection of the sources, the search terms used, the period of time covered, the articles selected in the search, and the application of backward and forward searches ( vom Brocke et al., 2009 ). In short, the rigour of any review article is reflected by the explicitness of its methods (i.e., transparency) and the soundness of the approach used. We refer those interested in the concepts of rigour and quality to the work of Templier and Paré (2015) which offers a detailed set of methodological guidelines for conducting and evaluating various types of review articles.

To conclude, our main objective in this chapter was to demystify the various types of literature reviews that are central to the continuous development of the eHealth field. It is our hope that our descriptive account will serve as a valuable source for those conducting, evaluating or using reviews in this important and growing domain.

• Ammenwerth E., de Keizer N. An inventory of evaluation studies of information technology in health care. Trends in evaluation research, 1982-2002. International Journal of Medical Informatics. 2004; 44 (1):44–56. [ PubMed : 15778794 ]
• Anderson S., Allen P., Peckham S., Goodwin N. Asking the right questions: scoping studies in the commissioning of research on the organisation and delivery of health services. Health Research Policy and Systems. 2008; 6 (7):1–12. [ PMC free article : PMC2500008 ] [ PubMed : 18613961 ] [ CrossRef ]
• Archer N., Fevrier-Thomas U., Lokker C., McKibbon K. A., Straus S.E. Personal health records: a scoping review. Journal of American Medical Informatics Association. 2011; 18 (4):515–522. [ PMC free article : PMC3128401 ] [ PubMed : 21672914 ]
• Arksey H., O’Malley L. Scoping studies: towards a methodological framework. International Journal of Social Research Methodology. 2005; 8 (1):19–32.
• A systematic, tool-supported method for conducting literature reviews in information systems. Paper presented at the Proceedings of the 19th European Conference on Information Systems ( ecis 2011); June 9 to 11; Helsinki, Finland. 2011.
• Baumeister R. F., Leary M.R. Writing narrative literature reviews. Review of General Psychology. 1997; 1 (3):311–320.
• Becker L. A., Oxman A.D. In: Cochrane handbook for systematic reviews of interventions. Higgins J. P. T., Green S., editors. Hoboken, nj : John Wiley & Sons, Ltd; 2008. Overviews of reviews; pp. 607–631.
• Borenstein M., Hedges L., Higgins J., Rothstein H. Introduction to meta-analysis. Hoboken, nj : John Wiley & Sons Inc; 2009.
• Cook D. J., Mulrow C. D., Haynes B. Systematic reviews: Synthesis of best evidence for clinical decisions. Annals of Internal Medicine. 1997; 126 (5):376–380. [ PubMed : 9054282 ]
• Cooper H., Hedges L.V. In: The handbook of research synthesis and meta-analysis. 2nd ed. Cooper H., Hedges L. V., Valentine J. C., editors. New York: Russell Sage Foundation; 2009. Research synthesis as a scientific process; pp. 3–17.
• Cooper H. M. Organizing knowledge syntheses: A taxonomy of literature reviews. Knowledge in Society. 1988; 1 (1):104–126.
• Cronin P., Ryan F., Coughlan M. Undertaking a literature review: a step-by-step approach. British Journal of Nursing. 2008; 17 (1):38–43. [ PubMed : 18399395 ]
• Darlow S., Wen K.Y. Development testing of mobile health interventions for cancer patient self-management: A review. Health Informatics Journal. 2015 (online before print). [ PubMed : 25916831 ] [ CrossRef ]
• Daudt H. M., van Mossel C., Scott S.J. Enhancing the scoping study methodology: a large, inter-professional team’s experience with Arksey and O’Malley’s framework. bmc Medical Research Methodology. 2013; 13 :48. [ PMC free article : PMC3614526 ] [ PubMed : 23522333 ] [ CrossRef ]
• Davies P. The relevance of systematic reviews to educational policy and practice. Oxford Review of Education. 2000; 26 (3-4):365–378.
• Deeks J. J., Higgins J. P. T., Altman D.G. In: Cochrane handbook for systematic reviews of interventions. Higgins J. P. T., Green S., editors. Hoboken, nj : John Wiley & Sons, Ltd; 2008. Analysing data and undertaking meta-analyses; pp. 243–296.
• Deshazo J. P., Lavallie D. L., Wolf F.M. Publication trends in the medical informatics literature: 20 years of “Medical Informatics” in mesh . bmc Medical Informatics and Decision Making. 2009; 9 :7. [ PMC free article : PMC2652453 ] [ PubMed : 19159472 ] [ CrossRef ]
• Dixon-Woods M., Agarwal S., Jones D., Young B., Sutton A. Synthesising qualitative and quantitative evidence: a review of possible methods. Journal of Health Services Research and Policy. 2005; 10 (1):45–53. [ PubMed : 15667704 ]
• Finfgeld-Connett D., Johnson E.D. Literature search strategies for conducting knowledge-building and theory-generating qualitative systematic reviews. Journal of Advanced Nursing. 2013; 69 (1):194–204. [ PMC free article : PMC3424349 ] [ PubMed : 22591030 ]
• Grady B., Myers K. M., Nelson E. L., Belz N., Bennett L., Carnahan L. … Guidelines Working Group. Evidence-based practice for telemental health. Telemedicine Journal and E Health. 2011; 17 (2):131–148. [ PubMed : 21385026 ]
• Green B. N., Johnson C. D., Adams A. Writing narrative literature reviews for peer-reviewed journals: secrets of the trade. Journal of Chiropractic Medicine. 2006; 5 (3):101–117. [ PMC free article : PMC2647067 ] [ PubMed : 19674681 ]
• Greenhalgh T., Wong G., Westhorp G., Pawson R. Protocol–realist and meta-narrative evidence synthesis: evolving standards ( rameses ). bmc Medical Research Methodology. 2011; 11 :115. [ PMC free article : PMC3173389 ] [ PubMed : 21843376 ]
• Gurol-Urganci I., de Jongh T., Vodopivec-Jamsek V., Atun R., Car J. Mobile phone messaging reminders for attendance at healthcare appointments. Cochrane Database System Review. 2013; 12 cd 007458. [ PMC free article : PMC6485985 ] [ PubMed : 24310741 ] [ CrossRef ]
• Hart C. Doing a literature review: Releasing the social science research imagination. London: SAGE Publications; 1998.
• Higgins J. P. T., Green S., editors. Cochrane handbook for systematic reviews of interventions: Cochrane book series. Hoboken, nj : Wiley-Blackwell; 2008.
• Jesson J., Matheson L., Lacey F.M. Doing your literature review: traditional and systematic techniques. Los Angeles & London: SAGE Publications; 2011.
• King W. R., He J. Understanding the role and methods of meta-analysis in IS research. Communications of the Association for Information Systems. 2005; 16 :1.
• Kirkevold M. Integrative nursing research — an important strategy to further the development of nursing science and nursing practice. Journal of Advanced Nursing. 1997; 25 (5):977–984. [ PubMed : 9147203 ]
• Kitchenham B., Charters S. ebse Technical Report Version 2.3. Keele & Durham. uk : Keele University & University of Durham; 2007. Guidelines for performing systematic literature reviews in software engineering.
• Kitsiou S., Paré G., Jaana M. Systematic reviews and meta-analyses of home telemonitoring interventions for patients with chronic diseases: a critical assessment of their methodological quality. Journal of Medical Internet Research. 2013; 15 (7):e150. [ PMC free article : PMC3785977 ] [ PubMed : 23880072 ]
• Kitsiou S., Paré G., Jaana M. Effects of home telemonitoring interventions on patients with chronic heart failure: an overview of systematic reviews. Journal of Medical Internet Research. 2015; 17 (3):e63. [ PMC free article : PMC4376138 ] [ PubMed : 25768664 ]
• Levac D., Colquhoun H., O’Brien K. K. Scoping studies: advancing the methodology. Implementation Science. 2010; 5 (1):69. [ PMC free article : PMC2954944 ] [ PubMed : 20854677 ]
• Levy Y., Ellis T.J. A systems approach to conduct an effective literature review in support of information systems research. Informing Science. 2006; 9 :181–211.
• Liberati A., Altman D. G., Tetzlaff J., Mulrow C., Gøtzsche P. C., Ioannidis J. P. A. et al. Moher D. The prisma statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: Explanation and elaboration. Annals of Internal Medicine. 2009; 151 (4):W-65. [ PubMed : 19622512 ]
• Lyden J. R., Zickmund S. L., Bhargava T. D., Bryce C. L., Conroy M. B., Fischer G. S. et al. McTigue K. M. Implementing health information technology in a patient-centered manner: Patient experiences with an online evidence-based lifestyle intervention. Journal for Healthcare Quality. 2013; 35 (5):47–57. [ PubMed : 24004039 ]
• Mickan S., Atherton H., Roberts N. W., Heneghan C., Tilson J.K. Use of handheld computers in clinical practice: a systematic review. bmc Medical Informatics and Decision Making. 2014; 14 :56. [ PMC free article : PMC4099138 ] [ PubMed : 24998515 ]
• Moher D. The problem of duplicate systematic reviews. British Medical Journal. 2013; 347 (5040) [ PubMed : 23945367 ] [ CrossRef ]
• Montori V. M., Wilczynski N. L., Morgan D., Haynes R. B., Hedges T. Systematic reviews: a cross-sectional study of location and citation counts. bmc Medicine. 2003; 1 :2. [ PMC free article : PMC281591 ] [ PubMed : 14633274 ]
• Mulrow C. D. The medical review article: state of the science. Annals of Internal Medicine. 1987; 106 (3):485–488. [ PubMed : 3813259 ] [ CrossRef ]
• Evidence-based information systems: A decade later. Proceedings of the European Conference on Information Systems ; 2011. Retrieved from http://aisel ​.aisnet.org/cgi/viewcontent ​.cgi?article ​=1221&context ​=ecis2011 .
• Okoli C., Schabram K. A guide to conducting a systematic literature review of information systems research. ssrn Electronic Journal. 2010
• Otte-Trojel T., de Bont A., Rundall T. G., van de Klundert J. How outcomes are achieved through patient portals: a realist review. Journal of American Medical Informatics Association. 2014; 21 (4):751–757. [ PMC free article : PMC4078283 ] [ PubMed : 24503882 ]
• Paré G., Trudel M.-C., Jaana M., Kitsiou S. Synthesizing information systems knowledge: A typology of literature reviews. Information & Management. 2015; 52 (2):183–199.
• Patsopoulos N. A., Analatos A. A., Ioannidis J.P. A. Relative citation impact of various study designs in the health sciences. Journal of the American Medical Association. 2005; 293 (19):2362–2366. [ PubMed : 15900006 ]
• Paul M. M., Greene C. M., Newton-Dame R., Thorpe L. E., Perlman S. E., McVeigh K. H., Gourevitch M.N. The state of population health surveillance using electronic health records: A narrative review. Population Health Management. 2015; 18 (3):209–216. [ PubMed : 25608033 ]
• Pawson R. Evidence-based policy: a realist perspective. London: SAGE Publications; 2006.
• Pawson R., Greenhalgh T., Harvey G., Walshe K. Realist review—a new method of systematic review designed for complex policy interventions. Journal of Health Services Research & Policy. 2005; 10 (Suppl 1):21–34. [ PubMed : 16053581 ]
• Petersen K., Vakkalanka S., Kuzniarz L. Guidelines for conducting systematic mapping studies in software engineering: An update. Information and Software Technology. 2015; 64 :1–18.
• Petticrew M., Roberts H. Systematic reviews in the social sciences: A practical guide. Malden, ma : Blackwell Publishing Co; 2006.
• Rousseau D. M., Manning J., Denyer D. Evidence in management and organizational science: Assembling the field’s full weight of scientific knowledge through syntheses. The Academy of Management Annals. 2008; 2 (1):475–515.
• Rowe F. What literature review is not: diversity, boundaries and recommendations. European Journal of Information Systems. 2014; 23 (3):241–255.
• Shea B. J., Hamel C., Wells G. A., Bouter L. M., Kristjansson E., Grimshaw J. et al. Boers M. amstar is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. Journal of Clinical Epidemiology. 2009; 62 (10):1013–1020. [ PubMed : 19230606 ]
• Shepperd S., Lewin S., Straus S., Clarke M., Eccles M. P., Fitzpatrick R. et al. Sheikh A. Can we systematically review studies that evaluate complex interventions? PLoS Medicine. 2009; 6 (8):e1000086. [ PMC free article : PMC2717209 ] [ PubMed : 19668360 ]
• Silva B. M., Rodrigues J. J., de la Torre Díez I., López-Coronado M., Saleem K. Mobile-health: A review of current state in 2015. Journal of Biomedical Informatics. 2015; 56 :265–272. [ PubMed : 26071682 ]
• Smith V., Devane D., Begley C., Clarke M. Methodology in conducting a systematic review of systematic reviews of healthcare interventions. bmc Medical Research Methodology. 2011; 11 (1):15. [ PMC free article : PMC3039637 ] [ PubMed : 21291558 ]
• Sylvester A., Tate M., Johnstone D. Beyond synthesis: re-presenting heterogeneous research literature. Behaviour & Information Technology. 2013; 32 (12):1199–1215.
• Templier M., Paré G. A framework for guiding and evaluating literature reviews. Communications of the Association for Information Systems. 2015; 37 (6):112–137.
• Thomas J., Harden A. Methods for the thematic synthesis of qualitative research in systematic reviews. bmc Medical Research Methodology. 2008; 8 (1):45. [ PMC free article : PMC2478656 ] [ PubMed : 18616818 ]
• Reconstructing the giant: on the importance of rigour in documenting the literature search process. Paper presented at the Proceedings of the 17th European Conference on Information Systems ( ecis 2009); Verona, Italy. 2009.
• Webster J., Watson R.T. Analyzing the past to prepare for the future: Writing a literature review. Management Information Systems Quarterly. 2002; 26 (2):11.
• Whitlock E. P., Lin J. S., Chou R., Shekelle P., Robinson K.A. Using existing systematic reviews in complex systematic reviews. Annals of Internal Medicine. 2008; 148 (10):776–782. [ PubMed : 18490690 ]

This publication is licensed under a Creative Commons License, Attribution-Noncommercial 4.0 International License (CC BY-NC 4.0): see https://creativecommons.org/licenses/by-nc/4.0/

• Cite this Page Paré G, Kitsiou S. Chapter 9 Methods for Literature Reviews. In: Lau F, Kuziemsky C, editors. Handbook of eHealth Evaluation: An Evidence-based Approach [Internet]. Victoria (BC): University of Victoria; 2017 Feb 27.
• PDF version of this title (4.5M)

In this Page

• Introduction
• Overview of the Literature Review Process and Steps
• Types of Review Articles and Brief Illustrations
• Concluding Remarks

Related information

• PMC PubMed Central citations
• PubMed Links to PubMed

Recent Activity

• Chapter 9 Methods for Literature Reviews - Handbook of eHealth Evaluation: An Ev... Chapter 9 Methods for Literature Reviews - Handbook of eHealth Evaluation: An Evidence-based Approach

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

Connect with NLM

National Library of Medicine 8600 Rockville Pike Bethesda, MD 20894

Web Policies FOIA HHS Vulnerability Disclosure

Help Accessibility Careers

Information

• Author Services

Initiatives

You are accessing a machine-readable page. In order to be human-readable, please install an RSS reader.

All articles published by MDPI are made immediately available worldwide under an open access license. No special permission is required to reuse all or part of the article published by MDPI, including figures and tables. For articles published under an open access Creative Common CC BY license, any part of the article may be reused without permission provided that the original article is clearly cited. For more information, please refer to https://www.mdpi.com/openaccess .

Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for future research directions and describes possible research applications.

Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive positive feedback from the reviewers.

Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world. Editors select a small number of articles recently published in the journal that they believe will be particularly interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the most exciting work published in the various research areas of the journal.

Original Submission Date Received: .

• Active Journals
• Find a Journal
• Proceedings Series
• For Authors
• For Reviewers
• For Editors
• For Librarians
• For Publishers
• For Societies
• For Conference Organizers
• Open Access Policy
• Institutional Open Access Program
• Special Issues Guidelines
• Editorial Process
• Research and Publication Ethics
• Article Processing Charges
• Testimonials
• Preprints.org
• SciProfiles
• Encyclopedia

Article Menu

• Subscribe SciFeed
• Recommended Articles
• Google Scholar
• on Google Scholar
• Table of Contents

Find support for a specific problem in the support section of our website.

Please let us know what you think of our products and services.

Visit our dedicated information section to learn more about MDPI.

JSmol Viewer

Understanding the wellbeing needs of first nations children in out-of-home care in australia: a comprehensive literature review.

1. Introduction

2. materials and methods, 2.1. research team, 2.2. search strategy, 2.3. eligibility criteria, 2.4. study selection, 2.5. data collection and analysis, 3.1. paper characteristics, 3.2. qualitative results, 3.3. being seen, being heard, 3.3.1. having autonomy and agency.

Talk to the actual kids, they’ll be very honest about their placement they don’t hide nothing get new workers to build trust with them kids and they will just spill their guts really . (Sally, worker) [ 47 ]

I think it’s like ya get dropped out of the loop… But basically we all are feeling so left out of things, these things are happening, we might be young but some sort of explanation would just go a mile… (Caroline, post-care, 19, Aboriginal) [ 33 ]

In some cases, while FACS [Family and Community Services] involved the children in some decision-making, the reviewer identified that this consultation was not ongoing. Children have the right to be involved in decisions that affect them and impact their lives, and this failure to consult on an ongoing basis was identified as disempowering practice . (Case file reviewer) [ 38 ]

3.3.2. Being Treated Like a Child, Not a Number

It is so important to know the kids you are working with; each person is an individual. It is just bloody critical that these kids are seen, known and not just a number. This is the work I am so passionate about . (Child Protection staff member) [ 33 ]

They should listen to young people. Hear what they have to say to ask them what they think and if they are safe . (15-year-old First Nations girl in care) [ 52 ]

I was lucky ’cos my grandparents made me feel like I was part of the family. I was never, ever introduced as their foster child and that made me feel loved and appreciated. I would encourage foster carers to try and do that, that was the most important thing . (Caden, post-care, 19, Aboriginal) [ 33 ]

My foster carers…were Aboriginal. They taught me stuff about culture. They helped me keep in contact with family. I stuck with one for most of it… They listened to me… [Carer] was really understanding. He understood why I was misbehaving sometimes . (Phoebe, returned home, 16, Aboriginal) [ 33 ]

3.4. A Sense of Stability

3.4.1. experiencing placement stability.

Stability for Aboriginal people is grounded in their sense of identity in connection to family, kin, culture and country. In our view, permanent care/adoption potentially places an emphasis on achieving stability of living arrangements and a secure legal status potentially at the cost of the child’s identity and enduring relationships with their extended family and connection with community and culture . (Victorian Aboriginal Child Care Agency) [ 34 ]

3.4.2. Receiving Support in School

I always tell them [First Nations children in OOHC engaging in education] who I am and what I’m there to do. And then I ask them if they want to. Because I make sure that they are involved in the decision making of being involved. And I did have two kids go, ‘No. I’m not quite sure’. But then in the end, became involved. I think part of it is also listening to them about what they want . (Education engagement intervention program teacher/mentor) [ 62 ]

3.4.3. Being on a Pathway to Culturally Appropriate Permanency

SNAICC submitted that permanency for Aboriginal children was ‘tied to existing identity, kinship relationships, and connections to culture and country’, and that it was important not to permanently deprive children of these connections through the application of ‘inflexible permanency planning measures . (SNAICC) [ 38 ]

3.5. Holistic Health Support

3.5.1. fulfilment of basic needs.

Most of us kids, the reason why we are in care is because our families are not reliable. You know, money problems, food, clothes, safety problems… The whole reason why they took us off our family was because we feel unsafe, we don’t feel much protected, there’s no food, and we’re not getting clothes… we’re not getting anything. But what’s the point of that if they do exactly the same in all these houses. It’s not better either way: living with our family, living with DCP [Department for Child Protection], government homes… or living on the streets… it’s not good anywhere . (17-year-old Aboriginal male, residential care) [ 35 ]

3.5.2. Receiving Care for Health and Physical Wellbeing

I think another trend that we found is that we’ve got a number of young people who have gone through the care system to be diagnosed as foetal alcohol syndrome at 18. And they’ve already been in and out of detention and they’ve got involvement with the justice system, and now they’re 18, it’s the adult justice system, which is a real concern. One young fella in particular I’m thinking of, was actually in residential care and wasn’t diagnosed until he was 18 . (Western Australian NGO) [ 49 ]

3.5.3. Provision of Trauma-Informed Care

FACS fails to acknowledge that the removal of Aboriginal children from their families often exposes them to danger and ‘immense trauma’, as opposed to ‘protection’, (National Congress of Australia’s First Peoples) and that FACS intervention in and of itself is an extremely arduous, traumatic process that is actively harmful to all involved, particularly children . (Grandmothers Against Removals New South Wales) [ 38 ]

A lot of kids have had severe trauma, been too exposed to a lot of negative experiences, and you can see it, like behavioural change. A lot of the kids are getting suspended all the time, they’re acting out, they just show all the different traits, like physically, emotionally. You can see, spiritually, that they’re impacted too, on a lot of different levels. Their confidence is low, self-esteem, yeah, just a lot of different things . (NSW ACCO) [ 49 ]

While in placement, with the support of a strong and therapeutic care team, an appropriate cultural support plan and a KESO [Koorie Engagement Support Officer], Molly’s [Aboriginal girl in OOHC] behaviours have settled. Molly has told child protection she feels safe and secure with her carers . (caseworker/reviewer) [ 42 ]

3.6. Social and Cultural Connections

3.6.1. fostering interconnected relationships.

The major difficulty in the urban setting was appropriately placing children culturally, working out where they belonged . (ACCO staff) [ 65 ]

3.6.2. Maintaining Cultural Knowledge and Identity

Being Aboriginal is the proudest thing in my life, to know that that’s my people. It made me so proud to see what we’ve actually done and how far we’ve come to this day. It taught me that no matter what, I can still get up and do what I want . (Aboriginal child in OOHC) [ 36 ]

They [First Nations children and youth upon entering cultural camps] didn’t know their connections to communities, didn’t know about the language, didn’t eat Aboriginal food, they knew nothing at all [of their culture] . (Aboriginal education officer) [ 36 ]

3.6.3. Feeling Connected to Community and Country

Aboriginal children coming into care should be placed in their own country. Just because they’re Aboriginal, isn’t good enough. You need to be placed with people who know your identity . (Non-First Nations carer) [ 54 ]

Participants identified a strong cultural identity and effective connection with community as a powerful source of resilience for Indigenous young people during and post transition from care . (ACCO and Government OOHC workers) [ 50 ]

3.6.4. Continued Links to Family and Kin

Give Aboriginal kids back to their home, their family, after you’ve gone through and made sure everything is all safe and all good. If not the mother and father, then maybe the kid has sisters, aunties, or an Aboriginal carer is available . (Aboriginal caregiver) [ 36 ]

Living there [in kinship care] feels like a family . (Shane, kinship care, 15, Aboriginal) [ 33 ]

Well, the strength [of kinship care] is that children remain within their extended family, which supports our philosophy around self-determination, self-management. The family best knows the family circumstances . (Jenny, worker) [ 37 ]

His [First Nations, 8 year old boy in relative care] older siblings were scattered geographically but it was clear from his narrative that he wanted regular contact with his older siblings . (OOHC team leader and art therapist) [ 52 ]

If I need to talk to someone now, my brother would be the first person I would talk to . (Ellie, residential care, 16, Aboriginal) [ 33 ]

I had someone sit down with me and go through everything, my mob, my family. There is nothing else I need to know . (Female, First Nations, 17 years) [ 48 ]

I want to find out if I have a cultural support plan so I can get help finding more info about my culture and where my family was from . (Female, First Nations, 14 years) [ 48 ]

Few months ago I asked [Department of Health and Human Services, Victoria] if I could find my dad. Haven’t seen him since I was one. Part of my life I’ve never met, so not good. My dad is the only actual family I know . (Evan, foster care, 15, Aboriginal) [ 33 ]

3.6.5. Being Supported by Friends

Q: Who do you go to for support? My friends, but more like my best friends. I’ve known them since I was like three and we’ve always stayed in contact and if I have a problem on my mind, I can always just go to his house . (Ethan, kinship care, 15, Aboriginal) [ 33 ]

3.7. Culturally Safe OOHC Providers

3.7.1. supported by oohc organizations trusted by first nations peoples.

We understand where people [Aboriginal families] come from you can’t just have a mainstream organisation culturally competent, its philosophy is driven by white people, how they were raised, how they understand programs and services . (ACCO staff) [ 47 ]

3.7.2. Provision of Support Services Grounded in Culturally Safe Approaches

Aboriginal community-controlled agencies are best placed to support Aboriginal children and young people in OOHC, including maintaining their connection to family, community, culture and Country that is central to identity development and wellbeing . (New South Wales Council of Social Service) [ 38 ]

3.8. Preparedness for Transitioning Out of Care

3.8.1. given adequate opportunities for reunification with family.

He [14-year-old Aboriginal/South Sea Islander boy in OOHC] showed little attachment to the carer in that he talked of running away and not needing anyone . (OOHC team leader and art therapist) [ 52 ]

We’ve got lots of kids walking from care and leaving at 15. And particularly going back to Country or trying to find Country . (New South Wales NGO) [ 49 ]

3.8.2. Provided with Life Skills for after Care

We know there are 16, 17, 18-year-olds out there that can’t even boil water, you know, yet they want to fall pregnant; so if you can get it in there early enough to get these old people to teach these children survival skills, and not just Indigenous (skills), but also how to cook a meal and sew a button on . (Carer) [ 39 ]

4. Discussion

Strengths and limitations, 5. conclusions, supplementary materials, author contributions, data availability statement, acknowledgments, conflicts of interest.

• Commonwealth of Australia. An Outline of National Standards for Out-of-Home Care ; Department of Families, Housing, Community Services and Indigenous Affairs, National Framework Implementation Working Group: Canberra, Australia, 2011. [ Google Scholar ]
• Australian Institute of Health and Welfare. Child protection Australia 2020–2021. Available online: https://www.aihw.gov.au/reports/child-protection/child-protection-australia-2020-21 (accessed on 30 September 2022).
• Liddle, C.; Gray, P.; Burton, J.; Prideaux, C.; Solomon, N.; Cackett, J.; Jones, M.; Bhathal, A.; Corrales, T.; Parolini, A.; et al. The Family Matters Report 2021 ; SNAICC: Melbourne, Australia, 2021. [ Google Scholar ]
• Productivity Commission. Closing the Gap Annual Data Compilation Report July 2023 ; Productivity Commission: Canberra, Australia, 2023. [ Google Scholar ]
• Australian Government Productivity Commission. Report on Government Services Part F, Section 16: Child Protection Services 2024 ; Australian Government, Productivity Commission: Melbourne, Australia, 2024. [ Google Scholar ]
• Commonwealth of Australia. Aboriginal and Torres Strait Islander First Action Plan 2023–2026 under Safe and Supported: The National Framework for Protecting Australia’s Children 2021–2031 ; Commonwealth of Australia: Canberra, Australia, 2022. [ Google Scholar ]
• Kickett, G.; Stubley, K. Moving to Child Safety and Family and Community Wellbeing ; Centre for Social Impact (University of Western Australia) Department of Communities Western Australia: Nedlands, Australia, 2020. [ Google Scholar ]
• Angell, B.; Muhunthan, J.; Eades, A.-M.; Cunningham, J.; Garvey, G.; Cass, A.; Howard, K.; Ratcliffe, J.; Eades, S.; Jan, S. The health-related quality of life of Indigenous populations: A global systematic review. Qual. Life Res. 2016 , 25 , 2161–2178. [ Google Scholar ] [ CrossRef ]
• Butler, T.L.; Anderson, K.; Garvey, G.; Cunningham, J.; Ratcliffe, J.; Tong, A.; Whop, L.J.; Cass, A.; Dickson, M.; Howard, K. Aboriginal and Torres Strait Islander people’s domains of wellbeing: A comprehensive literature review. Soc. Sci. Med. 2019 , 233 , 138–157. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Garvey, G.; Anderson, K.; Gall, A.; Butler, T.L.; Cunningham, J.; Whop, L.J.; Dickson, M.; Ratcliffe, J.; Cass, A.; Tong, A.; et al. What Matters 2 Adults (WM2Adults): Understanding the Foundations of Aboriginal and Torres Strait Islander Wellbeing. Int. J. Environ. Res. Public Health 2021 , 18 , 6193. [ Google Scholar ] [ CrossRef ]
• Garvey, G.; Anderson, K.; Gall, A.; Butler, T.L.; Whop, L.J.; Arley, B.; Cunningham, J.; Dickson, M.; Cass, A.; Ratcliffe, J.; et al. The Fabric of Aboriginal and Torres Strait Islander Wellbeing: A Conceptual Model. Int. J. Environ. Res. Public Health 2021 , 18 , 7745. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Anderson, K.; Elder-Robinson, E.; Gall, A.; Ngampromwongse, K.; Connolly, M.; Letendre, A.; Willing, E.; Akuhata-Huntington, Z.; Howard, K.; Dickson, M.; et al. Aspects of Wellbeing for Indigenous Youth in CANZUS Countries: A Systematic Review. Int. J. Environ. Res. Public Health 2022 , 19 , 13688. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Gall, A.; Anderson, K.; Howard, K.; Diaz, A.; King, A.; Willing, E.; Connolly, M.; Lindsay, D.; Garvey, G. Wellbeing of Indigenous Peoples in Canada, Aotearoa (New Zealand) and the United States: A Systematic Review. Int. J. Environ. Res. Public Health 2021 , 18 , 5832. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Dudgeon, P.; Bray, A.; Smallwood, G.; Walker, R.; Dalton, T. Wellbeing and Healing through Connection and Culture ; Lifeline Australia: Sydney, Australia, 2020. [ Google Scholar ] [ CrossRef ]
• United Nations (General Assembly). Convention on the Rights of the Child ; United Nations: New York, NY, USA, 1989. [ Google Scholar ]
• United Nations (General Assembly). Declaration on the Rights of Indigenous People ; United Nations: New York, NY, USA, 2007. [ Google Scholar ]
• Jackomos, A. Being strong in aboriginal identity and culture protects young people leaving care. Parity 2016 , 29 , 14–15. [ Google Scholar ]
• Burton, J.; Young, J.; Jayakody, N.; Ruggiero, E.; Thwaites, R. The Aboriginal and Torres Strait Islander Child Placement Principle: A Guide to Support Implementation ; SNAICC: Melbourne, Australia, 2019. [ Google Scholar ]
• Gatwiri, K.; McPherson, L.; Parmenter, N.; Cameron, N.; Rotumah, D. Indigenous Children and Young People in Residential Care: A Systematic Scoping Review. Trauma Violence Abus. 2019 , 22 , 829–842. [ Google Scholar ] [ CrossRef ]
• Richardson, N.; Bromfield, L. Cultural Considerations in Out-of-Home Care ; Australian Institute of Family Studies: Melbourne, Australia, 2007; Volume 8. [ Google Scholar ]
• Page, M.J.; McKenzie, J.E.; Bossuyt, P.M.; Boutron, I.; Hoffmann, T.C.; Mulrow, C.D.; Shamseer, L.; Tetzlaff, J.M.; Akl, E.A.; Brennan, S.E.; et al. The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ 2021 , 372 , n71. [ Google Scholar ] [ CrossRef ]
• Gharabaghi, K.; Anderson-Nathe, B. Strength-based research in a deficits-oriented context. Child Youth Serv. 2017 , 38 , 177–179. [ Google Scholar ] [ CrossRef ]
• Forsman, H.; Vinnerljung, B. Interventions aiming to improve school achievements of children in out-of-home care: A scoping review. Child. Youth Serv. Rev. 2012 , 34 , 1084–1091. [ Google Scholar ] [ CrossRef ]
• Kääriälä, A.; Hiilamo, H. Children in out-of-home care as young adults: A systematic review of outcomes in the Nordic countries. Child. Youth Serv. Rev. 2017 , 79 , 107–114. [ Google Scholar ] [ CrossRef ]
• Sinha, V.; Caldwell, J.; Paul, L.; Fumaneri, P.R. A Review of Literature on the Involvement of Children from Indigenous Communities in Anglo Child Welfare Systems: 1973–2018. Int. Indig. Policy J. 2021 , 12 , 1–43. [ Google Scholar ] [ CrossRef ]
• The EndNote Team. EndNote, EndNote 20 ; Clarivate: Philadelphia, PA, USA, 2013. [ Google Scholar ]
• Veritas Health Information. Covidence Systematic Review Software. Available online: https://www.covidence.org/ (accessed on 9 September 2022).
• Microsoft Corporation. Microsoft Excel. Available online: https://office.microsoft.com/excel (accessed on 9 September 2022).
• Lumivero. NVivo (Version 13) [Computer Software]. Available online: https://lumivero.com/products/nvivo/ (accessed on 24 October 2022).
• Braun, V.; Clarke, V. Reflecting on reflexive thematic analysis. Qual. Res. Sport. Exerc. Health 2019 , 11 , 589–597. [ Google Scholar ] [ CrossRef ]
• Byrne, D. A worked example of Braun and Clarke’s approach to reflexive thematic analysis. Qual. Quant. 2022 , 56 , 1391–1412. [ Google Scholar ] [ CrossRef ]
• Naeem, M.; Ozuem, W.; Howell, K.; Ranfagni, S. A Step-by-Step Process of Thematic Analysis to Develop a Conceptual Model in Qualitative Research. Int. J. Qual. Methods 2023 , 22 , 16094069231205789. [ Google Scholar ] [ CrossRef ]
• Commission for Children and Young People. ‘In Our Own Words’: Systemic Inquiry into the Lived Experience of Children and Young People in the Victorian Out-of-Home Care System ; Commission for Children and Young People: Melbourne, Australia, 2019. [ Google Scholar ]
• Turnbull-Roberts, V.; Salter, M.; Newton, B.J. Trauma then and now: Implications of adoption reform for first nations children. Child Fam. Soc. Work 2021 , 27 , 163–172. [ Google Scholar ] [ CrossRef ]
• Commissioner for Children and Young People WA. Speaking out about Raising Concerns in Care ; Commissioner for Children and Young People WA: Perth, Australia, 2016. [ Google Scholar ]
• AbSec—NSW Child Family and Community Peak Aboriginal Corporation. Hearing the Voices of Aboriginal People in Child Welfare ; Absec—NSW Child, Family and Community Peak Aboriginal Corporation: Sydney, Australia, 2020. [ Google Scholar ]
• Kiraly, M.; James, J.; Humphreys, C. “It”s a family responsibility’: Family and cultural connection for Aboriginal children in kinship care. Child. Aust. 2014 , 40 , 23–32. [ Google Scholar ] [ CrossRef ]
• Davis, M. Family Is Culture: Independent Review into Aboriginal Out-of-Home Care in NSW ; New South Wales Department of Family and Community Services: Sydney, Australia, 2019. [ Google Scholar ]
• McMahon, A.; Reck, L.; Walker, M. Defining well-being for Indigenous children in care. Child. Aust. 2007 , 32 , 15–20. [ Google Scholar ] [ CrossRef ]
• Advocate for Children Young People (NSW). The Voices of Children and Young People in Out-of-Home Care 2021 ; Advocate for Children and Young People (NSW): Sydney, Australia, 2022. [ Google Scholar ]
• Bamblett, M.; Frederico, M.; Harrison, J.; Jackson, A.; Lewis, P. ’Not One Size Fits All’ Understanding the Social & Emotional Wellbeing of Aboriginal Children ; La Trobe University: Bundoora, Australia, 2012. [ Google Scholar ]
• Commission for Children and Young People. ‘Always Was Always Will Be Koori Children’: Systemic Inquiry into Services Provided to Aboriginal Children and Young People in Out-of-Home Care in Victoria ; Commission for Children and Young People: Melbourne, Australia, 2016. [ Google Scholar ]
• Hermeston, W.; McDougall, J.; Burton, J.; Smith, F.; Sydenham, E. Achieving Stability for Aboriginal and Torres Strait Islander Children in Out-of-Home Care ; SNAICC: Melbourne, Australia, 2016. [ Google Scholar ]
• Higgins, D.J.; Bromfield, L.M.; Higgins, J.R.; Richardson, N. Protecting Indigenous children: Views of carers and young people on “out-of-home care”. Fam. Matters 2006 , 75 , 42–49. [ Google Scholar ]
• Kiraly, M.; Humphreys, C. ‘It Is the Story of All Us’. Learning from Aboriginal Communities about Supporting Family Connection ; State of Victoria: Melbourne, Australia, 2011. [ Google Scholar ]
• Lewis, N.; Weston, R.; Burton, J.; Young, J.; Jayakody, N.; Mastroianni, A.; Tan, W.W.; Parolini, A.; Shlonsky, A.; Tilbury, C. The Family Matters Report 2019 ; SNAICC: Melbourne, Australia, 2020. [ Google Scholar ]
• Libesman, T. Cultural Care for Aboriginal and Torres Strait Islander Children in Out of Home Care ; SNAICC: Melbourne, Australia, 2011. [ Google Scholar ]
• McDowall, J.J. Out-of-Home Care in Australia: Children and Young People’s Views after Five Years of National Standards ; CREATE Foundation: Sydney, Australia, 2018. [ Google Scholar ]
• Mendes, P.; Standfield, R.; Saunders, B.; McCurdy, S.; Walsh, J.; Turnbull, L. Indigenous youth transitioning from out-of-home care in Australia: A study of key challenges and effective practice responses. J. Child. Serv. 2022 , 17 , 16–32. [ Google Scholar ] [ CrossRef ]
• Mendes, P.; Standfield, R.; Saunders, B.; McCurdy, S.; Walsh, J.; Turnbull, L. Aboriginal and Torres Strait Islander (Indigenous) young people leaving out-of-home care in Australia: A national scoping study. Child. Youth Serv. Rev. 2021 , 121 , 105848. [ Google Scholar ] [ CrossRef ]
• Moore, T.; Bennett, B.; McArthur, M. They’ve Gotta Listen: Aboriginal and Torres Strait Islander Young People in Out of Home Care ; 192123914X; Australian Catholic University: Dickson, Australia, 2007. [ Google Scholar ]
• Moss, M. Broken circles to a different identity: An exploration of identity for children in out-of-home care in Queensland, Australia. Child. Fam. Soc. Work. 2009 , 14 , 311–321. [ Google Scholar ] [ CrossRef ]
• Spence, N. Kinship Care in Australia. Child Abus. Rev. 2004 , 13 , 263–276. [ Google Scholar ] [ CrossRef ]
• Higgins, D.; Bromfield, L.; Richardson, N.; National Child Protection Clearinghouse. Enhancing Out-of-Home Care for Aboriginal and Torres Strait Islander Young People ; Australian Institute of Family Studies: Melbourne, Australia, 2005. [ Google Scholar ]
• Liddle, C.; Gray, P.; Burton, J.; Taylor, M.; Young, G.; Kumar, R.; Turner, L.; Hutchins, L.; French, B.; Jones, L.A.; et al. The Family Matters Report 2023 ; SNAICC: Melbourne, Australia, 2023. [ Google Scholar ]
• Black, C.; Frederico, M.; Bamblett, M. ‘Healing through culture’: Aboriginal young people’s experiences of social and emotional wellbeing impacts of cultural strengthening programs. Child Abus. Negl. 2023 , 148 , 106206. [ Google Scholar ] [ CrossRef ]
• Creamer, S.; Blair, S.; Toombs, M.; Brolan, C.E. Indigenous services leading the way for Aboriginal and Torres Strait Islander children in out-of-home care. Soc. Work Educ. 2024 , 43 , 913–937. [ Google Scholar ] [ CrossRef ]
• Jau, J.; Mendes, P.; Chavulak, J.; Martin, R. The Housing Pathways and Experiences of Aboriginal and Torres Strait Islander Youth as They Transition from out of Home Care in Victoria and Western Australia. Int. J. Child Maltreatment 2022 , 5 , 319–336. [ Google Scholar ] [ CrossRef ]
• Krakouer, J. Journeys of culturally connecting: Aboriginal young people’s experiences of cultural connection in and beyond out-of-home care. Child Fam. Soc. Work 2023 , 28 , 822–832. [ Google Scholar ] [ CrossRef ]
• Liddle, C.; Gray, P.; Burton, J.; Kumar, R.; Tunny, T.; Prideaux, C.; Armstrong, E.; McCracken, A.; French, B.; Attenborough, I.; et al. The Family Matters Report 2022 ; SNAICC: Melbourne, Australia, 2022. [ Google Scholar ]
• Newton, B.J.; Katz, I.; Gray, P.; Frost, S.; Gelaw, Y.; Hu, N.; Lingam, R.; Stephensen, J. Restoration from out-of-home care for Aboriginal children: Evidence from the pathways of care longitudinal study and experiences of parents and children. Child Abus. Negl. 2023 , 149 , 106058. [ Google Scholar ] [ CrossRef ]
• Roche, S.; Dunk-West, P.; Otarra, C.; Taylor, R.; Moss, M. Exploring strategies for re-engaging children and young people in learning while living in out-of-home care in the Northern Territory, Australia. Child Fam. Soc. Work 2023 , 28 , 1022–1034. [ Google Scholar ] [ CrossRef ]
• Clarke, A.; Healy, K.; Lynch, D.; Featherstone, G. Stability in statutory kinship care: A grounded theory study of placement stability in Australia. Child Youth Serv. Rev. 2024 , 156 , 107289. [ Google Scholar ] [ CrossRef ]
• Wright, A.C.; Collings, S. Conceptual meanings of permanency: Photovoice with care-experienced youth. J. Youth Stud. 2023 , 1–18. [ Google Scholar ] [ CrossRef ]
• Raman, S.; Ruston, S.; Irwin, S.; Tran, P.; Hotton, P.; Thorne, S. Taking culture seriously: Can we improve the developmental health and well-being of Australian aboriginal children in out-of-home care? Child Care Health Dev. 2017 , 43 , 899–905. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Kor, K.; Park, J.; Dear, R.; Fabrianesi, B. Responding to children’s ambiguous loss in out-of-home care: The HEAR practice model. Child Fam. Soc. Work 2024 , 29 , 248–258. [ Google Scholar ] [ CrossRef ]
• Bamblett, M.; Lewis, P. Detoxifying the Child and Family Welfare System for Australian Indigenous Peoples: Self-determination, Rights and Culture as the Critical Tools. First Peoples Child Fam. Rev. 2007 , 3 , 43–56. [ Google Scholar ] [ CrossRef ]
• Hassall, A.; Olsen, A.; Bourke, S.; Pasalich, D.S. How do kinship and foster caregivers differ in their conceptualization of family connectedness? Child Abus. Negl. 2023 , 145 , 106391. [ Google Scholar ] [ CrossRef ]
• Chamberlain, C.; Gray, P.; Bennet, D.; Elliott, A.; Jackomos, M.; Krakouer, J.; Marriott, R.; O’Dea, B.; Andrews, J.; Andrews, S.; et al. Supporting Aboriginal and Torres Strait Islander Families to Stay Together from the Start (SAFeST Start): Urgent call to action to address crisis in infant removals. Aust. J. Soc. Issues 2022 , 57 , 252–273. [ Google Scholar ] [ CrossRef ]
• Fernandez, E.; Lee, J.-S.; McNamara, P. Understanding the Experience and Outcomes of Aboriginal and Torres Strait Islander Children in Out-of-Home Care during the Twentieth Century ; University of New South Wales: Kensington, Australia, 2018. [ Google Scholar ]
• Mendes, P. “The most significant child welfare reform in a generation”: An examination of the strategies used by the Home Stretch campaign. Aust. J. Soc. Issues 2024 , 59 , 328–343. [ Google Scholar ] [ CrossRef ]

Click here to enlarge figure

Share and Cite

Garvey, D.; Carter, K.; Anderson, K.; Gall, A.; Howard, K.; Venables, J.; Healy, K.; Bill, L.; Letendre, A.; Dickson, M.; et al. Understanding the Wellbeing Needs of First Nations Children in Out-of-Home Care in Australia: A Comprehensive Literature Review. Int. J. Environ. Res. Public Health 2024 , 21 , 1208. https://doi.org/10.3390/ijerph21091208

Garvey D, Carter K, Anderson K, Gall A, Howard K, Venables J, Healy K, Bill L, Letendre A, Dickson M, et al. Understanding the Wellbeing Needs of First Nations Children in Out-of-Home Care in Australia: A Comprehensive Literature Review. International Journal of Environmental Research and Public Health . 2024; 21(9):1208. https://doi.org/10.3390/ijerph21091208

Garvey, Darren, Ken Carter, Kate Anderson, Alana Gall, Kirsten Howard, Jemma Venables, Karen Healy, Lea Bill, Angeline Letendre, Michelle Dickson, and et al. 2024. "Understanding the Wellbeing Needs of First Nations Children in Out-of-Home Care in Australia: A Comprehensive Literature Review" International Journal of Environmental Research and Public Health 21, no. 9: 1208. https://doi.org/10.3390/ijerph21091208

Article Metrics

Article access statistics, supplementary material.

ZIP-Document (ZIP, 138 KiB)

Further Information

Mdpi initiatives, follow mdpi.

Subscribe to receive issue release notifications and newsletters from MDPI journals