Wynne Harlen and Ursula Schlapp, SCRE
In all areas of research, literature reviews are used to inform researchers of the background to research projects and to provide context and ideas for the design of new studies. Policy makers or prospective funders of research also use reviews of the literature to inform themselves of what existing research says in the fields of interest to them. This paper is mainly about reviews written for such purposes. It also looks at techniques developed for reviews which are regarded as research in their own right, which integrate findings of several related studies to answer research questions without collecting new data.
In this paper we consider the role of the literature review in the research process, some possible approaches to reviewing the literature, and the main steps and decisions to be taken and justified along the way. We conclude with some suggestions for writing up the findings.
THE ROLES OF LITERATURE REVIEWS
The various roles of reviews fall into two main categories:
The various roles of reviews fall into two main categories:
- guiding decisions about further research and providing a context for interpreting new findings
- informing policy and practice, based on existing research.
Guiding decisions about new research
A research review may have several important roles when carried out as a precursor to a new research study. These include:
A research review may have several important roles when carried out as a precursor to a new research study. These include:
- Firstly, and perhaps most obviously, a review of the literature in a particular field helps to clarify what is already known, what theoretical frameworks have been developed, and what has already been done, so that unintentional replications can be avoided and wheels not reinvented.
- A thorough literature review can also help to identify gaps in current knowledge. It can help to clarify what has not so far been systematically investigated, and where the focus of further enquiry should lie. At a more detailed level, information gained from a review of the literature can also help in the formulation of research questions, to clarify and limit the scope of a study.
- A review may also, from a study of the methodology used in previous research, give an indication of the research methods likely to be most effective in providing the required information, and perhaps also those which have been found to be less fruitful. Or the review may reveal that only a limited range of methodologies has been used so far.
- It is inevitable in research that, at times, related studies will throw up inconsistent results. By investigating the similarities and differences between groups of studies with varying results, a review of the literature may provide some insight into the reasons for inconsistencies. These might then be controlled in a new study, or at least be taken into account in interpreting the findings.
- Last, but by no means least, a review of the literature provides the context within which to interpret and report the findings of the new study when it is undertaken, allowing their relationship to previous knowledge to be explored and possible future directions for study to be suggested.
Informing policy and practice
Research reviews are increasingly used to provide a summary of the current state of knowledge in a given field. Journals, such as the Review of Educational Research, and the Review of Research in Education, exist to review existing work rather than to report original research. Policy makers use reviews of research to inform decisions about educational change and may commission literature reviews for this purpose. In this context reviews may serve the purposes of:
Research reviews are increasingly used to provide a summary of the current state of knowledge in a given field. Journals, such as the Review of Educational Research, and the Review of Research in Education, exist to review existing work rather than to report original research. Policy makers use reviews of research to inform decisions about educational change and may commission literature reviews for this purpose. In this context reviews may serve the purposes of:
- bringing together the range of available options for change; setting out what is known about the consequences of adopting each of the options
- combining different kinds of evidence that are needed to provide a firm basis for action - in particular surveys and case studies of the processes and interactions
- presenting findings of research on the effect of actions taken in different situations and of different actions in similar situations
- distinguishing the relevant studies from the less relevant, the good quality research from the poor quality
- providing guidance as to how dependable the available evidence is in relation to the area of proposed change.
In integrative reviews, where the findings from a number of related studies are combined in various ways in an attempt to answer a research question, the data come from the literature rather than from fieldwork.
APPROACHES TO REVIEWING THE LITERATURE
The greatest challenge in reviewing research is to reconcile the different findings from different studies. The difficulties of picking out the effect of certain factors in the context of many others which may interact with them to influence outcomes are so great that it is inevitable that contradictory findings will be reported. For example, in a review of the effect on pupil achievement of grouping by ability as compared with mixed ability teaching at the primary level, Slavin (1987) found almost equal numbers of studies showing that low achievers suffer as a result of ability grouping and those showing the exact opposite, with a larger number of studies showing no difference at all. Some reasons for this state of affairs were pointed out by Harlen and Malcolm in their review of setting and streaming (1997):
The greatest challenge in reviewing research is to reconcile the different findings from different studies. The difficulties of picking out the effect of certain factors in the context of many others which may interact with them to influence outcomes are so great that it is inevitable that contradictory findings will be reported. For example, in a review of the effect on pupil achievement of grouping by ability as compared with mixed ability teaching at the primary level, Slavin (1987) found almost equal numbers of studies showing that low achievers suffer as a result of ability grouping and those showing the exact opposite, with a larger number of studies showing no difference at all. Some reasons for this state of affairs were pointed out by Harlen and Malcolm in their review of setting and streaming (1997):
All studies reviewed, both quantitative and qualitative, suffer to some degree from the confounding of different effects, which is inherent in the major question they address, which is 'how does grouping by ability affect learning?' In practice pupils are not put into groups or classes by ability and then treated in exactly the same way as if they were in mixed ability groups or classes. Some would argue that this would be pointless, whilst others point to evidence that it is impossible, since the different social mix and the value position of the teacher would inevitably change the class or group interactions. In other words, it is never just the effect of grouping that is being compared but also the impact of different teaching methods, the quality of teaching, teacher expectations, pupil expectations and sometimes different materials.
Methods of reviewing try in various ways to deal with this problem. The most common are meta-analysis and best evidence synthesis. These have to some extent overtaken the traditional approach, which is first considered briefly.
The traditional approach
The traditional approach to literature review is to gather together all that can be found and to summarise, in narrative style, the studies and the key points arising from them. Often strengths or weaknesses of theory or methodology are identified but inevitably there is an element of subjectivity in the process of selecting and reporting findings. Literature reviews can be more helpful and serve their purpose better if certain procedures are used for minimising this subjectivity, or at least clarifying it to the reader of the review.
The traditional approach to literature review is to gather together all that can be found and to summarise, in narrative style, the studies and the key points arising from them. Often strengths or weaknesses of theory or methodology are identified but inevitably there is an element of subjectivity in the process of selecting and reporting findings. Literature reviews can be more helpful and serve their purpose better if certain procedures are used for minimising this subjectivity, or at least clarifying it to the reader of the review.
Meta-analysis
One of the most used approaches to integrating the findings of quantitative studies is that of meta-analysis, described by Glass (1976). Glass uses the term meta-analysis 'to refer to the statistical analysis of a large collection of analysis results from individual studies for the purpose of integrating the findings'. He claims that it offers 'a rigorous alternative to the casual, narrative discussions of research studies which typify our attempts to make sense of the rapidly expanding research literature'.
One of the most used approaches to integrating the findings of quantitative studies is that of meta-analysis, described by Glass (1976). Glass uses the term meta-analysis 'to refer to the statistical analysis of a large collection of analysis results from individual studies for the purpose of integrating the findings'. He claims that it offers 'a rigorous alternative to the casual, narrative discussions of research studies which typify our attempts to make sense of the rapidly expanding research literature'.
Before the use of meta-analyses, reviews attempting to integrate the findings of considerable numbers of conflicting studies tended to use the `vote taking' approach, the count being of statistically significant studies compared with those not finding significant differences. However, since studies with large samples are more likely to achieve statistically significant results, and no account is taken of the size of the effect in individual studies, there are considerable limitations to what can be inferred by using this approach. By contrast, meta-analyses are based on the synthesis of effect sizes - a measure of how much difference between trial and control groups a particular intervention has caused - which are calculated for each study in the review.
While meta-analysis has certainly found a place as a method for integrating the findings of quantitative studies, it is not without its critics. Several prominent figures in research have pointed to its 'mechanistic approach' by which it reduces the result of a study to a few numbers and in doing so sacrifices a great deal of information. Moreover it treats all studies as the same, regardless of methodological quality and their relevance to the issue in hand.
Best-evidence synthesis
An alternative approach, called 'best-evidence synthesis' by its initiator, Slavin, was designed so that studies reporting qualitative data as well as quantitative studies can be included. It combines 'the quantification and systematic literature search methods of quantitative syntheses with the attention to individual studies and methodological and substantive issues typical of the best narrative reviews' (Slavin, 1986).
An alternative approach, called 'best-evidence synthesis' by its initiator, Slavin, was designed so that studies reporting qualitative data as well as quantitative studies can be included. It combines 'the quantification and systematic literature search methods of quantitative syntheses with the attention to individual studies and methodological and substantive issues typical of the best narrative reviews' (Slavin, 1986).
A key aspect of the approach is regarded by Slavin as being that 'reviewers apply consistent, well justified, and clearly stated a priori inclusion criteria'. He contrasts this with the `haphazard study selection procedures characteristic of many narrative reviews' and the 'exhaustive inclusion principle' usually recommended for meta-analyses, where all that is required of studies is 'that [they] meet broad standards in terms of independent and dependent variables, avoiding any judgements of study quality'. 'good' studies give different results to `bad' studies, all studies relevant to the hypotheses being tested should be included. Slavin agrees that reviewer bias is likely if broad quality judgements are attempted. His solution to this difficulty is to admit that bias can never be eliminated and so what reviewers should do it to make their procedures explicit and open, and to say enough about the studies they review to give readers a clear idea of what the original evidence is.
This approach was used by Harlen and Malcolm (1997) who set out their criteria for inclusion of studies in drawing conclusions, and gave warnings when studies fell short of meeting these criteria but were still useful to report. The justification for including such studies is given by Slavin in addressing the problem that arises if no studies can be found which match the 'best evidence' criteria. In this case Slavin suggests that 'we might cautiously examine the less well designed studies to see if there is adequate unbiased information to come to any conclusions' (Slavin, 1986, p6).
Of these three approaches to conducting reviews, the most suitable approach to take in a particular case will depend largely on the intended purpose and readership of the review, as well as on the nature of the material being reviewed. For example, a review of the literature in a field where exclusively qualitative research has been carried out would have to be in traditional narrative form. However, it could take from the more quantitative approaches the idea of including a clear exposition of the sources searched, and details of any studies excluded from the review and the reasons for their exclusion.
CARRYING OUT A REVIEW OF THE LITERATURE
This section will make no attempt to go into detail of the mechanics of carrying out a literature review, as there are several books available which cover such matters. Instead we will consider briefly some of the key points at which decisions about strategy often have to be made.
This section will make no attempt to go into detail of the mechanics of carrying out a literature review, as there are several books available which cover such matters. Instead we will consider briefly some of the key points at which decisions about strategy often have to be made.
Defining the topic
Even if the area for review has been clearly defined at the start, as might be the case if someone else is commissioning the review, it will often be found that, once an initial search has been carried out, some refinement of the area of review will be needed. For instance, if only five relevant articles have emerged, it will be necessary to broaden the compass of investigation to include slightly different populations or approaches to a similar problem. Conversely, the initial search may have produced five hundred apparently relevant articles and so the need may be to reduce the scope of the review to make it more manageable and focused. If the review is being conducted for someone else it may be necessary to go back to them with initial findings to establish more precisely their interests, and possibly also the purpose for which they require the information, as this may help to shape decisions on which related information could make a valuable contribution and what could suitably be omitted.
Even if the area for review has been clearly defined at the start, as might be the case if someone else is commissioning the review, it will often be found that, once an initial search has been carried out, some refinement of the area of review will be needed. For instance, if only five relevant articles have emerged, it will be necessary to broaden the compass of investigation to include slightly different populations or approaches to a similar problem. Conversely, the initial search may have produced five hundred apparently relevant articles and so the need may be to reduce the scope of the review to make it more manageable and focused. If the review is being conducted for someone else it may be necessary to go back to them with initial findings to establish more precisely their interests, and possibly also the purpose for which they require the information, as this may help to shape decisions on which related information could make a valuable contribution and what could suitably be omitted.
Identifying sources of information
Because a review is concerned with 'the literature', it is easy to assume that the only interest is in written information. However, people can be very important sources in a number of ways. One of the most effective ways to get a foothold in the literature of an unfamiliar field is to ask for a list of key readings from an acknowledged expert. Such a person should be able to provide guidance to the 'classic' material, the latest findings, journals which publish particularly relevant material, and perhaps also to unpublished material and other useful contacts.
Because a review is concerned with 'the literature', it is easy to assume that the only interest is in written information. However, people can be very important sources in a number of ways. One of the most effective ways to get a foothold in the literature of an unfamiliar field is to ask for a list of key readings from an acknowledged expert. Such a person should be able to provide guidance to the 'classic' material, the latest findings, journals which publish particularly relevant material, and perhaps also to unpublished material and other useful contacts.
If an aim of the review is to inform practice, it may be helpful to take into account what the practitioners know or would like to know. As a result of practitioner interviews in the field of interest a review might look not only at reports of traditional research studies, but also at accounts of practice, action research, letters in the educational press and so on.
The task of searching the published literature, is made immeasurably easier these days through the existence of computer databases, computerised catalogues, specialised gateways on the internet, and so on. For those inexperienced in accessing such information, librarians and information specialists can advise and assist in making the most efficient use of such resources.
What counts as a sensible amount of material is clearly one of the decisions which is influenced by the purpose and audience of the review. If the purpose is simply providing background for an outline research proposal it will be necessary only to discuss a few 'classic' papers and one or two highly pertinent studies. On the other hand, for preparing a scholarly review of the development of research in certain field over the past five years, it will possibly be necessary to include several hundred papers.
Having identified the literature to be reviewed, ways have to be found of obtaining copies of it all. Books and journal articles should be fairly easy to obtain, through local libraries or loan services. One of the advantages of looking at published sources, especially articles in refereed journals, is that there has already been some kind of quality control applied. Nonetheless, each study will still need to be evaluated critically by the reviewer for any weaknesses or bias in the way it has been carried out or reported.
Whilst unpublished material such as theses, conference papers and so on may be harder to track down, it may be important to obtain at least some unpublished material, to counteract the possibility of publication bias interfering with the outcome of the review. Publication bias (see eg Light & Pillemer, 1984, for a discussion) is the tendency for only statistically significant results to be published.
Keeping records
An important adjunct to the whole process of identifying and locating the material for a review is the necessity for keeping full and accurate bibliographic details, including information on the location of materials to help in finding something again quickly if necessary. This may be particularly important if a review is carried out at an early stage of a project which spans several years. Something that seemed to be of only peripheral interest at that stage may take on far more significance as the project or field develops. The notes made at the early stage may have been adequate at the time, but it may later be necessary to return to the original and it can be frustrating not to be able to find something rapidly.
An important adjunct to the whole process of identifying and locating the material for a review is the necessity for keeping full and accurate bibliographic details, including information on the location of materials to help in finding something again quickly if necessary. This may be particularly important if a review is carried out at an early stage of a project which spans several years. Something that seemed to be of only peripheral interest at that stage may take on far more significance as the project or field develops. The notes made at the early stage may have been adequate at the time, but it may later be necessary to return to the original and it can be frustrating not to be able to find something rapidly.
Index cards are the classic format for storing bibliographic records. However, there is an increasing variety of computer-based record systems now available, ranging from simple databases which mimic the index card system in electronic form, to more powerful applications incorporating the ability to cross-reference, to attach fields for notes to the bibliographic details, and more.
Reading and note taking
When working in an entirely unfamiliar field it may be useful, if not essential, to do some reading before carrying out the main literature search, in order to clarify the kinds of terms to search for. In this case a previous review in the area (or closely related to it) may be very helpful, or a seminal book or paper recommended by an expert in the field.
When working in an entirely unfamiliar field it may be useful, if not essential, to do some reading before carrying out the main literature search, in order to clarify the kinds of terms to search for. In this case a previous review in the area (or closely related to it) may be very helpful, or a seminal book or paper recommended by an expert in the field.
Beyond this, the order and approach to reading and note taking adopted will be largely a matter of personal choice. It may be possible to focus simply on findings or methodology for some papers, especially those with a peripheral relation to the main subject of the review. However, much of the material will need to be read fully and reflectively, on the lookout for patterns, flaws of design or argument, and insights into possibly fruitful areas of further enquiry, ideas for design and methodology and so on. Trying to be systematic in the kind of details and information noted, and finding a way of classifying the information under various headings may make the process of converting notes into a coherent report easier. Again, keeping clearly in mind the purpose of the review and who is likely to be reading it, should help to ensure that all the relevant information is gathered as efficiently and effectively as possible.
Writing up the review
An important point to bear in mind in writing up a review, whatever approach is taken and whatever kind of information is included, is to keep it as clear, concise and readable as possible. In reviews covering a large amount of quantitative information, clearly presented tables of the data will need to be incorporated, whereas reviews of qualitative material alone will consist only of text. It is, however, possible to suggest certain key sections which should be included in some form in any good literature review. These closely relate to the sections often found in a report of empirical work: an introduction, a `methods' section indicating how and why the studies included in the review were selected, the main body of the review, and a conclusion.
An important point to bear in mind in writing up a review, whatever approach is taken and whatever kind of information is included, is to keep it as clear, concise and readable as possible. In reviews covering a large amount of quantitative information, clearly presented tables of the data will need to be incorporated, whereas reviews of qualitative material alone will consist only of text. It is, however, possible to suggest certain key sections which should be included in some form in any good literature review. These closely relate to the sections often found in a report of empirical work: an introduction, a `methods' section indicating how and why the studies included in the review were selected, the main body of the review, and a conclusion.
The introduction is necessary to give a clear indication of the area to be covered by the review, referring to earlier reviews and to established relevant theory, terms and concepts in the field. The methods section is the one which in frequently glossed over or missing from traditional narrative reviews, perhaps because the procedures for selecting studies were rather unsystematic. If clear criteria for selection of studies are employed, whatever these may be, and a systematic search procedure is adhered to, there should be no difficulty in outlining the details of these fully enough for readers to be clear about what has been included, what has not been included, and why.
The main body of the review may include tables of study characteristics, statistics where these have formed part of the review, and a variety of other kinds of information about and discussion of the studies reviewed. This section will often consist of a number of subsections addressing different aspects or groups of studies. Slavin (1986) in outlining the requirements for a good 'best-evidence synthesis' review, proposes that the review 'should provide the reader with enough information about the primary research on which the review is based to reach independent conclusions.' This seems a reasonable ideal to aim for in any kind of review. However, many quantitative syntheses concentrate almost exclusively on the statistical procedures and results, and give woefully inadequate descriptions of the studies from which the results derive.
The section on conclusions is an important one, since a research review should be more than a summary which selects from and repeats what has been found before but should bring the findings from different studies together. Indeed the value of a research review lies in just this process, since no single research study is ever conclusive. Whereas even tentative findings, if repeated consistently across several studies, acquire a greater significance. Equally important, if studies produce conflicting findings, it is essential for the review to reveal the inconclusive state of the research evidence overall. Thus, for example, in the review of setting and streaming (Harlen & Malcolm, 1997) it was possible to find a number of studies providing evidence of a positive effect of setting and streaming on achievements but just as many producing evidence contradicting this. Thus the only conclusion that could be drawn was that there was no consistent and reliable evidence of positive effects of setting and streaming. At the same time, the review of the evidence from various studies enabled more helpful statements to be made about the need for teaching methods appropriate to various groupings and the importance of teachers' attitudes to heterogeneous and homogeneous groupings.
There are often difficult judgments to be made in drawing conclusions in such situations and it may necessary to explain how conflicting findings have been weighed against each other. The matter of quality of the research is a factor to be taken into account and this is where the criteria for selection of studies will be relevant to the discussion. Not all studies reviewed will meet the criteria for selection equally well and it is appropriate to give more weight to those studies that very adequately meet the criteria of quality than those which do so marginally and have some short-comings.
The guiding principle for providing a valuable literature review can be summarised as making all procedures and decisions explicit in the review, so that readers can be clear about the evidence-base on which the review is founded.
References and further reading
Cooper, H.M. (1984) The Integrative Research Review: A Systematic Approach. Sage Publications.
Cooper, H.M. (1984) The Integrative Research Review: A Systematic Approach. Sage Publications.
Glass, G.V. (1976) Primary, secondary and meta-analysis of research. Educational Researcher, 5(10), pp 3-8.
Harlen, W. and Malcolm, H. (1997) Setting and Streaming: A Research Review. Edinburgh: Scottish Council for Research in Education.
Light, R.J. and Pillemer, D.B. (1984) Summing Up: The Science of Review in Research. London: Harvard University Press.
Slavin, R.E. (1986) Best-evidence synthesis: an alternative to meta-analytic and traditional reviews, Educational Researcher, 15 (9), pp 5-11.
Slavin, R.E. (1987) Ability grouping and student achievement in elementary schools: a best evidence syntheses. Review of Education Research, 57 (3), 293-336.