Os IGNORANTES, que acham saber tudo, privam -se de um dos maiores prazeres da vida: APRENDER.
Início Ensino a Distância Online Tutoring e-Book Chapter 5 - Evaluation

Online Tutoring e-Book Chapter 5 - Evaluation

 

*Jen Harvey, Carol Higgison and Cathy Gunn
 
1. Introduction
Interest in educational evaluation has increased over the last few years. Nationally and internationally government policies have promoted the embedding of learning technologies in education and considerable amounts of money have been invested with patchy results (Dearing, 1997). Investing in learning technology is expensive and users are looking for ways to develop a better understanding of factors which influence and affect this embedding process (for example the UK Technology in Teaching and Learning Programme phase 3 http://www.ncteam.ac.uk/projects/tltp/index.htm and the European Union Socrates programme http://europa.eu.int/comm/education/socrates.html ). Evaluation is seen as the key to this understanding (Oliver and Harvey, 2000).
However, despite this increasing interest in evaluation, there is often little variation in the methodologies we use in the evaluation process, particularly when evaluating new learning technologies. Frequently, an evaluation comprises an end of module questionnaire devised with little thought as to content or the purpose of collecting this information. This limited approach can result from time or resource constraints, perhaps a lack of experience or just insufficient design of an appropriate evaluation strategy in advance. A last minute feel good questionnaire can be very comforting in providing us with positive feedback but does not help us to make improvements or find out how we might help our students in the learning process.
This chapter draws on the experiences of the e-workshop participants and on published literature to present a clearer understanding of the role and methods of evaluation in online learning and teaching.
References given without dates are references to OTiS e-workshop case study contributions. Details are given in Appendix A.
 
 
 
 
 

2. What is Evaluation?

Evaluation is the process through which we examine the learning opportunities and experiences we offer our students and make judgements about their effectiveness and value. In the context of learning technology "these judgements usually concern the educational value of innovation, or the pragmatics of introducing novel teaching techniques and resources. Less frequent, but still important, are judgements about the costs of such innovations" (Oliver, 2000). Oliver (ibid) defines learning technology as "the use of technology to support innovations in learning and teaching" which includes online learning and tutoring.
Evaluation can provide a range of different and useful information to assist in future course design, planning and implementation. This can include:
·  measuring achievement of a course against specific learning outcomes,
·  identifying any problems which occur in the development or running of a course with the aim of providing solutions,
·  carrying out a needs analysis to justify a decision to invest in additional resources,
·  selecting new resources and improving the way in which they are used within a course,
·  exploring the materials or course to identify issues relating to the context in which they are used,
·  establishing whether materials have met specific quality assurance criteria.
Evaluation can be carried out at different times, for different audiences and for different purposes, including:
·  formative evaluation,
·  summative evaluation,
·  interpretive or illuminative evaluation (Partlett and Hamilton, 1972),
·  integrative evaluation (Draper et al, 1996).

2.1 Formative and summative evaluation

Formative and summative evaluations are differentiated both by their timing and purpose:
"When a cook tastes the soup, it is formative evaluation; when the dinner guest tastes the soup, it is summative evaluation." (Harvey, 1998)
Laurillard (1993) offers working definitions for formative and summative evaluation:
·  formative evaluation describes the evaluation of course materials or learning environments with the objective of providing information for improvement during the design and implementation phases,
·  summative evaluation describes the evaluation of course materials or learning environments with the objective of providing information on the outcomes of implementation and use by students.
The OTiS case studies provide examples of both formative and summative evaluation:
·  formative evaluation
"An independent evaluation team undertook constant formative evaluation, which was fed back to tutors, facilitators, the course co-ordinator and the course team. This allowed fine-tuning of the course delivery and prompt identification of key issues and problems." (Higgison)
"One of the greatest enablers of success was a willingness to make changes during the course process, based on the feedback from the participants." (McKenzie)
"I regularly request input from students on how the course is working for them and use their feedback to help shape the course as the semester progresses." (Hird)


·  summative evaluation
"Student pass rates remained stable and high, although the first running of the module revealed dissatisfaction with modes of delivery, which were due mainly to difficulties with support systems and frequent hardware computer crashes and delivery problems… Some students in the first running of the module expressed discomfort with the module's dependence on virtual tutorials, and suggested the possibility of building in the occasional real tutorial to help them feel more at ease with disparities in asynchronous and synchronous learning methods. The second running of the module addressed these issues, and surveys undertaken after module delivery suggested general satisfaction with the combination offered." (Finkelstein)
"A Final Report was written 'Discover VET in Schools, LearnScope Project 1999' … which provides an in-depth evaluation of the process from the participants' and tutors' perspectives." (Murray)

The distinction between formative and summative studies is useful in some cases. However, it may be more appropriate to treat evaluation as a continuous process (Gunn, 1999) with a shifting focus appropriate to the phase of design, development or implementation reached as, for example, described by Salmon:
"We … put in a 'point of learning' conference at each of the five stages, where trainees are deliberately asked to reflect on their experience of the programme to date. In addition, we monitored the work of the trainees after they commenced facilitation online with their students. This has enabled the training programme to be updated and improved week on week and over four years." (Salmon)

2.2 Illuminative evaluation

'Illuminative evaluation' is an observational approach to evaluation that is inspired by ethnographic research and methods (Parlett and Hamilton, 1972). Its aim is to discover the factors and issues that are important to the participants in a particular situation rather than how well an innovation performs against standard measures of evaluation. It attempts to explain new learning practice in terms of theories and beliefs about the learning of knowledge, skills and attitudes (ELICT, 2000).
An illuminative approach to evaluation allows us to report on factors important in a particular context and to identify unexpected factors or outcomes. Several OTiS case studies reported the use of methods which support the process of illuminative evaluation by encouraging participants to reflect on their experiences through journals (eg White and Moussou) and reflective logs (eg Cowan). Nurmela describes how participants were encouraged to self-evaluate and develop personal study plans to focus on their concerns and perceptions rather than those of the tutor:
"We use personal study plans and self-evaluations in many of our courses ... I think it is most useful to focus the evaluation more to something the student is doing themselves and not just the tutor/teacher outside the student. . . . self-evaluation is intended to focus on the content itself, tutoring and learning online (practices and principles). Questions participants considered included: how has their attitude towards learning and tutoring online been changing during the course and what new practices they have discovered." (Nurmela)
Ewing, in e-learning is not always easy learning, describes how factors and issues identified by students as important are made available to the next year's cohort:
"Student comments on how to overcome this unhelpful perception have been video-recorded for use with the following year's cohort." (Ewing)

2.3 Integrative evaluation

Integrative evaluation aims to improve teaching and learning by integrating innovative materials and techniques into the overall situation more effectively, "'to discover how an education intervention performs' by observing and measuring the teaching and learning process" (Draper et al, 1996). This approach to evaluation is discussed in Section 3.2.1.

2.4 Evaluation and assessment

It is important at this point to stress the distinction between assessment and evaluation. Although assessment data is one of the inputs to evaluation, the purposes are rather different, as noted by Phillips et al (2000):
"We are using evaluation in terms of looking at a broad range of evidence in order to gauge the effectiveness of a [computer-facilitated learning] project. Assessment is the process whereby teachers set specific tasks related to the learning outcomes which students undertake to do. Students all undertake formal and informal assessment tasks in the subjects they are studying and so we always have assessment data to use in evaluation. Their success in these tasks provides evidence of how effective their learning has been. But assessment results make up only one set of measures and these need to be considered alongside other pieces of evidence. While all evaluation plans should contain assessment data, that is just one aspect of evaluation."
Indeed, planning for the inclusion of course assessment as a method of evaluation data collection keeps quality issues clearly in focus and puts fewer demands on students than other methods that yield them no personal gain.
Assessment can be the focus of the evaluation, for example it is useful to verify that the assessment process is consistent across the course:
"Although double marking of assignments was not used, on several occasions the tutors inadvertently marked the same assignment (discovered before the sending of the mark to the student) and it was noted they were within a few points of each other and with consistent comments, each time." (Janes)
And assessment can be used as a measure of the educational impact of a new innovation by comparing the results of assessments across years or comparing results with a similar group:
"Course marks for the module were also compared with course marks for other modules in the Masters programme." (Clarke)
 

3. A Theoretical Framework for Evaluation

This section provides an overview of the evolution of the theoretical perspectives (Section 3.1) that have influenced the development of contemporary evaluation methodologies (Section 3.2) and in particular those methodologies used to evaluate the deployment of learning technologies in the educational process. The section concludes with some recommendations and guidance on key questions and an overview of evaluation methods (Section 3.3).

3.1 Evolution of evaluation methodologies

Educational research literature records a fundamental shift over the last fifty years in evaluation methodologies from quantitative experimental approaches to a predominance of qualitative methods. Many current evaluations adopt a hybrid approach that combines qualitative and quantitative methods that support the shift towards evaluation of authentic learning experiences in their natural context. This has resulted in a shift from evaluation as an external process to a more collaborative process between evaluator and practitioner aimed at building a mutual understanding of what is occurring.
3.1.1 Quantitative and qualitative approaches
Quantitative evaluation focuses on measurement, is externally directed and value-free. Examples of preferred indicators include student pass rates, student retention and student progression, for example:
"Eighty percent of enrolments still active online at the end of the course; sixty percent of enrolments submit a portfolio and achieve accreditation. For the period 1998-1999, we have consistently exceeded the targets." (Pickering and Duggleby)
"Initial reaction to this method of delivery was negative due to the complex nature of the WebCT version used at this time. This complex method of delivery resulted in a thirty per cent drop out rate…" (McFarlane)
"The completion rate from this course is ninety percent and eighty-six percent of students surveyed stated that they would recommend our course to friends. A significant number of students wish to continue their studies with us by distance learning to MSc level." (Kennedy and Duffy)
Qualitative evaluation focuses on the educational process, is directed by the user and takes account of values. Preferred indicators include student comments and evidence for reasons for change, for example:
"We used an extended form of Stop/Start/Continue enquiry, asking our students to tell us what they wanted us to stop doing - and why? What they would like us to start doing, which we were not then doing - and why they thought that could be useful to them. And what they wanted us to continue doing, with an explanation of the ways in which that was helpful to them. We stressed that the reasons were the important part of this feedback, and that we might come back for clarification of anything we did not understand. We undertook to modify our commenting accordingly." (Cowan)
"The evaluation revealed that although the emphasis on collaborative learning did not suit all students, they appreciated the integration of activities with assessment, because it guaranteed the involvement of all students." (Macdonald)
Essentially qualitative and quantitative approaches to evaluation complement each other with qualitative techniques being well suited to exploring, identifying and explaining and quantitative techniques to demonstrating, measuring and generalising (Oliver, 1997).
3.1.2 Early experimental approaches to evaluation
The early days of computer supported learning evaluation in the 1960s and 70s followed the traditions of educational research that were prevalent at that point in time. Until the 1960s, the majority of evaluation studies were conducted in contrived and ostensibly controlled environments. As such, evaluations were mainly concerned with quantitative experimental studies designed to measure effects and to make comparisons between different educational approaches and materials. This approach was based on the assumption that learning is an objective phenomenon. It required large study populations so that results could be generalised, isolation of the effects of one resource or intervention and balancing of all other variable factors so their effects could be discounted. The basis of this approach is in the physical sciences and early studies highlighted serious limitations as a means to evaluate human behaviour in learning situations (Laurillard, 1978). Studies produced adequate evidence of the outcomes that occurred but gave little insight into the causal factors or the process through which they were achieved so were of little use in planning and design. The majority of evaluations reported in the OTiS case studies look beyond the quantitative data for reasons to explain the figures:
"Progression rates through the programme are higher than the client has seen in previous attempts at this sort of training. We believe that the drop out rate in terms of distance learning is relatively low due to the high level of interpersonal support that was provided….
"Student focus groups and evaluations suggest a changing degree of acceptance of the technology, and participation rates give an indication that the online collaborative process improves gradually over the course of the programme." (MacKenzie)
3.1.3 Evolution of contextual approaches to evaluation
The beginnings of a contemporary preference for authentic settings with real target users and an evaluative approach coincides with the early days of computer assisted learning - before the advent of the Internet. It was gradually accepted that a more holistic approach was needed to support examination of the range of influential factors including individual and situational ones.
"Additional issues relating more directly to the online medium include the way in which web resources are integrated with discussions, their personal experience of the discussions, and the tutor's management of these. Support issues also become much more significant in this situation, and the students are asked to evaluate technical, pedagogical and information (library) support structures, all of which are provided at a distance." (Creanor-D, 9 May 2000)
There were also many practical and pragmatic reasons for this shift. Setting up large experimental studies was resource intensive and although they could identify what was happening they could not really explain why or how.
"We use statistical analysis of results, comparing the results gained by students undertaking exactly the same modules, and undertaking exactly the same assignments, by distance learning and by classroom based learning." (Kennedy and Duffy)
Although not set in a 'laboratory' using this performance indicator in isolation, it would be difficult for Kennedy and Duffy to attribute the cause of any differences in performance between the two groups to online learning. In their case study they discuss other potential causes, for example differences in student population, particularly nationality, the local context and motivational factors.
The increasing use of computers, initially to deliver programmed instruction and later multimedia, computer mediated communication and online interaction and the corresponding increase in the investment in innovative teaching methods, demands a more fine-grained approach to evaluation. The need to define and achieve effectiveness is increasing as academic institutions strive to make informed choices about course presentation and to maintain their position in a competitive global market. Also, where face-to-face contact between teachers and learners may be minimal or non-existent (as in Kennedy and Duffy) then conducting experimental studies in these situations is both impractical and impossible.
3.1.4 Evaluation and online learning
Online learning has essentially grown out of recent educational theory and practice with the addition of new tools and methods (see Chapter 1: Learning Online (Cornelius, 2001)). It would be reasonable to expect that evaluation has evolved in a similar way. In some respects this is true, however, the most common form of evaluation of courses and teaching relies rather heavily on one source of data - the 'student questionnaire'. Conducted once at the end of each module or course, it has been suggested that this method is to evaluation what exams are to assessment of learning, namely convenient, simple to administer and easily reduced to comparable outcomes but largely inadequate as a means of measuring effectiveness and identifying influencing factors for success. Recent research suggests that even the most useful of these instruments are not geared to evaluation of online teaching and learning. This concern was raised during the e-workshop.
"One concern that I have is the fit between the departmental course evaluation form students complete in all their courses and the online course… there may be a need to develop an evaluation form specific to online courses. One recommendation that I would have for anyone involved in new online course design is to make provisions for evaluation above and beyond that which is provided by the department or institution. What I have learned is that there are critical questions specific to online learning that need to be asked ... Student feedback relating to the amount of time spent on the course, the level of technical support needed, and the effectiveness of each online assignment is needed to improve the course … Another issue that is essential to explore is the extent to which online learning options affect the student's choice of institution." (Hird)
There are however, many other contemporary approaches to evaluation that are more suitable for evaluating online teaching and learning. New environments and new methods of teaching can be well served by re-purposing established evaluation practice (Section 3.2 and 3.3).
3.1.5 Further reading on the theory
Evaluation of learning and teaching which incorporates the use of learning technologies is complex and our approach will depend on our underlying assumptions about learning and teaching. If you wish to explore the theoretical underpinnings of evaluation methodologies the following readings provide a useful starting point.
For a historical overview of the parallel developments in technology and evaluation methods, see Isolation or Integration by Cathy Gunn in the LTDI Evaluation Cookbook Online (1998), Jen Harvey (ed) at http://www.icbl.hw.ac.uk/ltdi/cookbook/info_isolation_or_integration/index.html#endhead
Reeves (1997) has mapped the dominant paradigms, which are used in evaluation studies, and the models researchers use within these paradigms. These are concisely summarised and presented by Phillips et al (2000) in Table 1.3 on page 1.5 in their Handbook for Learning-centred Evaluation of Computer-facilitated Learning Projects in Higher Education available online at http://cleo.murdoch.edu.au/projects/cutsd99
Martin Oliver (1997) provides an in-depth review of the main evaluation methodologies in A framework for evaluation the use of Learning Technology available online at http://www.unl.ac.uk/elt/elt1.htm
Martin Oliver (2000) also summarises the important debates and complex issues surrounding the evaluation of learning technologies in An Introduction to the Evaluation of Learning Technology, Educational Technology & Society 3(4) online at http://ifets.massey.ac.nz/periodical/vol_4_2000/intro.html.

3.2 Contemporary approaches to evaluation

The complexity of situations addressed by evaluation studies demands a range of approaches to suit different purposes and it is a case of selecting the most appropriate for the circumstances.
Common features of evaluations
Reviews of contemporary approaches to evaluation and appropriate areas of application are offered by Oliver (1997) and Oliver and Harvey (2000). Features common to most contemporary approaches include:
  • the need to evaluate in authentic contexts,
  • use of a range of data sources,
  • the importance of integration,
  • study of complete learning environments,
  • focus on individual and situational aspects.
These features characterise the evaluation strategy reported by Morrison in her case study T171: the pilot year experience, which is based on the OU evaluation methodology - CIAO! -described in Section 3.2.6:
"At the end of each of the three modules, the student completed a web page questionnaire which collected their comments about the materials, the tutor group activities and the module assignment. Summarised results of these questionnaires were notified to the tutors in the national tutor conference, together with statistical information about drop-out rates.
"In addition, approximately half way through the course, the tutor emailed each student a message with two questionnaires to be completed and returned either by email or by post. One concentrated on the student's progress and concerns and the second on the tutor-student relationship. The most common comment was on the benefit to the student of being able to work wherever and whenever was most suitable to them.
"Tutors also completed a web page questionnaire at the end of each module and discussed suggested improvements to the course in the national tutor conferences." (Morrison)
Selection criteria for choosing an evaluation methodology
Oliver and Conole (1998) suggest three qualities that can be used to select an appropriate methodology (Figure 3.1) for the situation being evaluated:
  • authenticity: how closely the methodology can capture the context of an existing course,
  • exploration: the extent to which the methodology supports evaluating an open problem or a well defined hypothesis,
  • scale: the number of participants which can easily be incorporated into the study.
 
 
Authenticity
 
Exploration
 
Scale
Integrative evaluation (s3.2.1)
Low
High
Low
High
Low
High
Experimental (not described)
x
 
x
 
 
x
Illuminative evaluation (s2.2)
 
x
 
x
 
x
Integrative evaluation (s3.2.1)
 
x
 
x
 
x
Situated evaluation (SECAL) (s3.2.2)
 
x
 
x
x
 
Action inquiry (s3.2.3)
 
x
 
x
x
 
Case study (s3.2.4)
 
x
 
x
x
x
Flashlight (s3.2.5)
 
x
x
 
 
x
UK Open University CIAO! (s3.2.6)
 
x
x
 
 
x
Cost benefit (s3.2.7)
x
 
x
 
x
 
Figure 3.1 - Selection criteria for choosing an evaluation methodology*
* Please note that the ELT toolkit (s3.2.8), the LTDI Evaluation Cookbook (s3.2.9) and the Evaluation Toolkit for Practitioners (s3.2.10) are meta level toolkits which guide practitioners in selecting the methodology most appropriate to their evaluation study.
The theoretical models and frameworks that support this contemporary approach result from parallel developments in different parts of the world and are known by different names. The ones cited here are featured because of perceived generality, familiarity and subjective choice. Others defined by different names may fit the bill equally well.
3.2.1 Integrative evaluation
The TILT (Teaching with Independent Learning Technologies) framework (Draper et al, 1994, 1996, 1997) is based on the illuminative approach to evaluation (Parlett and Dearden, 1977 and s2.2) and aims to improve learning by integrating educational technology as effectively as possible into the learning environment. Integrative evaluation adopts Laurillard's (1993) conversational framework as a model for student-teacher interactions, allowing the framework to focus on educational interactions.
This framework aims to evaluate each course as a whole, rather than simply the resource(s) being used, and improve learning by integrating learning materials as effectively as possible. The main aim is "to help teachers make better use of CAL by adjusting how it is used" (Draper et al, 1996) and "by providing better information than is ordinarily available about what is going on and its effects" (ibid). The evaluation is viewed as an active collaboration involving the evaluators, the teachers and the students. The teachers' co-operation is seen as essential and their stated learning outcomes are central to framing the evaluation questions. Integrative evaluation is an empirical approach based on observing learning in an authentic context which draws on a range of qualitative and quantitative methods (Section 3.3.3) including questionnaires (pre and post session, computer and task experience, learning resource), observations, confidence logs, knowledge quizzes, focus groups and interviews.
Integrative evaluation is distinct (Draper et al, 1994, 1996, 1997) because:
  • it focuses on the student and observes what they actually do and feel,
  • it attempts to measure learning for each learning objective,
  • it makes substantial and systematic use of open-ended observation to identify unforeseen factors.
This is exemplified by the approach described in White and Moussou:
"I think we do a good job of process evaluation, short-term experiences of the learner etc. The student response, especially in the journals, tipped us off to areas of student concern, what they were 'getting' and what was not clear, allowing us to adjust the pacing, content or style of presentation. The feedback is such a help from a tutor perspective. I do not think we would get such feedback, however, without the journals. They seem to be key - a safe and designated place for such feedback."
 
 
3.2.2 Situated evaluation of computer assisted learning (SECAL)
The Situated Evaluation of Computer Assisted Learning (SECAL) (Gunn 1996, 1997) is a framework for designing case based evaluation activities. The framework recognizes the impact of situational factors on learning and attempts to capture rather than balance and disregard the complex range of variables at play in a given situation. The range of variables includes factors intrinsic to learning situations such as instructional design and strategies (Merrill, 2000), content coverage, presentation, quality, motivational aspects (Keller, 1987) and learning support. Other factors less directly causal, though with equal potential to influence outcomes, include institutional context and support, classroom culture and overall integration of activities and resources within courses (Draper et al, 1996).
The SECAL framework adopts methods and techniques as appropriate and is ideally used with small sample sizes. It aims to provide rapid and immediately applicable results. It is these two features that distinguish it from illuminative evaluation (Oliver and Harvey, 2000).
3.2.3 Action inquiry
Action research brings together stakeholders from different disciplines for the purpose of conducting research that will inform strategies for ongoing development. The following definition of 'action research' is offered by Zuber-Skerritt (1990):
  • Critical (and self critical) collaborative enquiry by
  • Reflective practitioners being
  • Accountable and making the results of their enquiry public,
  • Self evaluating their practice and engaged in
  • Participative problem-solving and continuing professional development.
Action research (Kember and Kelly, 1993) reflects a dialectical relationship between educational theory and practice where action and practical experience provide the basis for research. This research informs practice that leads to further action. The model is well suited to evaluating innovations because it can advance knowledge and understanding on the basis of practical, collaborative experience and at the same time, contributes to development of grounded theory (Glaser and Strauss, 1967).
The action research process consists of repeated cycles of planning, action, observation and reflection. The final stage, reflection on outcome and process, is where new plans evolve and a new cycle is initiated. The major strengths of action research in this context are:
  • its practical and collaborative nature,
  • the critical and self-critical approach of those involved,
  • the interpretive treatment of results,
  • the iterative nature of the model,
  • the systematic approach to monitoring the effects of change in learning environments.
Action inquiry in practice
The Australian Committee for University Teaching and Staff Development (CUTSD) has adopted this learner centred action research model for evaluating learning technology innovations. It has produced a Handbook for Learning-centred Evaluation of Computer-facilitated Learning Projects in Higher Education (Phillips et al, 2000), which is available at http://cleo.murdoch.edu.au/projects/cutsd99.
A consortium of Australian Universities and the Australasian Society for Computers in Learning in Tertiary Education (ASCILITE) developed the handbook. They have taken a pragmatic approach to the evaluation process and ensured that the handbook would be used effectively within their institutions. The handbook guides the user through a series of questions about the evaluation then focuses on four key areas:
  • Analysis and design: curriculum analysis, teaching-for-learning analysis, specification of innovation.
  • Development: formative monitoring of learning environment, formative monitoring of learning process.
  • Implementation: summative evaluation of learning process, summative evaluation of learning outcome, summative evaluation of innovation appropriateness.
  • Institutionalisation: impact evaluation, maintenance evaluation.
Cowan in his case study Personal Development Planning documents some of the benefits of using an action research approach to evaluation:
"What made this a successful piece of staff and curriculum development? I suggest that:
  • "we had experienced action researchers in our number,
  • "we were willing to learn from and with the students, and let that be seen by them,
  • "the students knew that the immediate outcome of the action research was fed into improvement of the tutorial support that they received,
  • "we made attempts to generalise what we found, to provide some (admittedly particular) theoretical underpinning." (Cowan)
3.2.4 Case study
The case study is "an empirical investigation of a particular contemporary phenomenon within its real life context using multiple sources of evidence" (Robson, 1993 p52).
Yin (1991) offers a technical description of a case study as an investigation of a contemporary phenomenon within its real life context when the boundaries between the phenomenon and context are not clearly evident and in which multiple sources of evidence are used. In the current context:
  • online teaching is a contemporary phenomenon which is reaching across the spectrum of learning environments,
  • it is already clear that some methods that work for some students in some situations do not work for or in others. Narrow focus is the only way to identify what factors drive this success and failure, (Laurillard 1978),
  • triangulation of data from multiple sources provides support for assumptions and conclusions where, in most cases, statistical proof is not an available option.
Within the methodological framework, a variety of data collection methods may be used to support:
  • the qualitative, descriptive approach,
  • its inductive nature,
  • recognition that all influential variables may not be anticipated at the start of an evaluation.
Case study evaluations should also be related to current literature in order to conceptualise issues, design appropriate studies and interpret results. This also helps to identify what is common and may contribute to grounded theory, and what is case specific and non-generalisable.
At one end of the scale, case study research allows attention to minute levels of detail in specific contexts. At the other, it offers methods for production of less fine-grained and more standardised data. The reality it must be designed to cope with is that the outcomes of evaluation, like the range of influences on learning, are not always intrinsic or educational ones. While a single case study is treated as a project in itself, publication of methods and findings allows the experience to be shared, theory constructed and attempts at generalisation made.
Case studies in practice The OTiS e-Workshop is based on a case study approach. The OTiS Case Studies (http://otis.scotcit.ac.uk/casestudy) record experiences of online tutoring from a range of practitioner perspectives. Authors were provided with a template (http://otis.scotcit.ac.uk/casestudy/example.html) which enabled us to identify common themes and factors but also provided the flexibility to record individual factors and circumstances.
The online book (http://otis.scotcit.ac.uk/onlinebook/) attempts to link these practitioners' experiences to current literature, to conceptualise issues and to identify what is common and may contribute to the 'common body' of knowledge and effective practice in online learning and teaching.
3.2.5 The Flashlight Project
The Flashlight project, supported by the American Association for Higher Education (AAHE), has developed a framework that provides a simple structure for the evaluation of learning technology innovations by practitioners. The framework is based on the premise that "very different educators need to ask similar questions" (Ehrmann, 1999a). It relies on an analysis of three elements (Oliver and Harvey, 2000):
  • a technology,
  • an activity for which is used,
  • the educational outcome of the activity.
The focus of the tool is to identify and develop appropriate questions which can be used (in questionnaires or structured interviews) to generate the data necessary to address the focus of the evaluation.
"The Flashlight: The act of program evaluation in education is like using a small dim flashlight to decide what sort of animal might be in front of you in a pitch black cave…The relative brightness (rigor) of the flashlight (evaluation) is less important than where one points the beam (asking the right evaluative question). Each evaluative question is equivalent of pointing the tiny beam in a particular direction and waiting to see what walks into the light." (Ehrmann, 1997)
Questions are generated through focussed brainstorming activities and drawn from existing questionnaire banks such as the Student Inventory, a repository of questions developed by other users of the Flashlight toolkit. The tool can be used for a variety of applications including (Ehrmann, 1997):
  • guiding and improvement of courses,
  • evaluating funded projects,
  • improving technology based support services,
  • supporting strategic thinking about the curriculum and technology services,
  • preparing for accreditation,
  • helping faculty, departments and institutions compare their uses of technology and outcomes,
  • redesigning student evaluations of faculty.
The Flashlight toolkit is documented in the Flashlight Evaluation Handbook by Stephen Ehrmann (1999b).
Many OTiS case studies reported the use of questionnaires including Muirhead, Blom and Jarosleva and Milulecka, all of whom include their questionnaires in their studies. Questionnaires can be used for formative and summative feedback (Webster), to find out about a diverse range of topics including technical and pedagogical issues (Creanor), preferred learning styles (Whittington and Dewar), online activities, students' progress and concerns (Morrison), and styles of communication (Muirhead).
3.2.6 The context, interactions and outcomes (CIAO!) framework
The CIAO! evaluation framework (Scanlon et al, 2000) has been developed by the UK Open University over a period of twenty-five years to evaluate information and communications technologies for learning. It focuses on context, interactions and outcomes. The CIAO! model (Jones et al, 1996) is reproduced in Figure 3.2 below. It draws on a variety of methods including: large-scale pre and post questionnaires; interviews with staff, students and course designers; and automated data collection of computer usage times. Morrison's and Macdonald's case studies report evaluations based on this framework.
 
 
 
 
 
Context
Interactions
Outcomes
Rationale
In order to evaluate learning technologies (LT) we must know about its aims and the context of its use.
Need to observe and examine the learning interactions in order to focus on the learning process.
Need to assess the achievement of learning outcomes (changes in cognitive and affective) and attempt to attribute these to the use of LT.
Data
Course/CAL designers' aims, policy documents and meeting records.
Records of student interactions, student diaries and online logs.
Measures of learning, changes in students' attitudes and perceptions.
Methods
Interviews with course/CAL designers, analysis of policy documents.
Observation, diaries, video/audio recordings and computer logs.
Interviews, questionnaires, tests.
Figure 3.2 - CIAO! evaluation framework, UK Open University
This model has been adapted and used to evaluate the used of learning technologies in the UK Further Education sector (Jones and Scanlon, 1999). Barnard et al (2000) have produced a generic set of evaluation tools, which consists of questionnaire templates for managers, lecturers and students, which can be customized easily. The approach recommends that the following features be considered in any evaluation:
  • that findings are more persuasive if data is gathered from as many different sources as possible and the conclusions are reached by a process of triangulation (Section 3.3.3),
  • evaluation should be directed by each manager's or lecturer's objectives,
  • evaluation should involve real students in real situations,
  • open-ended questions should be included to investigate unanticipated issues,
  • strategies to develop ownership of the evaluation need to be developed.
3.2.7 Evaluating cost effectiveness
Many current evaluation methodologies do not address the issue of the costs. The Cost of Networked Learning project (http://www.shu.ac.uk/cnl/) aims to establish the true costs of implementing learning technologies, including hidden costs, for all participants. The project is currently recommending an activity-based costing model. The first report (http://www.shu.ac.uk/cnl/report1.html) includes a suggested framework for evaluating costs and the second report (http://www.shu.ac.uk/cnl/report2.html available from September 2001) presents the results of applying this framework in an authentic context.
None of the OTiS cases studies reported evaluations of cost effectiveness or cost benefits.
3.2.8 Evaluation of Learning Technology (ELT) Framework
The BP funded ELT framework is a meta-toolkit which outlines a model for evaluating learning technologies (Oliver et al, 1998; Oliver and Conole, 1998) focusing on the 'qualities' associated with the different evaluation methods, types and data capture and data analysis (Oliver and Harvey, 2000). The move towards evaluating learning technologies in authentic learning and teaching contexts means that the teachers often become the (inexperienced) evaluators. This toolkit helps novice evaluators select from the many evaluation methodologies available and choose the one most appropriate to their needs. The framework provides a model of the evaluation process and provides tools to help evaluators make decisions at key points in this process scaffolding the evaluator through the entire process. The evaluation toolkit consists of six steps:
  • identification of the audience for the evaluation (the stakeholders),
  • selecting and refining the evaluation question,
  • selecting an evaluation methodology,
  • selecting appropriate data capture methods,
  • selecting appropriate data analysis methods,
  • selecting an appropriate format to present the findings.
In essence, the needs of the audience, ie the stakeholders, drive the evaluation process. Each step is supported by tools and activities that allow the evaluator to make informed decisions and move forward to the next step. The ELT toolkit is available in paper-based format (Oliver, 1999a) downloadable from http://www.unl.ac.uk/tltc/elt/toolkit.pdf.
3.2.9 The LTDI Evaluation Cookbook
The LTDI Evaluation Cookbook (Harvey, 1998) is a practical guide to evaluation for non-specialist evaluators including lecturers, tutors, developers and other learning support staff. The cookbook (a meta toolkit) includes a range of information, guidance, resources, ideas and suggestions to help the user design an evaluation process that meets their specific needs. The cookbook is arranged to present this information in a variety of ways:
  • Preparation pages that provide a framework for planning and preparing an evaluation.
  • Recipe pages that provide a summary of the main applications for each evaluation method including guidance on time, effort and resources involved.
  • Information pages provide practical suggestions and advice on the different evaluation methods.
  • Tasting, refining and presentation pages provide guidance on interpreting and using the outcomes of the evaluation and suggesting ways of acting on the results.
  • Serving suggestions provide descriptions of evaluations in practice from some of the contributing authors.
  • selecting an appropriate format to present the findings.
The LTDI Evaluation Cookbook Online is available at http://www.icbl.hw.ac.uk/ltdi/cookbook/contents.html
3.2.10 Evaluation Toolkit for Practitioners (online)
The Evaluation Toolkit for Practitioners was funded by the UK Join Information Systems Committee for Awareness, Liaison and Training. This web based guide builds on the existing work of the ELT framework (Section 3.2.8) and the LTDI Evaluation Cookbook (Section 3.2.9). The online toolkit combines the process model, tools and activities to support decision making with a knowledge base on methods, data capture and analysis techniques to provide an online guide for the evaluation process. It removes the need for the evaluator to have a detailed understanding of each stage of an evaluation and provides a method that guides them through the selection and application process. The toolkit consists of:
  • an evaluation planner,
  • an evaluation advisor,
  • an evaluation presenter.
An overview of The Evaluation Toolkit for Practitioners is available at http://www.ltss.bris.ac.uk/interact21/in21p06.htm and the toolkit is available at http://www.ltss.bris.ac.uk/jcalt

 

 

3.3 Choosing your method - getting started

"Clarity is the key to successful evaluation." (Phillips et al, 2000, p1.3)
It is essential to be clear about (Phillips et al, 2000; Oliver, 1999a):
  • What are the aims or purposes of the evaluation?
  • The target audience(s) - who is the evaluation for?
  • Who should take part?
  • Who will read the results of the evaluation?
  • How will you take action on the results of the evaluation?
(Milne and Heath, 1998) define evaluation procedures and methods suitable for assessing the impact of teaching innovations. General guidelines are that an evaluation plan for an innovative learning environment must identify:
  • the specific learning (or other) objectives the environment was designed to achieve,
  • the available range of data sources and methods for continuous study of learning process and outcomes with minimal imposition of extra workloads on staff and students,
  • suitable means of assessing software related issues throughout the development, implementation and operation phases,
  • the range of situational factors that may influence learning and the means of measuring their impact,
  • anticipated results, stakeholders' interests and means of dissemination of findings.
3.3.1 The stakeholders
Identifying the relevant and appropriate stakeholders (as participants or targets for the evaluation report) is key to a successful evaluation. If we can identify the key people involved, we can then focus the evaluation on the key questions they want answered. In an educational context these stakeholders are likely to include some or all of lecturers/teachers, students, managers, funders and support staff stakeholders. These are shown in Figure 3.3 with issues they may be interested in exploring (ELT toolkit, Oliver, 1999a).
Phillips et al (2000) consider these issues from the perspective of the vested interests of stakeholders and extend the stakeholders to include employers and professional accrediting bodies.
Stakeholder
Issues/questions
Lecturer/teacher
· The quality of the students educational experience.
· The time spent on the activity.
· Suitability of the resource technically and in terms of contents.
· The extent to which the resource is integrated with the other aspects of the course.
Students
· The time required.
· Whether this will help them pass their assessments.
· Whether the resource is easy to use.· Whether other course activities depend on them having used the resource.
Managers(course, subject, faculty co-ordinators)
· Efficiency (is this the best use of time and resources?)
· Cost-benefits (do the benefits justify the resources required?)
· Student retention and progression.
· Development of key and employability skills.
Senior managers
Funding bodies
· Does it support/contribute to the institution's strategic mission? (eg Learning and teaching, key skills, widening participation and diversity).
· Quality of the learning experience.
· Value for money.
Support staff
· What additional support and maintenance is required?
· Is the resource properly integrated with other services?
Figure 3.3 - typical stakeholders and their concerns
3.3.2 Formulating the evaluation questions
A detailed statement of objectives is the starting point for any evaluation study. This allows us to identify the achievable means of measuring these objectives. The choice of question can have a considerable impact on the evaluation (Oliver, 1999a). Typical objectives of evaluation studies, stated in general terms, might include:
  • to evaluate learning outcomes,
  • to assess the quality, accessibility and usability of resources,
  • to assess the educational merit of the instructional strategies employed,
  • to reflect on how evaluation findings relate to current theoretical understanding.
The questions should be derived form the list of stakeholder concerns previously identified. We need to formulate a question that addresses a concern, which has the following characteristics (ibid):
  • question format - what, when, for whom, in what situation, and so on,
  • involves a comparison,
  • involves some type of measurement.
An evaluation plan should include detailed descriptions of the objectives as well as the specific criteria that will be used for measurement.
Gwynne and Chester considered some very explicit evaluation questions in their case study Personal Identity and Community in Cyberspace:
  • "How might our sense of who we are be constrained in interactions where the body does not exist? How might it be more fully realised?
  • "What are the implications of the ease of identity play in the virtual context for real interactions?
  • "Without conventional geographic and ethnic markers, are new kinds of communities beginning to emerge? How do these new communities inform their real life counterparts?
  • "In considering the best mode of delivery, the lecturers decided to use the medium of cyberspace as both the content and the process of the subject. The aim was to immerse the students in the technology; to have the process of the subject, that is researching and interacting in cyberspace, become the content through a practice of critical self-reflexivity.
  • "Finally, the decision to present the subject via electronic mode was also informed by an increasing pressure to get aboard the technological bandwagon. Concerned by the possible economic imperatives driving this agenda, the lecturers were keen to evaluate online delivery and consider the implications of such changes for both students and teachers. Could online delivery provide a quality teaching and learning experience?" (Gwynne and Chester)
 
 
3.3.3 Evaluation methods - capture, analysis and triangulation
There is a range of data collection and analysis methods that is common to all qualitative approaches. The differences lie in the types of data collected, the basis for analysis used, the purposes served and the time and resources involved. Oliver and Conole (1998) and Oliver (1999a) distinguish between data capture and data analysis methods.
Data capture methods An important underlying principle is to make evaluation as unobtrusive and integrated as possible to ensure the integrity of the data. Students and staff may not appreciate having demands put on their time with little perceived benefit, and the quality of responses may suffer as a result.
Di