Chapter III- Criteria for Evaluating Data on Teaching

 

This discussion of evaluation criteria is meant to assist the department or college in establishing its own system for the evaluation of teaching. The dictionary uses the terms "standard, rule or test" in defining criterion. Tenure decisions require a general rule for defining excellence that can accommodate the variety of disciplinary-based instructional traditions, while honoring the individual's freedom to express and develop personal style in teaching. An example of such a general rule might be, "To what degree does the data supplied support the reasoned opinion of those most competent to judge that the candidate has-and will continue to-demonstrate the capacity to improve instructional practice?"

A distinction has been made between promotion and tenure criteria: promotion criteria focus on merit of the candidate's professional and scholastic contributions and promise, whereas tenure criteria focus on the long-time worth of the candidate's professional and scholastic contributions and promise. "Worth requires merit, but merit is not a sufficient condition for worth."¹

"Merit is free of the specifics of the pool; it is criterion referenced and deals with the candidate's ranking on those criteria. Worth is utility to the hiring party."² Based on these definitions, if tenure candidates sufficiently prove their capacity and commitment to continually improving research and teaching practice, their long-time worth to the unit and institution will be greater than if they have merely made a case that their performance has measured up to a universally established absolute standard. Normative standards are necessary for determining merit, but merit is a concept that may be more relevant at the point of hiring and when the candidate is being considered for promotion.

In general, criteria for evaluating teaching will be more useful in the tenure and promotion process if they

can discriminate between teachers in terms of specific competencies

can reliably and consistently measure a specific competency both for the same individual over time and between individuals

maintain a neutral orientation relative to individual style and viewpoint

yield information about instructional situations where the teacher functions best³

This chapter provides suggested criteria to use to evaluate teaching, based on the data categories described in Chapter 3. It begins with a general discussion of effective teaching and goes on to include criteria for use by students, criteria relevant to evaluation of teaching materials, and criteria for use by peers, including classroom observations, all of which have been developed through controlled inquiry in work carried out within the last 25 years.

Effective Teachers - A Description
During the past 50 years the debate over effective teaching has moved from a discussion of technical, classroom skills, or process skills as they have been called, to a focus on skills necessary to make the subject matter understandable to the student, what one author calls "the pedagogy of substance." Thinking dichotomously about teaching-as either a technical process skill divorced from the subject matter or solely a matter of translating abstract and technical information into understandable terms-limits the conception of what teaching is. Looking at teaching as a scholarly activity that is connected to research suggests a dialogue between the tasks of understanding a body of knowledge and explaining it. Effective teaching must be concerned with both of these areas of expertise: I am no more effective if I have a body of knowledge to profess but am unable to communicate it than I am if I can hold students rapt in wonder who do not know what I am talking about.

The dichotomy can be avoided by a more integrative model of teaching: effective teachers are able to understand enough about their students' ways of thinking that they can translate their own understanding of the subject matter into a form that connects with their students.

...one of the things we see when we look at teaching analytically is this combination of an emphasis on understanding the subject matter, understanding how it is represented in the heads of students and then being able to generate representations of your own as a teacher that will be a bridge between the subject matter and the students.⁴

Recent work on teacher effectiveness has yielded the following observations which support an integrative model that is both process- and content-based (italics in original):

Teachers promote learning by communicating to their students what is expected and why

Effective teachers not only know the subject matter they intend their students to learn but also know the misconceptions their students bring to the classroom that will interfere with their learning of that subject matter

Effective teachers are clear about what they intend to accomplish through their instruction, and they keep these goals in mind both in designing the instruction and in communicating its purposes to the students. They make certain that their students understand and are satisfied by the reasons given for why they should learn what they are asked to learn.

Effective instruction provides students with metacognitive strategies to use in regulating and enhancing their learning. It also provides them with structured opportunities to exercise and practice independent learning strategies.

Effective teachers create learning situations in which their students are expected not just to learn facts and solve given problems but to organize information in new ways and formulate problems for themselves. Such learning situations include creative writing opportunities in language arts, problem-formulation activities in mathematics, and independent projects in science, social studies and literature.

Effective teachers continuously monitor their students' understanding of presentations and responses to assignments. They routinely provide timely and detailed feedback, but not necessarily in the same ways for all students.

Effective teachers realize that what is learned is more likely to be remembered and used in the future if it serves students' purposes beyond meeting school requirements.

. . . effective teachers . . . take time for reflection and self-evaluation, monitor their instruction to make sure that worthwhile content is being taught to all students, and accept responsibility for guiding student learning and behavior. . . . the same research . . . has made it clear that few teachers follow all of these practices all of the time.

. . . teachers must cope with a full agenda that typically precludes time for serious reflection . . .⁵

The last two points raised in the quotes above deserve to be emphasized. First, teachers are human and not machines. Strict adherence to a set of principles does not in itself establish effectiveness. I may, for any number of acceptable reasons, occasionally exhibit inconsistency in teaching practice. The more important issues are: to what degree is my practice governed by some explicit pedagogical framework, and how frequently am I unable to follow my own guiding principles of teaching, which my experience has shown to produce desirable results. Second, the extent to which I can be effective will be governed, to a certain degree, by the environment and conditions under which I must work. I only have so much time and energy, and I have a life beyond my work, which has its own demands. These are facts we take for granted, but because we take them for granted, we may be in danger of forgetting them during the rigor of a tenure decision. A case where a newly hired faculty member is assigned to teach five courses represents a much more stressful situation than a case with a lighter teaching load. Work load is an important factor to be considered when evaluating a candidate on the following departmentally based criteria:

Has the candidate assumed the responsibilities related to the department's or university's teaching mission?

Does the candidate recognize the problems that hinder good teaching in his or her institution and does he or she take a responsible part in trying to solve them?

If all members of the faculty were like this individual, what would the college be like?

To what extent is the candidate striving for excellence in teaching?⁶

If teaching is to be adequately rewarded as a valued activity and contribution to the department or unit, the degree to which a candidate has accomplished the following should be recognized:

whether there is sufficient data on teaching quality

whether alternative teaching methods have been explored

whether changes have been made in the candidate's courses over time

whether the candidate sought aid in trying new teaching ideas

whether the candidate developed special teaching materials

whether the candidate participated in teaching improvement opportunities

A study carried out at Berkeley (Hildebrand, Wilson & Dienst, 1971) was designed to discriminate between best and worst teachers. One set of scales were factor-analyzed out of student survey data that are relevant to evaluating teaching by students. They are:

Scale 1: Analytic/Synthetic Approach, relates to scholarship, with emphasis on breadth, analytic ability, and conceptual understanding.

Scale 2: Organization/Clarity, relates to skill at presentation, but is subject-related, not student-related, and not concerned merely with rhetorical skill.

Scale 3: Instructor-Group Interaction, relates to rapport with the class as a whole, sensitivity to class response, and skill at securing active class participation.

Scale 4: Instructor-Individual Student Interaction, relates to mutual respect and rapport between the instructor and the individual student.

Scale 5: Dynamism/Enthusiasm relates to the flair and infectious enthusiasm that comes with confidence, excitement for the subject, and the pleasure in teaching⁷

A second set of scales were factor-analyzed out of survey data from faculty colleagues. These surveys were also designed to discriminate between the best and worst teachers. These scales are relevant for use by colleagues in evaluating a candidate's teaching. They are:

Scale 1: Research Activity and Recognition

Scale 2: Intellectual Breadth

Scale 3: Participation in the Academic Community

Scale 4: Relations with Students

Scale 5: Concern for Teaching⁸

The criteria identified with each scale for use by students and colleagues and which were the most discriminating between the best and worst teachers are included in the tables below. (Factor analysis coefficients that were used to associate the item with the particular scale are included.)

Table 1⁹

Components of Effective Teachers as Perceived by Students*

[scale 2 is not included here because it is colleague, and not student related]

 Scale 1. Analytic/Synthetic Approach                                                Factor coefficient

 

 1. Discuss points of view other than their own                                     .70

 2. Contrast implications of various theories                                         .66

 3. Discuss recent developments in the field                                         .64

 4. Present origins of ideas and concepts                                             .60

 5. Give references for more interesting and involved points                  .53

 6. Present facts and concepts and related fields                                  .53

 7. Emphasize conceptual understanding                                              .46

 8. Explain clearly                                                                               .78

 9. Are well prepared                                                                          .63

 10. Give lectures that are easy to take notes in                                    .62

 

 11. Are careful and precise in answering questions                              .61

 12. Summarize major points                                                               .51

 13. State objectives for each class session                                          .50

 14. Identify what they consider important                                           .47

 15. Encourage class discussion                                                           .70

 16. Invite students to share their knowledge and experiences              .65

 17. Clarify thinking by identifying reasons for questions                       .64

 18. Invite criticism of their own ideas                                                  .62

 19. Know if the class is understanding them or not                              .58

 20. Know when students are bored or confused                                 .57

 21. Have interest and concern in the quality of their teaching               .48

 22. Have students apply concepts to demonstrate understanding         .43

 23 Have a genuine interest in students                                                 .74

 24. Are friendly toward students                                                        .71

 25. Relate to students as individuals                                                    .69

 26. Recognize and greet students out of class                                     .69

 27. Are accessible to students out of class                                          .65

 28. Are valued for advice not directly related to the course                 .64

 29. Respect students as persons                                                         .60

 30. Are dynamic and energetic persons                                              .80

 31. Have an interesting style of presentation                                        .76

 32. Seem to enjoy teaching                                                                .74

 33. Are enthusiastic about their subject                                               .65

 34. Seem to have self-confidence                                                       .64

 35. Vary the speed and tone of their voice                                          .63

 36. Have a sense of humor                                                                 .53

*Based on 1968 Survey, N = 1015

These items can be used either to develope end-of-semester summative evaluation questionnaires or to evaluate other student data on teaching, such as letters. If a numeric evaluation schema is adopted, caution should be exercised in establishing normative data. "The usual overall evaluation of teaching will provide for evaluation on a five-point scale and will permit a classification of teachers as poor, adequate, good, excellent, or outstanding. In practice, the bottom end of the scale is rarely used and the actual range varies between a little under 3.0 to a little over 4.5 That is, anything under 3.0 is poor, and anything over 4.5 is outstanding; the other classifications are arranged in between these two extremes. . . . The concept of improvement implies progressing up the scale."¹⁰

Evaluation Criteria for Use by Peers
Making global assessments of an instructor's overall teaching effectiveness is a practice that is unsatisfactory to the candidate, to those who must make the evaluation, and to the department. A more useful and practical practice is for colleagues to focus on certain qualities associated with good teaching that they are in a good position to judge. The items listed below were those most discriminative between best and worst teachers as perceived by their colleagues (Hildebrand, et al., 1971). They are included here to provide a general profile of effective teaching from which a department may develop its own profile. Because these items discriminated between the best and worst teachers at the p < .001 level they have a high level of validity. The authors of the study suggest they be used as a supplement (and not as a substitution) for student ratings.

Table 2¹¹

Components of the Activities of Effective Teachers as Perceived by Colleagues*

 Scale 1. Research Activity and Recognition                                     Factor coefficient

 

 1. Do work that receives serious attention from others                       .69

 2. Correspond with others about their research                                 .69

 3. Do original and creative work                                                       .64

 4. Express interest in the research of colleagues                                 .55

 5. Give many papers at conferences                                                  .55

 6. Keep current with developments in their field                                 .49

 7. Have done work to which I refer in teaching                                  .48

 8. Have talked with me about their research                                      .38

 9. Seem well read beyond the subject they teach                               .66

 10. Are sought by others for advice on research                                .60

 11. Can suggest reading in any area of their general field                    .59

 12. Know about developments in fields other than their own              .51

 13. Are sought by colleagues for advice on academic matters            .43

 14. Encourage students to talk with them on matters of concern         .60

 15. Are involved in campus activities that affect students                    .58

 16. Attend many lectures and other events on campus                       .47

 17. Have a congenial relationship with colleagues                              .39

 18. Meet with students informally out of class                                    .58

 19. Are conscientious about keeping appointments with students       .57

 20. Meet with students out of regular office hours                              .57

 21. Encourage students to talk with them on matters of concern         .55

 22. Recognize and greet students out of class                                    .37

 23. Seek advice from others about the courses they teach                 .70

 24. Discuss teaching in general with colleagues                                  .60

 25. Do not seek close friendships with colleagues (Negative)             -.47

 26. Are people with whom I have discussed my teaching                   .45

 27. Are interested in and informed about the work of colleagues        .44

 28. Express interest and concern about quality of their teaching          .40

*Based on 1967 survey, N = 119

Once a candidate has been evaluated on these or other criteria, certain precautions are necessary to ensure fairness: "include a review of central tendencies and variations in the rating results; an analysis of the effects of ecological factors, including different types of courses, students, and time frames on ratings in the unit; and the establishment of agreed-upon standards and steps to be taken in the application of the standards."¹²

Classroom Observation by Peers
What happens in the classroom can have a substantial impact on student relationships with the course material. It is therefore important to add to students' and the candidate's own perspective a third view of classroom performance by peers through planned observations. Studies seeking to determine whether peers can reliably and validly evaluate classroom performance through observations have been discouraging, however. "It is not clear . . . whether the validity and reliability of classroom observation procedures warrant their being considered as a legitimate approach for summative evaluation."¹³

Reliability and validity of classroom observations can be enhanced if guidelines are established that address the following issues: how many visits and when are they carried out; who does the observing; how are observers selected and how many people are involved in the observations; what is observed, and, consequently, what is the character of the observational report; and to whom do observers report?

The following guidelines can enhance the quality of classroom observation by peers:

1. use with caution, training of observers is suggested to minimize bias

2. use several observations by several people over time

3. select observers with no biases (use multiple observers)

4. observations should be done with prior notification of candidate

5. observational criteria should be oriented towards currency/accuracy of material & ethical conduct (content & professionally oriented) rather than stylistically/rapport-oriented

6. records of colleague observational data should be summarized with explicit descriptions of the context of the observation¹⁴

Staff in the Office of Instructional Support have developed a protocol for classroom observation and performance review, which they have taught successfully to many individuals. This protocol is based on a cognitive development paradigm that fosters the improvement of practice, rather than on a remedial approach that limits teaching to a set of technical skills. More will be said about this process in Chapter 5. The following questions are included to assist the department in developing a comprehensive and consistent classroom observation protocol.

Classroom Observation Items

The instructor was fully prepared for class.

The instructor provides an overview of what is planned for the class period.

The instructor emphasizes the conceptual basis of the material.

The instructor's lectures are well organized.

The instructor provides periodic summaries of what has been covered or discussed.

The instructor uses class time efficiently.

The instructor ties things together at the end of class.

The instructor chooses appropriate activities for learning the material.

The instructor asks questions that encourage students to think about the subject.

The instructor is animated.

The instructor clearly explains instructions for completing required tasks.

The instructor leaves enough wait time after asking questions for students to think of a response.

The instructor uses eye contact effectively.

The instructor provides clear and comprehensive explanations when required.

The instructor encourages students to ask questions and express their opinions.

The instructor gives clear and understandable responses to students' questions.

The instructor seems genuinely concerned about the students' learning.

The instructor is actively helpful when students need assistance.

The instructor is skillful at promoting interaction among students.

The instructor is able to involve everyone in the class.

The instructor listens carefully to student questions and comments.

The instructor knows when students seem confused.

The instructor provides clear, relevant and understandable responses to student questions.

The instructor periodically checks to make sure everyone understands what has been covered.