Barbara A. Frey, D.Ed.
Cranberry Training and Development and
Community College of Allegheny County, Pennsylvania
 Karen Overfield, Ed.D.
Education Management Corporation
Pittsburgh, Pennsylvania
 

        Evaluation of student learning represents a crucial, time-consuming part of an educator’s job.  In our years of teaching, we observed many faculty members who, while they have extensive content knowledge, lack skills in assessing student achievement.  As practitioners in a system of 21 career-focused, post-secondary schools, we sought to change this situation in our organization.  The faculty in our system consists of over 1,500 instructors teaching in 15 different majors in the areas of applied arts, design, and culinary arts.  Many of our faculty members come to the school with a great deal of professional expertise but limited background in teaching methodology.   This article addresses the challenges of developing a faculty professional development workshop on assessment, measurement, and evaluation of achievement in adult learners. The workshop represented a developmental experience for faculty based on their needs.
 

        How do I know if my students really learned?  Are my tests valid?  What are authentic assessments?  How do I assess whether students really achieved the learning objectives?

        As educators, we ask these and other questions related to student evaluation.  For most educators, evaluation represents not only the biggest challenge in their teaching but also the part of the job they least like.  Over the years, we have found this to be true based on personal experience, observation, and discussion with colleagues.  Moreover, it does not seem to matter what area of education you are in -- K-12, higher education, public, private, or career-focused; in general, evaluation presents the biggest obstacle for faculty.

        Evaluation of student learning represents a crucial, time-consuming part of an instructor’s job. Evaluation, moreover, presents a key element in the educational experience.  In fact, many people feel evaluation is what sets education apart from the other types of learning experiences we encounter.  This is true particularly of the adult learner.  Adults have many avenues for learning both in and out of the classroom.  What defines the educational learning experience from the others is evaluation.  This critical success factor represents one of the most emotionally intense phases in any learning event for both instructor and student.  Competent adults performing effectively in their “day jobs,” successful in managing their lives, often can be reduced to fear and trembling on the prospect of being evaluated.  For evaluation to be effective and meaningful, it needs to be done right by people skilled in its application.

        Steven Brookfield (1992) states “The only educational justification for evaluation is to assist learning” (p. 22).  If we take this approach, evaluation takes on a different dimension.  Instead of serving to sort students, evaluation becomes a tool faculty use to enhance student learning, self- assessment ability, and reflection.

        But how do we do this?  And, how do we help faculty accomplish this?  In our role in faculty development, we provide support for faculty in a system of career-focused, postsecondary schools.  Our students come to us right out of high school, as adults who have decided they want to gain skill, and as career changers.  All of them have a dream.  And as faculty our role is to help them attain it.  One way to accomplish this is to use evaluation as a tool to enhance student learning.  When we do this, we eliminate the use of evaluation as a way to punish students or as a control mechanism to assert power over them.

        Our  system consists of career-focused, post-secondary institutions, 19 schools spread coast-to-coast.  Over 1000 faculty members teach in 15 different majors including business, culinary, technology, and creative fields.  Associate and bachelor's degree programs are granted; enrollment consists of 14,000 traditional and nontraditional students.

        Our organization recruits faculty from industry. The system of schools takes pride in hiring faculty to teach who are successful and recognized in the field.  Faculty members are practitioners.  They come to the school with a great deal of professional expertise but little background in education or teaching.  The area of assessment and evaluation presents a challenge to our faculty.  How do we provide them with the knowledge they need to be skilled educators?

            To add to the dilemma, educational terminology contains several closely related, and often confused, words.  These terms include assessment, grading, measurement, and evaluation.  The terms are often confused and used interchangeably by educators.  If these terms create confusion for educators, what about people who entered the field through another route?  What is the difference between them?  One way to define the terms is as follows:

Assessment     Analyzes student accomplishment, careful judging of the quality and range of achievement.

Focus is on observing and improving student learning.

Evaluation  The systematic process of collecting, analyzing, and interpreting information to determine the extent to which pupils

are achieving instructional objectives.  Evaluation denotes placing a value on something.

Grading The process of labeling an evaluation.

Measurement   The process of obtaining a numerical description of the degree to which an individual posses a particular characteristic.
 

        Today, educators find themselves in an era in education of increasing concern for accountability.  From this perspective, evaluation takes on even greater significance.  Grade inflation, social promotion, grade retention, and the standards movement signaled educational reform related to evaluation.  Parents sue schools because their children cannot read.  Students sue schools because they graduated and cannot get jobs.  Employers criticize public education because they cannot find employees who possess the skills needed in today’s workforce.  It is a vicious cycle.  Educators are no longer protected from personal liability related to their judgements about and evaluation of students.

        Grades impact students’ personal and professional goals as well as their self-esteem.  Professional schools use grades in making decisions on whom to admit, employers use grades to make decisions on whom to offer a job to; financial aid agencies use grades to qualify people for aid.  Valid grades support learning through feedback on student achievement and progress.  But what if the grades are not valid?  And, how can we be sure they are?  What grade do we as educators get for the marks we give?

        But how could we address this critical success factor in education?  Developing a policy, defining a procedure, sending out a memo did not seem appropriate.  Since we are a school, one way to get at these questions was to provide a learning opportunity for our faculty.  We wanted to enhance institution effectiveness by addressing the root cause and attacking the problem.  We felt that we could all get on the same level and at the problem through creating a community of scholars and learning together and from each other.
 
 

        Since we are responsible for the faculty development function, the authors were faced with the challenge of developing a faculty-training workshop on assessment of student achievement. The goal of the program was to design a faculty development workshop for the system that addressed the area of assessment, measurement, grading, and evaluation.  The intent was that the program could be implemented at all of the schools.  The objective of the program was to provide faculty with the skills they needed to assess student learning.  Implementation thus represented a major challenge.

        Because we wanted to involve faculty, as well as instructional supervisors in the overall evaluation process, we formed a task force to develop it.  Since the task force was geographically dispersed and limited resources of time and money excluded face-to-face meetings, we relied on technology for our communications.  We chose the lowest level of technology to accomplish this.  Task force meetings thus took place via audio conference call.

               As the task force began to discuss the program and consider alternatives for curriculum design, we discovered Donald Kirkpatrick.  Kirkpatrick  (1994) identified a ten-step process for curriculum development.  We based our decision to choose this model on several criteria, including the following:
 

·   Address areas of implementation

·   Provide for program as well as student evaluation

·   Include administration, coordination, and evaluation components

·   Approach from an adult learning perspective

·   Consider the learner as well as other stakeholders in design and development
 

 

Kirkpatrick’s model contains these components:

1.      Determining the needs
2.      Setting objectives
3.      Determining subject content
4.      Selecting participants
5.      Determining the best schedule
6.      Selecting appropriate facilities
7.      Selecting appropriate instructors
8.      Selecting and preparing audiovisual aids
9.      Coordinating the program
10.    Evaluating the program

        Kirkpatrick’s model, moreover, served as a project plan for the task force.  It provided a step-by-step process to follow to take the idea from concept to reality.  The model helped us manage the project and turn a plan into a reality.  A travesty in education is that we often have wonderful plans that never get implemented.
 
 

        Since the audience for the workshop was faculty within the entire system, the need for training in assessment of student achievement was identified by using several different methods.  The use of these different methods provided the opportunity to include a wider sample of the intended audience.
 
 

·         Faculty Survey. After a review of literature on assessment in higher, continuing, and adult education, questionnaires were developed to identify the needs of students, faculty, and administration. Appendix  A gives a sample questionnaire.  This survey was distributed through Deans of Education to 148 faculty within all of the schools.  Forty-two completed surveys were returned, compiled, and tabulated.  The high response rate of 29% we achieved indicated the interest in this critical topic.

·         Faculty Focus Groups. The purpose of the focus groups was to discuss training needs.  In order to reach more of the faculty, two types of faculty focus groups were conducted.  One type consisted of a face-to-face meeting with from 8 to 14 persons.  This focus group was held at the school.  Appendix B shows a sample of the questions that were asked.  The other type of focus group consisted of an audio conference call.  To reach the different time zones and keep the number of participants manageable on the calls, we conducted two audio conference calls.  By doing this, we could obtain a sampling from schools represented in the system.

·          Student Focus Groups. We felt that if we wanted to get a realistic picture of the performance gaps, we needed to question the recipients of the grades as well as the givers of them.  We, therefore, conducted two focus groups consisting of a total of 23 students at the schools.  We asked basically the same questions that we asked of the faculty focus groups only we reversed the wording.

        The results of the needs assessment reflected a need for training in alternatives in measurement and evaluation.  The areas of high interest consisted of the following: learning contracts, class critiques, checklists, rating scales, pretests, case studies, and self-evaluations.  Most faculty members demonstrated high interest and open minds toward the topic.  It was obvious that faculty had many styles of grading that included both norm-referenced and criterion-referenced standards, point and letter grade systems, and various levels of organization and criteria.  Not surprisingly, the majority of assessments were conducted in the cognitive domain.

        Students expressed an overall satisfaction with the assessment techniques already in place. Their primary interest was in as much feedback as possible in as many formats as possible, other than cognitive-type tests.

        Nine administrators completed our needs assessment survey.  Administration consisted of deans of academic affairs and department chairpersons. Their areas of focus were establishing criteria and developing a variety of criterion referenced assessment techniques.  They expressed concern over grade inflation throughout the system.  Administrators were also looking for communication and documentation to reduce legal and ethical challenges of the education system.

Setting Objectives

The following objectives resulted from a thorough analysis of the needs assessment data:

 

·          Define and explain basic assessment terms as they relate to the classroom environment.

·          Identify and determine components of instructional objectives.

·          Determine assessment techniques which support instructional objectives.

·          Compare and contrast assessment techniques for effectiveness in the cognitive, affective, and psychomotor domains.

·          Develop techniques to assess student achievement in each of the three learning domains.

·          Critique instructor feedback comments to identify those that support student learning and improvement.

·          Modify course grading system.

·          Compare and contrast traditional letter, point, and percentage systems of determining student grades.

·          Assess personal performance in communicating clear, accurate, and fair student evaluations.

 

Determining Subject Content

The instructional objectives developed into the following evaluation workshop content outline:

 

I. What are we talking about?
        A. Program overview
           B. Terminology
                     1. Measurement
                     2. Assessment
                     3. Evaluation
                              a. Summative
                              b. Formative
II. How do we do it?
           A. Instructional Objectives
                      1. Learning experience
                      2. Observable outcomes
           B. Level of mastery
                      1. Norm-referenced
                      2. Criterion-referenced
III. How can I assess it?
             A. Knowledge: Cognitive domain
                     1. Pre/post-tests
                     2. Test construction
                     3. Learning contracts
             B. Attitudes: Affective domain
                     1. Rating scales
                     2. Case studies
                     3. Essays
             C. Skills: Psychomotor domain
                     1. Self/peer assessments
                     2. Critiques
                     3. Checklists
IV. What about grades?
             A. Providing constructive feedback
             B. Grading systems
                     1. Conversion
                     2. Weighted points
V. What are the issues?
             A. Grade inflation
             B. Instruction/evaluation improvement

 

 

To role model what we were teaching, we used various types of assessment techniques and aligned the objectives with the content and activities.
 

        For us, workshop operations represented an extensive, time consuming step.  For convenience, we included steps four through nine in Kirkpatrick’s (1994) ten-step planning and implementing process:  selecting participants, determining the best schedule, selecting appropriate facilities, selecting appropriate instructors, selecting and preparing audiovisual aids, and coordinating the program.  Because of the close relationship of all of these steps, much of the work on them overlapped and thus was done simultaneously.

        The task force debated much over the audience for the workshop.  Some felt schools should deliver it as part of their in-service program.  Others felt it should be part of an orientation/induction program.  Still others felt it should be delivered on an as needed basis.  Another issue was whether to make the training voluntary or mandatory.  To ensure quality dialogue and transfer of learning, the faculty training group size was limited to 20 - 22 participants.

        Another related administrative issue was the length of time for the workshop.  The module’s length was projected at 3 hours.  Because of time limitations on faculty, one option was to hold the session as a half-day workshop.  Another was to hold it during the lunch hour or preparation time for 45 minutes every week for 3 weeks.  Still another was to pay faculty to come in and hire substitutes to teach their classes.

        Where to hold the session represented another challenge.  Should it be at the school or off site?  If it were at the school, faculty would likely be interrupted.  If it were off-site, we would have transportation and facilities costs to consider.

        Because of the unique needs of the schools, we left these coordination choices up to the schools.  As a job aid for the program planner, however, we included administrative checklists in the leader’s guide.

        The design of the workshop itself was based on theories of adult learning.  The task force agreed that we needed to model the principles of adult learning in the workshop if we wanted our faculty to. We therefore wanted to design a highly interactive format making use of individual, partner, and small group experiential activities.

        A related design issue was consistency and duplication of delivery.  As a task force, we reached consensus that we wanted the workshop to “belong” to the schools.  Since we had 19 schools and did not have someone dedicated as a “system” trainer, we needed a way to create a package someone could deliver on demand at the school.  We agreed we wanted a format for the leader’s guide that would serve as a resource for the facilitator.  The format should be flexible enough for the facilitators to “make it their own,” yet structured enough so they would not have to duplicate effort.  We, therefore, devoted much time and effort to the design of the leader’s guide, support materials, and packaging.  The leader’s guide contained both an outline of each section of the module as well as a “scripted” version to help the facilitator prepare to teach the module.  Transparencies and participant handouts were created as well as answer keys and explanations. Participant manuals provided the course and content overview, activities, assessment examples, related articles, and a resource list.  We provided disks for the facilitators to modify and duplicate the documentation.

        Once the design was completed, we conducted a pilot.  The pilot provided a way for us to conduct formative evaluation.  The pilot, moreover, gave use a way to evaluate workshop effectiveness in a clear, timely manner.  One of the schools volunteered to conduct the pilot as part of their regular faculty in-service program.  Minor modifications were made to the program resulting from feedback from the pilot study.

        To build a community of learning, we decided to certify people in the system as facilitators.  This process served as a form of reward and recognition for faculty.  Schools could nominate strong faculty to go through the certification process.  The process consisted of participating in the workshop, team teaching it with a “certified” facilitator, and then teaching it on their own.  Once “certified” they could teach the module at their own school or perhaps be called upon by other schools to deliver the workshop at their location to their faculty.
 
 

        According to Kirkpatrick's ten-step process for planning and implementing programs, evaluation constitutes the last step.  How can you tell if the program worked?  What value did you place on the event?

        Evaluation represents a critical success factor in program planning.  In fact, evaluation plans can comprise an entire plan within the program plan.  Unfortunately program planners quite often leave this step out.  In their haste to implement the program and get it out of the door, they neglect to evaluate it or try to retrofit something after the fact.

        In reality, program planners should select indicators they plan to use for the evaluation as part of the up-front planning process.  If you do not have a clear objective of what you want to measure, it is difficult if not impossible to measure it.  When benchmarks are not identified, organizations have no way to identify the value added the program provides.  This practice in many cases leads to the reality that the first area organizations cut when times are tough are the training programs, simply because they cannot prove whether or not they did what they said they would.  We as educators need to become more proficient at identifying both the qualitative and quantitative benefits our programs provide.

        Kirkpatrick developed a 4-level evaluation model that correlates to his ten-step process for planning and implementing programs.  This evaluation model is widely used in business, augments the ten-step process model, and correlates well for education.   Kirkpatrick's evaluation model contains these levels:
 

1. Learner reaction             Were the participants satisfied with the program
2. Learning                          Did the participants change their attitudes, enhance their knowledge, or increase skill levels?
3. Application                      Did the participants use what they learned?
4. Results                            What was the benefit to the organization?

        Following this evaluation model, the program contained several levels of evaluation.  Four measures were used to establish the effectiveness of the program:

 

§          End of workshop reaction surveys

§          Review of instructor records and course syllabi

§          Classroom observations

§          Stakeholder focus groups

§          

        Logic tells us that a positive reaction is more likely to enhance a transfer of learning.  One way, therefore to measure the effectiveness of the workshop was whether the participants reacted favorably to it.  Reaction surveys included ratings on facilities, content, presentation, and activities.  Space for additional comments was also provided; and signatures were optional.  Each time the workshop was delivered, the same evaluation was used.  In this way we could analyze, monitor and compare the results.

        During the workshop, faculty had opportunity to revise and modify several of their course records.  The review of instructor records provided a basis of comparison between before and after the training session. The records that were reviewed included course syllabi, project descriptions, and faculty grade books.  Through assessment techniques and strategies designed for the workshop, faculty increased their awareness of the concepts.  Greater attention seemed to transfer to more specific, carefully worded objectives.  Objectives were more closely linked to evaluation techniques.

        In addition, instructional supervisors periodically review course records generated by faculty.  After faculty attended the workshop, instructional supervisors commented that tests reflected more multiple choice items at various levels of Bloom's Taxonomy and less true/false and essay questions.

        Evaluation of behavior is Kirkpatrick's third level of evaluation (1994).  Classroom observations reflected the level of communication of grades and grading criteria.  Classes targeted were first course meetings, class critique sessions, and sessions in which major projects or exams were returned to students.  Moreover, in the focus groups conducted as follow up to the workshop, faculty reported more time and attention devoted to writing meaningful comments.  They also stressed their goal to return assignments in a more timely fashion.  Overall, the workshop seemed to enhance instructors' confidence in their grading system.

        Focus groups with stakeholders were a part of the initial needs assessment.  Following the training, focus group meetings with students, faculty, and academic department chairpersons compared the same issues to compare the productivity of the training workshop.
 
 

        A review of the literature written on faculty development noted a resistance of faculty members to participate in training. Generally, faculty are overwhelmed with time constraints and view training as necessary for others, but not for themselves (Maxwell and Kazlauskas, 1992 and Millis, 1994).

        In this case, workshop designers recognized the importance of faculty participation in the planning of the faculty development effort.  There was a successful balance of faculty, student, and administration participation in planning.  The design of the program was not aimed at ineffective faculty, but offered opportunities for all.  Having a faculty member co-present with an administrator aided in achieving faculty support for the program.  The message to faculty was they were valued and respected.  Excellence in higher education is inseparable from the excellence of its faculty.

        Conducting a pilot workshop with a small group of faculty was essential to the ultimate success of this project.  The piloted study allowed for reinvestigation of activities and time allocation.  Feedback allowed trainers to evaluate the readiness of faculty for the program content.  Design modifications were made to allow for a more effective transfer of learning.

        It appears well planned faculty development programs can make a difference.  In the future, faculty will be faced with the continued challenges of under-prepared students, technological innovations, and increased accountability.  Faculty must be helped to meet these and many more challenges through lifelong learning and development.
 
 

        Angelo, Thomas A. (1993). Classroom assessment techniques. San Francisco: Jossey-Bass.

        Brookfield, S. (1992). Giving helpful evaluations to learners.  Adult Learning: 22-24.

        Kirkpatrick, D.L. (1994). Evaluating training programs. San Francisco: Berrett-Koehler..

        Maxwell, W.E. & Kazlauskas, E.J. (1992) Which faculty development methods really work in community college? A review of literature. Community/Junior College Quarterly, 16, 351-360.

        Millis, B.J. (1994). Faculty development in the 1990's: What it is and why we can’t wait. Journal  of Counseling and Development, 72, 454-464.
 
 

Appendix A

Faculty Needs Assessment Questionnaire
Assessment, Measurement, and Evaluation

Please answer the following questions on techniques to measure or assess student achievement.  We will use your feedback to guide us in future faculty development initiatives.  We are interested in how you are evaluating your students and establishing grading criteria, as well as your interest in developing additional techniques.

1. Please indicate with a check mark which methods you are currently using and those you would like to consider or learn more about.

  
  
 

  2. Indicate with a check mark which of the following criteria you include in determining students’ grades.
  

 
 

3. If you give paper and pencil tests, please indicate with a check mark which types of questions you currently use or those you would like to consider or learn more about.
 

4. If you teach a studio course, please indicate with a check mark those techniques you use to assess student achievement.  Also check those techniques on which you would like more information for future consideration.
   

5. Please answer the following questions based upon the grading procedures of your classes. 
  
 

QUESTION	YES	NO	N/A
A. Do your students know ahead of time what is expected to earn each grade?
B. Is the grading criteria of each course stated on your course syllabus?
C. Are grades students earn in your class an accurate indication of what they have learned?
D. Does every student have an equal opportunity to earn each grade?
E. Do you maintain appropriate documentation and accurate records to verify grades?
F. Are course objectives the basis for rating student achievement?
G. Is the competence of each student based on pre-set standards?
H. Are evaluation changes or modifications made based on a thorough analysis of grading procedures and/or policies?
I. Is the number of projects or tasks completed during a quarter considered in your grading criteria?
J. Are your grading policies designed to reduce competition among students in the class?
K. Are grades of a class determined on a curve?
L. Has “grade inflation” occurred in the grading process of your course?
M. Do you determine grades by converting raw scores to a percentage?
N. Do you provide feedback in the form of comments on student tests, projects, or assignments?
O. Are all student projects graded on the same set of characteristics?