Assessing Assessment

If there were an assessment of progress on the Ministry of Education's reforms to assessment and evaluation practices for secondary schools from 1997, it might read something like this:

Level 2: Pockets of excellence, need to apply ideas more thoroughly and connect theory with practice more consistently.

In 2001 Lorna Earl and a team of OISE/UT researchers examined the early impact of Secondary School Reform (SSR) in schools around the province and found that:

Teachers, students and parents were struggling to understand the changes. Teachers felt as if they were trying to . move things around in ways they didn't understand and weren't sure that they agreed with, without the necessary resources, training and support to do it well.

The questions now are: does this confusion and lack of agreement persist or have teachers now adapted to the revised requirements for assessing and evaluating their students? The answer to both is yes.

The goal of assessment reform was to improve the accountability, effectiveness and quality of Ontario's schools through defined standards, centralized curricula, indicators, rubrics and aligned assessments, among other things.

Wilma Davis, the ministry's senior media-relations co-ordinator, explains that with today's standards-based curriculum and achievement charts teachers are evaluating student work using common province-wide criteria. "In the previous system of assessment, standards varied from teacher to teacher and school to school. Now we have well-defined standards commonly understood and consistently applied across the province. As a result, assessment of student achievement is fairer and more reliable."

The bad with the good

Reg Hawes, who teaches at the University of Toronto Schools and is principal of OISE/UT's AQ History, Geography and Social Sciences courses, is one of many who see the changes to assessment as positive.

"Before, students had one chance on an assignment," says Hawes. "If they failed, then too bad because the class had already moved on by the time it was returned." Now teachers are responsible for getting students to where they should be using various types of formative assessments and evaluations. This can help students succeed.

Hawes believes that well-written rubrics are fair and transparent. They link assignments to curriculum expectations. Students are better prepared for major evaluations because they know what is expected of them and have received feedback as they have learned and practised new skills.

The objections that persist are not because of the intention or the value of the new assessment strategies.

Barbara Bodkin, head of Continuing Education and AQs at OISE/UT, stresses that changes such as the curriculum planner and provincial report card were based on sound research and exemplary educational practice. "Both were based on the design-down model and reflect a major shift and improvement in how we develop and deliver curriculum and assess and evaluate students."

"If lateness is not being marked what are we teaching about the need to meet deadlines?" Geoff Watson

However, when asked about implementation in all schools, Bodkin replies, "Not yet; deep and meaningful change takes time, but these changes are important and we'll get there."

Teachers who were both leading and taking recent AQ courses at OISE/UT are quite clear about the difficulties.

"It's difficult to use a variety of strategies to ensure that students know exactly where their marks and grades come from," says Katy Whitfield, who has been teaching in the Toronto District School Board (TDSB), for three years. "I know the least about rubrics, even though I have sought resources, spoken with colleagues and tried to follow the examples from the ministry."

"I understand that I am unable to take off marks for being late," says Geoff Watson, now in his sixth year teaching geography and science for the TDSB. "But if lateness is not being marked, what are we teaching about the need to meet deadlines?"

The ministry's Program Planning Assessment posits that taking off late marks gives an inaccurate picture of students' acquisition of curriculum expectations. Instead, students' lateness should be reflected separately in the learning skills' evaluation on their report cards - unlike in the past when an outstanding assignment could receive a failing grade if handed in late.

Davis says that this separate reporting "allows teachers to provide specific information to parents and clearly identify students' strengths, weaknesses and the necessary next steps."

Frustration persists

"Tension remains in schools over the changes," says Fraser Cartwright, who worked in York Region for 28 years as a geography teacher, department head and consultant and who now teaches the Geography Honours Specialist AQ at OISE/UT.

"Students and parents demand marks - numbers," says one teacher who has taught both elementary and secondary students in the Peel District School board for seven years, "And they want biweekly mark printouts. How do we reconcile this if we're using so many rubrics?"

The ministry directs that formative assessment helps students understand teachers' marking criteria and expectations in advance of "marks that count." But as one TDSB teacher put it, "It's frustrating to create assignments, mark them and then not see them count towards the final mark."

Cartwright also laments that even new teachers often conform to pre-SSR ways of doing things. "It's easier to use existing teaching and learning materials, many of which do not link assessments and evaluations to the learning expectations." But he also sees improvements. More teachers are aware of new assessment practices and approaches to curriculum design.

Ontario faculties of education are striving to ensure that their courses meet today's assessment and evaluation requirements.

"Our instructors challenge AQ candidates to be creative with curricular designs and apply what they learn." Sheila Pinchin

Sheila Pinchin recently co-ordinated revisions to Queen's University's AQ courses, most of which are now delivered online. "We wanted to encourage teachers to create more opportunities for students to demonstrate their learning," says Pinchin, "especially using higher-order thinking skills and meta-cognition."

Pinchin and her colleagues applied the design-down approach to curriculum, advocated by Wiggins and McTighe in Understanding by Design, and embraced the ministry's desire for a variety of assessment and evaluation tools.

Both Pinchin at Queen's and Hawes at OISE/UT agree that today's assessment practices are better because, among other things, they force teachers to be explicit about their teaching and grading as they begin each new unit, rather than at the end when they return a marked assignment or test.

"Our AQ courses encourage teachers to develop a continuum of formative and summative assessment tasks to use with students," says Pinchin. At the beginning they place pre-tests, checklists and diagnostic tests. Towards the middle we see self- and peer-assessment, short-answer questions and academic-prompt questions such as essays and activities where students apply what they have learned to other situations. Near the end we find portfolios, performance tasks, culminating activities and large review projects that help students connect the various skills and content they have learned.

"Our instructors challenge AQ candidates to be creative with their curricular designs and to apply what they learn to their own practice," says Pinchin.

Meeting the challenge

There's no doubt that teachers have had to consider major changes to their practice in recent years.

"For meaningful change to happen," says Ken Hanson, "teachers must be able to articulate to themselves, students and parents what they are doing and why."

Hanson taught history and social sciences in Toronto before retiring as a principal in 2003 and has taught the Contemporary Studies AQ at OISE/UT since 1992. He paraphrases Michael Fullan's three levels for approaching change:

• Read a range of topics including how to make distinctions between assessment, evaluation and grading; why and how to formatively assess students with conferences, checklists and rubrics; and how to effectively use authentic performance evaluations. Hanson suggests the work of Grant Wiggins as an excellent place to start.
• Apply what they learn by developing and testing rubrics and trying authentic performance tasks.
• Believe that what we are doing is the best way to help our students learn and succeed.

"We don't spend as much time on content," he says when asked how his AQ instruction has changed since SSR. "We focus on good pedagogy, clustering expectations to allow for deep exploration of key expectations and sound assessment and evaluation practices." He stresses the importance of curriculum design that links expectations to assessment, saying that teachers must think like assessors rather than activity designers by beginning with the end in mind. (See Professionally Speaking for Understanding by Design, June 2004, and Curriculum Mapping, September 2004.)

"Teachers must be able to articulate what they are doing and why." Ken Hanson

Mark Melnyk became head of history at Markville Secondary School in the York board in 2000, the year he took Ken Hanson's Contemporary Studies AQ. "Ken taught us the differences between assessment and evaluation. That's a key distinction that many teachers struggled with."

Melnyk says, "It took time for teachers and students to appreciate that, while formative assessments didn't count towards the final mark, they did give students vital feedback as they acquired and refined the skills and knowledge necessary for success in major evaluations."

Assessment itself takes time. And teachers can be overwhelmed with the volume of marking they have to do to give students meaningful feedback on how well they are meeting the new expectations. Melnyk and his colleagues quickly learned that survival required easy-to-use, detailed and specific assessment rubrics and checklists.

"We realized that we needed fewer major evaluation tasks, which meant less marking," says Melnyk. Oral feedback is also encouraged. It takes some class time but can greatly reduce the marking load.

Good rubrics

The creation of good rubrics is both difficult and time consuming.

"If we're going to have province-wide standards, expectations and exemplars," says Melnyk, "then it would make sense to have rubrics that all teachers can use and modify." Like many teachers, he finds it frustrating that the ministry has not set up a bank of exemplary rubrics for all subjects. He stresses that, while rubrics should be detailed and specific, good ones can be modified endlessly.

Melnyk thinks that most rubrics try to cover too much. He has several criteria for a good rubric. It should be specific to the skill(s) being assessed, and the differences between achievement levels should not be as simple as "never, rarely, sometimes and always."

He also points out that achievement charts are not meant to be used as rubrics for assessment or evaluation. Rather, they provide students and parents with an overall description of achievement at each level and a general description of strengths, weaknesses and the next level to work towards.

Despite the ministry's endorsement of rubrics, they have limits. "Many teachers use rubrics designed to measure how much students are learning," says Sheila Pinchin. "But these cannot capture the quality of or growth in students' learning experiences."

"We encourage using the ICE model as another way of thinking about assessment and rubrics." Elaine Van Melle

Queen's University doctoral candidate Elaine Van Melle has drawn on the work of Sue Fostaty Young and Robert J. Wilson to develop a rubric that encourages teachers to move beyond describing the ideas accumulated to assessing the development of higher-order thinking skills.

"The ministry wants students thinking about and extending what they are learning, and that's challenging for teachers to teach and assess," says Van Melle. "We encourage using the ICE [Ideas, Connections and Extensions] model as another way of thinking about assessment and rubrics."

The ICE concept was first explored by Wilson in Assessing Students in Classrooms and Schools (1996). It requires teachers to teach and assess students as their thinking progresses along a continuum.

"Because there are three, not four levels," says Pinchin about ICE, "teachers cannot automatically link this rubric to achievement and marks. Many forget that rubrics and marks are distinct, and our AQ courses help them turn assessment data from rubrics into marks."

Teachers continue to be challenged by the important task of assessing their students. But there is exemplary work in our schools and universities and it is moving us towards what Bodkin calls deep, meaningful and important change.

Assessment and Evaluation Suggestions from Ken Hanson and Mark Melnyk

• Ensure alignment between assessments, expectations and your big questions so that students can demonstrate achievement of the expectations.
• Explain to students and parents how formative assessments guide students to success with culminating evaluations and that while not everything has to count in the final evaluation, everything contributes to greater success on those evaluations.
• Remember that assessments improve learning; they do not address class management or student motivation.
• Communicate what assessments will be done in each unit and what will be judged or evaluated. In other words, plan with the end in mind and think quality not quantity.
• Use detailed and focused rubrics to show students what they are doing well and where and how they must improve.
• Keep rubrics focused on a few skills to accelerate the marking process.
• Don't equate levels of performance in the rubric with specific marks - use performances as pieces of evidence to be added to the mix of student marks.
• Use rubrics to assess complex performance tasks and share them with students at the beginning of the task.
• Have students create rubrics from exemplars or from brainstorming and let them assess themselves against the rubric, keeping in mind that the teacher assigns the grade at the end of the day.
• Teach students how to peer- and self-assess, but avoid having them assign marks to one another in peer- and self-assessments. Students can grade short answer and other types of recall tests.
• Conference orally as much as possible and keep assessment notes on your students.
• Show students examples of level-four work and post exemplars in your classroom and on your school or department web site.
• Share with colleagues and administration - don't reinvent the wheel.
• Avoid assigning grades to everything students do.
• Mark what you value and what will allow students to demonstrate their learning.

Melnyk and his colleagues have posted some exemplary rubrics on their school's web site. Visit www.markville.ss.yrdsb.edu.on.ca/history/ for a rubric used in summative evaluation for Grade 10 Canadian history.

Ideas, connections and extensions

The ICE model helps to identify what students will do with the learning from your course.

Ideas

Ideas are demonstrated when students convey

• the fundamentals
• basic facts
• vocabulary/definitions
• details
• elemental concepts

Connections

Connections are made when students demonstrate relationships

• among the basic concepts
• between what is learned and what they already know

Extensions

Extensions are revealed when students

• use their new learning in novel ways, apart from the initial learning situation
• answer the hypothetical questions: What does this mean? How does this shape my view of the world?

By the end of my course, students should be able to .

• Explain the basic principles of .
• Solve problems using .
• Apply what they have learned to .

Adapted by Elaine Van Melle from Assessment and Learning: The ICE Approach, S. Fostaty Young and R.J. Wilson (Winnipeg: Portage and Main Press, 2000)