04 August 2008

No child left unpunished

One of the major problems [in evaluating public sector services] may be that those aspects of performance which are most easily quantified are not necessarily the most significant parts of the public sector organisation or individuals' work. …

[One] example of the problems inherent in the use of such performance indicators can be seen in the publication of school league tables of examination and test performance. The problem here is that the environmental differences between schools are neglected – together with the starting points from which their pupils begin. Some unofficial attempts have been made to assess the 'added' value by schools but these have received much less attention than the misleading crude headline figures.

(Stephen Tansey, Politics: The Basics, 3rd ed., 2004, 227-228)

Amen! Standardized testing focuses exclusively on the finish line. It totally misses the point that some runners have only 50 yards to go, while others have to run a full 200 yards. It directly penalizes schools that reach out to disadvantaged students, even denying them accreditation.

In fact, it's worse than that. It has been said, "First, do no harm." It is imperative upon the observer to ensure that the evaluation, itself, does no harm. And yet, every teacher I've known agrees that standardized testing reduces the quality of education. At best, tests take time away from teaching. At worst, they encourage teachers to teach "to the test" instead of teaching kids to think.

The Founding Fathers knew that government is most effective at the most local level possible. They instituted the 10th Amendment to ensure that travesties like the No Child Left Behind Act would not happen. And yet, there it is.

But let's assume for the sake of argument that it really is the federal government's job to manage public education. What are some effective ways to evaluate the performance of schools and school systems?

Let's talk first about evaluation metrics in general. Intuitively, they must satisfy at least two criteria:

  1. Each metric, or all metrics taken together, must measure the results that you really care about.
  2. The evaluation, itself, must do no harm.

Standardized testing fails both criteria.

So, what is it we really care about in public education? Some possible answers, off the top of my head:

  1. All children receive equal benefits under the law. Specifically, disadvantaged children must receive full benefits, and so must exceptionally gifted children.
  2. All children graduate high-school with the skills needed to enter college and/or function as an adult in society. These include at least basic reading skills, writing skills, math skills, and social skills. (Provisions must also be made for those who are so impaired that they cannot achieve these basic competencies.)
  3. All children are encouraged to advance beyond these basic skills to the limit of their abilities.
  4. All children have opportunities for education in less basic areas, such as foreign languages and the arts.
  5. All children develop a love for learning.
  6. All children enjoy the education experience.
  7. Every year of every child's life brings about positive growth in the direction of the previous 6 goals.

(5) and (6), especially, are difficult to measure, but cannot be neglected. Standardized testing focuses exclusively on (2) and the corresponding element of (7), ignoring the rest.

(2) is perhaps the easiest to measure, so let's start there. Can we measure how well students are being prepared for college and/or adult life, without causing any harm in the evaluation? Well,we can begin with statistics that are readily available. For each high-school and school system, look at:

  1. Percentage of students that graduate high-school.
  2. Percentage of students that go to college.

Of course, this has the same problem of looking only at the finish line without considering the starting points. So, factor in:

  1. Percentage of student parents who graduated high-school.
  2. Percentage of student parents who went to college.

If (b) is substantially higher than (d), this is seen as successful. So, neighborhood A may have only 30% of graduates go to college, but only 5% of the kids' parents went to college. This neighborhood rates higher than neighborhood B that has 80% of graduates go to college, whose parents are also 80% college graduates.

Note that under the current system of standardized testing, neighborhood B would rate far better than neighborhood A, even though all they're doing is maintaining the status quo, while neighborhood A is bringing about positive generational change.

While we're at it, we can factor in other factors related to starting points, such as:

  1. Percentage of students/parents who are immigrants.
  2. Percentage of students/parents who speak English as a non-native language.
  3. Percentage of two-parent households.
  4. Median household income.

So far we've considered only statistics that are readily available. Even so, we have produced a set of metrics that (I would maintain) are better than standardized testing at reflecting the results we really care about, and do no harm.

We would need to confirm that last point, of course. In particular, (a) provides an incentive for school systems to graduate all students regardless of whether theyhave mastered the most basic life skills. We would need to find some way to offset this influence.

(5) and (6), we said, are difficult to measure. But are they really? How about if you ask the students? Better yet, ask their parents, who are less likely to be coerced into giving favorable answers. Posing a questionnaire to all parents offers logistical challenges, but no greater thanthe challenges of testing all students. On the positive side, the answers to parent surveys would provide useful feedback that many principals and school boards would use to improve the schools. So, factor in:

  1. Answers to parent surveys.

The methods we have considered so far provide a pretty clear picture of high-schools and school systems in general. However, the only information on lower level schools is provided by (i). I would maintain that if this is all we ever get, it's still better than what we have now – at least it does no harm. I also suspect that if smart people who have studied these issues put their minds together, they can come up with other metrics that are more effective than standardized testing, and do no harm.

If all else fails, we could always open up elementary schools to free-market competition. But that's a subject for another day.

No comments:

Copyright © 1993-2009 by Gerrit Erasmus. All rights reserved.