I came across an interesting article in the NY Times yesterday. It concerned teacher ratings in schools. I’m not that familiar with how the system works, but it looks to be based on pupil performance pre and post the school year. Teachers are rated on the differential between beginning-of-year and end-of-year scores adjusted for a host of other factors (race, background, area, etc.) that also affect student performance.
On the face of it, it seems like a performance metric that is unquestionably accurate – if student’s don’t improve their test scores the teacher obviously hasn’t done their job.
That is until you read this quote from a NY Principle:
“If I thought they gave accurate information, I would take them more seriously,” the principal of P.S. 321, Elizabeth Phillips, said about the rankings. “But some of my best teachers have the absolute worst scores,” she said, adding that she had based her assessment of those teachers on “classroom observations, talking to the children and the number of parents begging me to put their kids in their classes.”
As it turns out there are not only a slew of data issues (accuracy and reliable collection and attribution), the correlation of how well a teacher performs in one year compared to the next is a miserly 0.3.
0.3. Let’s put that in perspective. That means that if you out-perform in one year – so you’re a great teacher and all of your kids improve out of sight – in the next year, you only have a 0.3 chance to be rated the same! Unless in the school break teachers regularly binge drink themselves into a permanent skill reducing stupor (no doubt it’s happened), that’s an awful record for a performance metric!
Good teachers, you would think, should consistently out-perform, and bad teachers should consistently, well, be bad.
While this is how I remember my old teachers in school (the good ones tended to stay good), it turns out that statistically, this is very difficult to prove.
But as the article goes on to point out, shoddy correlations between performance metrics and performance are the norm, not the exception. SAT scores correlate to college performance at a 0.35 rate. Between season correlations of professional ball player batting averages is only 0.36. In fact, the study cites a meta-study that found most complex performance based metrics fall between 0.33 and 0.40.
Rather than being astounded by this fact, I find it refreshing. It corroborates much of what I have seen regarding business metrics in the corporate world.
When you get outside of the realm of finance and operations with simple relationships between things like profit, costs, revenue, throughput, etc. and you get into the realm of return on innovation, marketing, customer service, it gets harder to predict outcomes.
That’s because these ‘softer’ measures of business represent more, not less, complex systems. The problem with measuring them isn’t a dearth of data, it’s out inability to understand the myriad of interrelationships.
Business advertising like this drums home the idea that the more information we collect, the more we know. It’s a false idea fast being perpetuated by our information crazy world.
It’s the idea that if we can describe it enough, and in enough detail, we can understand it. Ask any really good teacher why this just doesn’t work.
