Saturday, February 22, 2014

Bayes's Theorem and Hiring Your Way To Great Teachers

Bayes' Theorem is the most important statistics theorem that nobody knows.  In essence, it says that when computing the probability of an event occurring, you must take into account the information you already know about what theoretical outcomes are actually possible.  In mathematical terms:  the probability of A given that B has occurred equals the probability that both occur, divided by the probability that B occurs (with or without A).

Bayes's theorem has wide applications in statistical inference, but today we're going to talk about one that's of crucial concern to anyone trying to improve teaching in a particular school:  hiring teachers.  Suppose that your interview protocol allows you to identify satisfactory-or-better teachers with 80% accuracy (if you have such a protocol, tell me!).  And suppose that 80% of the teachers in the applicant pool are satisfactory-or-better.  Then if you interview 100 teachers, here's what happens:

  • 80 teachers are satisfactory-or-better.  Of these, your interview protocol says that 80%, or 64, really are satisfactory-or-better, and 20%, or 16, it rates as unsatisfactory.
  • 20 teachers are unsatisfactory.  Of these, your interview protocol says that 80%, or 16, are unsatisfactory, and 20%, or 4, are inaccurately rated as satisfactory-or-better.
What's the takeaway?  Well, there are 68 teachers rated as satisfactory-or-better, but of these, 4 are actually unsatisfactory.  Thus if you hire one of the 68 teachers, you have about a 94% chance of getting a satisfactory-or-better teacher.

So far, the odds sound pretty good.  But those odds are highly dependent on the assumptions we made: that 80% of the candidates were satisfactory-or-better, and that your protocol helps you tell good from bad 80% of the time.  In my experience, neither of these things is necessarily true:
  • Many of the satisfactory-or-better teachers are pretty happy where they are, and aren't looking for new jobs.  In my experience, the applicant pool for Chicago Public Schools jobs is more like 40% satisfactory-or-better.  (Note: I'm not, not, not saying that only 40% of CPS teachers are satisfactory.  There are lots of satisfactory-or-better CPS teachers--but in my experience, many of them are committed to the schools where they teach.  What I'm saying is that when I was interviewing applicants for jobs at my school, only 40% of the applicants were satisfactory-or-better.)
  • Teachers can have great credentials and interview well without being great in the classroom.  You can catch that with demo lessons (I've known of candidates who hit home runs in the interview only to totally whiff the demo lesson), but even that won't tell you how well they relate to students and parents over the long-term, how well they collaborate with colleagues, and how committed they really are to improving their practice.  So my guess is that interview protocols are less than 80% reliable, say 70%.
Running that same thought-experiment with our revised assumptions yields very different results:
  • Only 40 applicants are satisfactory-or-better, and our protocol identifies 70%, or 28, of them as such.  12 applicants are incorrectly identified as unsatisfactory.
  • 60 applicants are unsatisfactory, and our protocol identifies 70%, or 42, of them as such.  18 applicants are incorrectly identified as satisfactory.
Thus 46 applicants are identified as satisfactory-or-better, of whom only 28 really are satisfactory.  So the probability that a given applicant who passes the interview process is actually satisfactory is 28/46, or about 61%.  Under these conditions, then, if you hire five candidates, only three will probably work out.  And you'll be stuck with two whom you sort of wish you didn't hire.  

What are some conclusions we can draw?
  • The usefulness of hiring procedures has as much to do with the overall quality of the applicant pool as it does with the theoretical reliability of the procedure itself.  If the pool has lots of unsatisfactory teachers, even a good test will end up with many of the apparently-satisfactory teachers being actually unsatisfactory.
  • If you don't have a great applicant pool, firing a teacher who isn't working out will only result in a substantial improvement 60% of the time.  That's better than nothing, but a whole lot less than the "Just fire all the bad teachers" voices usually let on.
  • If lots of bad teachers are suddenly fired, the applicant pool will get worse, both because the fired teachers are now in it, and because lots of people are trying to hire the good ones (remember that 3/5 of the teachers we hire, even under the pessimistic assumption, are good teachers).  And then as we've seen, hiring procedures become less effective at securing satisfactory teachers for jobs.  So as a system-wide policy, "fire the bad teachers" is unlikely to produce substantial improvements for a large fraction (probably more than half) of the kids in the system.
So it's unlikely that we can hire--or fire--our way to great teachers.  We need to take the teachers we already have and develop them instead.