In 2003, when Parag Pathak was a graduate student at Harvard, he helped design the computer algorithm that matches some 80,000 eighth-graders to more than 400 high schools across New York City each year. Students have the freedom to apply to almost any public high school throughout the five boroughs, and it was a complicated task to factor in students’ preferences with each school’s admissions standards.
More than a dozen years later, working as a professor at Massachusetts Insitute of Technology, Pathak teamed back up with Duke University economist Atila Abdulkadiroglu, with whom he had worked in 2003, to see what their algorithm had wrought. They wondered if the school choice system in New York City was working in practice the way free-market proponents said it should. Since parents are asked to list their top schools and rank them, it was an opportunity to see if families were making choices that would promote competition among schools to improve. Together with two colleagues from the University of California, Berkeley, they asked a question: were parents choosing schools that were the most effective?
The answer: no.
They found that New York City families preferred high schools that enrolled the highest-achieving students, largely measured by eighth-grade test scores, whether or not students learned a lot once they were in the high school. High-achieving peers aren’t always a sign of a good school, though; if a school admits high-scoring eighth-graders, they could still produce impressive high school test scores even if the school’s instruction is lousy. Pathak and his colleagues mathematically teased out how much of student performance at a high school could be attributed to the achievement of the students before they entered, and how much could be attributed to what the school had taught them. The economists found that parents were regularly applying to less effective schools with higher-achieving peers over more effective schools with lower-achieving peers.
Even among schools of equal student caliber, parents weren’t ranking higher the more effective schools that taught kids more.
“If parents put all this weight on peers, rather than effectiveness, then the system creates incentives for schools to screen more and invest in attracting the best students, instead of improving the school,” said Pathak.
For example, schools might spend more money marketing themselves to prospective students or creating attractive logos than refining algebra lesson plans or coaching teachers. In the school-matching algorithm, the more applications a school gets, the more selective it can be in admitting students with higher grades and test scores. So if parents are prioritizing peer achievement and applying to the schools with the best test scores, that creates a snowball effect, where the selective schools become even more popular and harder to get into. What appears to be a good school from the outside, one with high test scores, might just be one that has successfully promoted itself.
“I wouldn’t say this is an indictment of school choice,” said Pathak. “But the argument for school choice is that it creates demand-side pressure for schools to improve. Our paper says, ‘Wait just a second, how reliable are these demand-side signals?’
“Will choice by itself lead to school improvement? You can be skeptical of that from this paper.”
The study is a working paper distributed online by the National Bureau of Economic Research in October 2017. It is part of a growing body of research questioning whether school choice, a priority for the Trump administration, improves schools or gives students a better education. For example, recent studies of Louisiana’s voucher program found that student test scores dropped when parents took advantage of a private school option.
Measuring school effectiveness is tricky. The researchers in the New York City study used sophisticated statistical techniques to compare students’ eighth-grade test scores with their later New York state high school exam and PSAT scores. Schools whose students posted greater-than-expected test score growth were deemed to be more effective. The researchers also looked beyond test scores to high school graduation rates, college enrollment and the quality of the colleges that students eventually attended. As with the test scores, if these data points were higher than what you might predict from the students’ eighth-grade achievement, the high school was considered to be more effective.
Of course, parents don’t have access to these sophisticated statistical techniques, or years of individual student data for each high school. There isn’t a way for a parent to know exactly how effective a school is. Even these economists’ effectiveness ratings for each school aren’t disclosed in the paper, and remain hidden. It’s entirely reasonable for parents to base their application decisions on information they can easily see and understand, such as recent test scores or the opinions of other parents and friends.
Indeed, when the economists first ran the numbers, they initially found that schools with high-achieving students tended to be more effective than those with low-achieving students. At first blush, it would seem that test scores are a good proxy for school quality.
But once they separated peer quality from school quality, a surprising result appeared. Parents showed no preference for the more effective schools when they were ranking schools of equal peer quality. Consider a parent choosing between two schools with identical eighth-grade test scores for incoming ninth graders, but one school produces stronger students at the end. Parents didn’t favor the more effective one it by ranking it higher. This pattern was true for parents of both high-achieving and low-achieving students alike.
One solution would be to give parents effectiveness ratings for each school, but these are often hard to understand and riddled with assumptions that intelligent people could quibble with. It’s not an easy information problem to solve.
Pathak said he has no regrets in helping to build the school-choice algorithm, even if it hasn’t done much to improve schools. School choice predated the algorithm, and before it was introduced in 2004, the city’s admissions process had been a mess that left tens of thousands of students without a high school assignment in the spring. The process is smoother now, if not less stressful for the families going through it.
“It’s frustrating that the choice process becomes a distraction from improving school quality,” said Pathak. “What we need to do is make school better.”
At The Hechinger Report, we publish thoughtful letters from readers that contribute to the ongoing discussion about the education topics we cover. Please read our guidelines for more information. We will not consider letters that do not contain a full name and valid email address. You may submit news tips or ideas here without a full name, but not letters.
By submitting your name, you grant us permission to publish it with your letter. We will never publish your email address. You must fill out all fields to submit a letter.