Measuring achievement gaps between rich and poor might seem like a straightforward exercise for education experts. Simply look up the test scores for rich kids and subtract the tests scores for poor kids. But despite this apparent simplicity, two prominent education researchers have arrived at different answers. Sean Reardon says that achievement gaps have grown a whopping 40 percent in the last 50 years. Eric Hanushek says they haven’t budged.
Reardon, a sociologist, says the growing achievement gaps he has found stem from increasing income inequality in our society and the decisions of many rich parents to invest more in their kids, from private tutors to after-school programs. Hanushek, an economist, believes that the inability to close the achievement gap shows the failure of our education policies to help the poor, especially the $26 billion a year the federal government spends on Title I funding on poor schools and for Head Start preschool programs.
They’re both highly regarded quantitative researchers at Stanford University. Reardon is a professor at the Graduate School of Education. Hanushek is a senior fellow at the Hoover Institution, a politically conservative center for research on economic and social issues based at the university.
Hanushek, who is known for igniting provocative research debates in education, launched this one in March 2019, when he and Paul Petersen of Harvard University released their research and directly challenged Reardon’s assertion that achievement gaps between rich and poor had grown. “[W]e are unable to replicate his finding,” they wrote. Hanushek and Peterson further publicized their controversial findings in the Wall Street Journal, Education Next and at a public event in Washington D.C.. Reardon earned a lot of attention, too, after he discovered a growing rich-poor education gap in 2011. The New York Times covered it and then Reardon himself wrote about it in the Times’s opinion pages. The idea that achievement gaps have grown was largely accepted as fact, especially among journalists, and fit in with a larger narrative about growing income inequality in our society, which isn’t disputed.
Whom should we believe? The short answer is, I’m not sure. It’s possible that they’re both right although Hanushek told me he thinks that’s “unlikely.”
Hanushek and Reardon both agree that rich kids have much higher test scores than poor kids and that’s a problem. But the first step toward solving a problem is to define what’s wrong. What’s clear from this academic squabble is that the quality of data in education is so flawed that the finest minds can’t confidently answer this basic question of whether achievement is getting worse, better or staying the same for poor kids. It’s important to know. Billions of dollars are at stake.
How did Reardon and Hanushek arrive at different answers to the same question? In their studies, the researchers measure wealth and poverty differently and they use different tests to gauge student achievement. Reardon focuses on family income and stitches together different tests over the decades, taken by students at different ages from kindergarten to high school, to document how well rich and poor students are doing. Hanushek has the benefit of looking at just four tests for which he has student scores over a much longer period of time. But instead of family income, he’s created his own index of socioeconomic status based on parent education and possessions in the home, as reported by students on surveys.
Both approaches have shortcomings.
“The advantage of what I did is that income reported by parents is more reliable but I’m using different kinds of tests,” said Reardon, who admits there’s no ideal way to study achievement gaps over long periods of history with existing data. Meanwhile, Hanushek has more consistent test results but has a questionable measure of socioeconomic status that indirectly captures income.
Related: Schools exacerbate the growing achievement gap between rich and poor, a 33-country study finds
Reardon published his study, “The Widening Academic Achievement Gap Between the Rich and the Poor: New Evidence and Possible Explanations,” in 2011 as the fifth chapter in the book, “Whither Opportunity? Rising Inequality, Schools, and Children’s Life Chances.” His data comes from 19 different datasets to cover five decades. One of them tracks a group of students who started kindergarten in 1988 until eighth grade in 2007. And it uses different reading and math tests than this one that begins tracking high school students in 1980. The graphic on this page shows some of the tests that he had to stitch together. Quantitative experts argue over these methods because they say the results can differ depending on what choices Reardon made. “I’m not confident that these very disparate tests Sean uses over time can just be thrown in together,” Hanushek said in an interview.
Family income isn’t straightforward either. It’s based on parent responses in these national surveys and they might not always be accurate. Sometimes it’s based on responses from students who might be guessing how much their parents make. (Hanushek also questions how Reardon decides exactly which students to include at the top 90th percentile when multiple-choice survey questions don’t break down income brackets finely enough.)
Then Reardon compared test scores of a child born in a family at the 90th percentile of the income scale (top 10 percent), which began at an annual income of $160,000 in 2008, with a child at the bottom 10 percent, which started at an annual income of $17,500 the same year. Reardon found that the gaps in test scores between these two extremes grew the most for students born in the 25 years between the mid 1970s and 2000 — by roughly 40 to 50 percent. That’s big.
In Hanushek’s analysis, “The Unwavering SES Achievement Gap: Trends in U.S. Student Performance,” he teamed up with three other researchers: Paul Peterson at Harvard, Laura Talpey at Stanford and Ludger Woessmann at the University of Munich. They looked at U.S. scores on two international tests (TIMSS and PISA) and two domestic tests (NAEP and Long-term Trend NAEP). They concentrated on the tests taken by students between the ages of 13 and 15. The four tests don’t cover the entire 50-year time period. PISA, for example, wasn’t created until 2000. So there’s a bit of complex mathematical work to average the tests together.
The more complicated part is that these tests don’t tell you about the family incomes of the students who take them. The researchers had to figure it out from surveys the students filled out about their homes, such as the number of books they have, and about their parents’ education. From this, the researchers created their own socioeconomic index. Usually, socioeconomic status is a composite of income, education and occupational prestige that tends, for example, to equate a lower paid professor with a higher paid real estate investor who has fewer years of education. But information on parents’ jobs wasn’t available for all the tests, so the researchers had to ignore that component.
It’s unclear how accurately possessions in the home measure wealth and poverty. In 1999, students on the TIMSS test were asked if they owned a “Walkman” (remember those portable cassette players?) or a telephone answering machine. In 2015, students were asked if they had their own cell phone and an internet connection. Nowadays, very poor kids sometimes have cell phones and computer games. Sometimes highly educated families restrict technology. It’s possible that a kid who really is at the bottom 10 percent is getting mischaracterized as higher status and vice versa.
But it’s also possible that this isn’t a debate about data and methodology at all.
I talked with Anna Chmielewski, who used to be Sean Reardon’s research assistant at Stanford and worked on Reardon’s 2011 study. She’s now an assistant professor at the University of Toronto’s Ontario Institute for Studies in Education. She just published a study, “The Global Increase in the Socioeconomic Achievement Gap, 1964 to 2015,” in the American Sociological Review, a peer-reviewed journal, in May 2019. She looked at the same international test data that Hanushek did but applied Reardon’s quantitative methods. Like Hanushek, she found that achievement gaps between high socioeconomic status and low socioeconomic status kids have not been increasing in the United States. (The United States proved to be an exception to the world-wide trend of increasing achievement gaps.) On the TIMSS test, U.S. achievement gaps have been constant, she found. And on the PISA test, the gap between the top and the bottom has actually been narrowing.
But Chmielewski says this doesn’t prove that Reardon’s 2011 study was wrong. When she was working on that earlier paper, she told me, she also ran the numbers to see how academic achievement had changed based on parent education instead of income. She found that those achievement gaps hadn’t changed much. The very same test data that showed a mushrooming gap between rich and poor simultaneously showed that the gap between kids of highly educated parents and kids of uneducated parents had increased only slightly. At the time, it didn’t seem like a big deal since their focus was income, and that parent-education finding was set aside in an appendix, Chmielewski said.
What Chmielewski is discovering is a big deal for researchers in education. There might be a real difference between income and socioeconomic status. Perhaps they cannot be used as interchangeably as they traditionally have been in research.
One theory is this: families at the high end of the income scale, at the 90th percentile, are investing in their kids’ educations more than they used to decades ago. In other words, each dollar of parent income is “buying” you more points in test scores than it did in the past. A socioeconomic index that gives a heavy weight to parent education doesn’t capture that. Perhaps test scores at the 90th percentile of the income distribution are rising more rapidly than test scores at the 90th percentile of socioeconomic status.
It’s also unclear how increases in college education are affecting socioeconomic status. Many more Americans are getting a college education today. Maybe there’s a new generation of college-educated parents who aren’t dramatically changing the way that they parent at home. Perhaps they’re not reading to their kids as much or engaging in the kinds of dinner conversations that college-educated parents typically used to in the 1970s. That could affect academic outcomes too, and show lower test scores for kids whose families have risen up the socioeconomic ladder.
“It raises a lot of questions,” said Chmielewski. Until some of those questions are answered, the debate will remain unresolved.
This story about the achievement gap in education was written by Jill Barshay and produced by The Hechinger Report, a nonprofit, independent news organization focused on inequality and innovation in education. Sign up for the Hechinger newsletter.
At The Hechinger Report, we publish thoughtful letters from readers that contribute to the ongoing discussion about the education topics we cover. Please read our guidelines for more information. We will not consider letters that do not contain a full name and valid email address. You may submit news tips or ideas here without a full name, but not letters.
By submitting your name, you grant us permission to publish it with your letter. We will never publish your email address. You must fill out all fields to submit a letter.