One of the most popular English lessons in the instructional marketplace Teachers Pay Teachers is a unit on how to teach William Shakespeare’s “Romeo and Juliet.” It costs only $14.99 and claims to explain how to teach the tragedy of the star-crossed lovers in a fun-filled way while hitting almost 50 Common Core standards in five weeks. More than 1,200 Yelp-style reviews from teachers are posted. Many gush like this one: “Everything is clearly laid out and takes the guesswork out of trying to feel like I have to do it all. Such a lifesaver!”
There’s just one problem. A curriculum expert who reviewed the pedagogy explained to me that the unit was “weak” and does not meet Common Core standards as the creator claimed. The one exception was the essay prompts, which the reviewer rated as good. But otherwise, instruction in this Shakespeare unit rarely goes deeper than a surface understanding of the text. It doesn’t push students to delve into the themes or how characters drive the plot. Too often, students are encouraged to think about how aspects of the story relate to their own lives instead of analyzing Shakespeare’s play, the reviewer said. To understand Juliet’s line, “A rose by any other name would smell as sweet,” students are asked about how names matter in our lives. “If you had a different first name, would your life have been different?”
“Playing a game of insults, that’s all well and good,” said Jennifer Dean, an educational consultant and former English teacher who has a Ph.D in English literature and was the lead reviewer in a December 2019 study of online instructional materials. “But you have to have the excitement of digging into the text. That’s what’s missing here….And it pings from sonnets to tone to character without any logic.”
This Romeo and Juliet unit isn’t an outlier, according to the expert reviewers in the study. Five reviewers rated most of the online materials that teachers download as “mediocre” or “probably not worth using.” Specifically, they studied 328 of the most downloaded units and lessons in high school English on three websites (Teachers Pay Teachers, ReadWriteThink and Share My Lesson) and rated their quality in ten areas, such as building students’ content knowledge and how cognitively demanding it is. They found that 64 percent of them “should not be used” or were “probably not worth using.” A majority of the materials were rated 0 or 1 on a 0-3 quality scale. Lessons were often missing explanations for how to use the materials in the classroom or how to differentiate instruction for struggling students.
“Virtually all teachers supplement to some degree,” said Morgan Polikoff, an associate professor at the University of Southern California who co-authored the study with Dean. “This is a huge phenomenon that is woefully understudied. This first study certainly casts doubt on the content of these lessons in high school English. There’s a lot of not very good stuff out there. And it’s going to require a lot of work to identify what’s good.”
Related: Scientific evidence on how to teach writing is slim
“We need to do more to get quality materials in teachers’ hands,” added Polikoff.
Polikoff and Dean’s study, “The Supplemental Curriculum Bazaar: Is What’s Online Any Good?” was published on Dec. 10, 2019 by the Thomas B. Fordham Institute, a conservative think tank. This research was funded by the Chan Zuckerberg Initiative and the Thomas B. Fordham Foundation. The Chan Zuckerberg Initiative is among the many funders of The Hechinger Report.
The online world of supplemental teaching materials is enormous and the researchers started by limiting the universe to a particular subject and a narrow range of grades. Polikoff focused on high school English because textbooks are uncommon and teachers are often on their own as they select which novels to teach. Reviewers only analyzed complete lessons or multi-lesson units. Individual worksheets or games were excluded because they cannot be expected to fulfill so many educational goals.
Some of the quality ratings are subjective — just as movie critics have different opinions. Reasonable people can disagree. Even some of the categories are debatable. Should you ding a lesson because it doesn’t include diverse authors or cover culturally diverse topics? These reviewers did. But there were enough categories that a lesson wouldn’t be classified as mediocre for shortcomings in only one or two areas.
Not all the lessons were disappointing to the expert reviewers. Dean pointed me to another Romeo and Juliet lesson on Share My Lesson that wasn’t as “snazzy and beautiful” as the Teachers Pay Teachers one described above but was instructionally complete, she said. “Here you have themes, conflicts, figurative language,” Dean said. “It’s progressing in a thoughtful, logical way, lesson to lesson.” The unit begins with a lesson on the Hatfields and the McCoys so that students can see parallels with the Capulets and the Montagues in later weeks. “This amazing lesson is free,” Dean said. “The other amazing thing is that an amazing lesson is rare.”
Teachers Pay Teachers is one of the most popular sites for all kinds of materials. More than 80 percent of U.S. teachers use it, according to the company. (According to a 2017 federal survey of teachers, more than half — millions of teachers — were using materials from Teachers Pay Teachers every week.) But the other two giants where teachers seek materials, Pinterest and Google, were conspicuously missing from the study. The researchers excluded those sites, they explained, because they couldn’t identify which lessons are intended for high school English and they were unable to sort lessons by downloads.
Polikoff hopes to extend this analysis beyond the narrow slice of high school English to other subjects and grades. If the findings are replicated in future studies, it calls into question the effectiveness of online marketplaces for teaching materials and if we can rely on crowdsourcing to spot quality. It was surprising to me that even the nonprofit site ReadWriteThink, which carefully approves who can post lessons and insists that the lessons be peer reviewed, had such uneven quality. ReadWriteThink is a joint venture between the International Literacy Association and the National Council of Teachers of English. Share My Lesson, also a nonprofit, is run by the national teacher’s union, the American Federation of Teachers, and allows any teacher with an account to post materials just like Teachers Pay Teachers.
Related: The ‘dirty secret’ about educational innovation
Why is the market dysfunctional? Part of the problem is an excess of choice. More than four million items are on the Teachers Pay Teachers site, according to the for-profit company. How is a teacher to sort through them and choose? The star ratings aren’t much of a guide. Polikoff told me that the average star rating of the materials he downloaded was 3.98 stars out of 4. “That really flies in the face of the idea that somehow this market is going to sort of self correct in a way that results in the best materials rising to the top,” Polikoff said.
Dean, the lead reviewer, suspects that many teachers are lured by slick marketing. “Presentations can be dazzling,” she said. “It leads teachers astray because it looks so good.”
Many teachers might also genuinely think that a lesson is aligned with Common Core standards when it isn’t. A 2018 RAND study documented how a majority of teachers give the wrong answer to questions about what the Common Core standards are asking for.
“The market is breaking down, I think, because teachers are not generally experts in creating curriculum materials or maybe even identifying good curriculum materials,” said Polikoff. “I don’t view that as a failing of them as teachers or as humans. I view that as a failure of… their teacher education programs and their in-service professional development programs in the school districts that they work in.”
Polikoff says the burden shouldn’t be on teachers to sort through thousands of lessons. He suggests that districts and states, and even textbook publishers, could get in the business of recommending specific supplemental materials that are good.
In a written statement, Teachers Pay Teachers asserted that teachers are finding “high quality, standards aligned materials that support effective instruction” on its site, but says it plans to revamp its ratings and review system in January 2020. “Improving our ratings and reviews system has been a major companywide priority,” a company spokeswoman wrote me. The company will allow users to give feedback on the accuracy of the standards tagging, and it will ask new questions, such as whether the material is at the appropriate level of difficulty.
Related: 3 lessons from data on how students are actually using educational apps and software at school
Share My Lesson says it goes beyond crowdsourcing to indicate which lessons are high quality. It gives a green checkmark to lessons that are vetted by a cadre of expert teachers. Still, many popular lessons don’t have those green checks. (I reached out to ReadWriteThink, but didn’t hear back from the organization before deadline.)
As part of the study, researchers interviewed teachers who told them that the ability to download lessons online is a welcome time saver. I wondered if students are worse or better off now that teachers have access to a vast marketplace of lessons of questionable quality. In previous decades, English teachers had to spend hours developing lessons themselves. Probably some of those self-created lessons were good and some weren’t. Even though the average quality of materials online isn’t great, that doesn’t necessarily mean that they’re worse than what teachers otherwise would have taught to students. Answering this question would be a great area for future research.
This story about Teachers Pay Teachers was written by Jill Barshay and produced by The Hechinger Report, a nonprofit, independent news organization focused on inequality and innovation in education. Sign up for the Hechinger newsletter.
At The Hechinger Report, we publish thoughtful letters from readers that contribute to the ongoing discussion about the education topics we cover. Please read our guidelines for more information. We will not consider letters that do not contain a full name and valid email address. You may submit news tips or ideas here without a full name, but not letters.
By submitting your name, you grant us permission to publish it with your letter. We will never publish your email address. You must fill out all fields to submit a letter.