Testing and the future of education: Anya Kamenetz on Future Trends Forum #6

Anya Kamenetz was our Future Trends Forum guest on March 15th, and she sparked one of our most energetic discussions to date, focused on testing.  Questions and comments raced over Shindig, and live tweeting was vigorous.

We’ve made the full recording available as usual, via this direct link or the following embed:

Inspired by her recent book, The Test, I began the conversation by asking how testing may change over the next few years, according to what the author learned from researching its history.

Rachel M. poses a question for Anya K.

1. BRYAN AND ANYA DISCUSS

Anya responded by arguing that testing has always been important, both face to face and online, adding that self-criticism is important to learning. Now, data-driven testing is enormous and growing, which gives rise to important questions: who owns testing data? what happens to user privacy and data?

I was struck by Anya’s next point: these conversations about testing are a prelude to a larger conversation.  Will tests reward and punish students, or lead to better living through algorithms?

Additionally, Anya wanted us to reflect on what we are not capturing through testing.  What happens to the social, emotional, human, qualitative aspects of learning?

As one anecdote to consider, Kamenetz mentioned a new product, which provides surveillance software for K-12.  The tool gathers data down to the level of a student’s individual private browsing and search history, both on school premises and elsewhere – i.e., at home.  The company making this software claimed as an achievement to have prevented three suicides by detecting self-harm signals through pupils’ digital activity.

This example surfaces new questions about student data and privacy, such as: are administrators mandatory reporters?  Because we can know everything, ultimately, should we?  As Rodney Hargis tweeted, paraphrasing Spiderman, “[w]ith great data comes great responsibility”. For a parallel I mentioned antiterrorism technologies, some of which also take data gathering to new heights (or depths).

Anya contrasted this data-gathering by authorities for monitoring purposes with the idea that education is about learners going through the process of self-authoring.  While a school keeps something on each student, those pupils are empowered with the artifacts made along the way of learning: projects, papers, reports, other materials produced, a transcript.

Some of this self-authoring comes through qualitative development, which is harder to measure.  Indeed, Anya stated they educators are “fumbling” on qualitative assessment so far.  Testing tools aren’t very good now.  Instead, “[w]e have to fall back desperately on human judgement.”  She cited the recent Google artificial intelligence success AlphaGo; I recommended this Wired article.

So what is it that humans do well?  One answer:

We are coming to the realization that non-quantitative measurement is important.

I asked about other countries, in case any have done better at this than the United States.  Anya responded that America might have a unique mania for measurement, including adding economic value to qualitative attributes.

Looking ahead, I asked if the new secretary of education will change the testing environment.  Anya responded that the recent authorization of the ESSA law kept testing in place, but granted flexibility on stakes assigned to tested.  In testimony John King has stated that testing has become redundant to some degree, and also taken some of the joy out of school.  Overall, Anya continued, “[t]here are very few defenders of the status quo” in testing.

However, reforming testing is like reforming the tax code; it’s easy to discuss, but paring it down is, in reality, very difficult.  There is funding in ESSA to allow the auditing of states for testing redundancy and testing quality issues, so some states could take advantage of this to reduce testing.  Anya offered an anecdote, that some district-level benchmark tests are starting to get cut back by using software-based assessments.

At this point we encouraged the audience to mingle and converse with each other.  Anya and I removed ourselves from the Shindig “stage”.

2. THE PARTICIPANTS TAKE OVER

Participants had planety of questions, which I’ll summarize, paraphrase, or quote outright.

Q: What’s happening with competency-based education (CBE) in terms of testing?

A: At worst it amounts to a new bunch of computer-based tests, and boxes to check off.  To be done well, CBE needs more work – more people, more time?

It’s not easy to assign resources for this.  One example: eportfolios dropped in one case because doing them decently was seen as too onerous.  Anya asked us instead to see eportfolios as integrated into learning, and saving time from testing.

Q: Preston recommended that community colleges need a better testing system in order to find out what students actually know.  They could then use that tool to stage interventions.

A: Remedial eduction is often a stumbling block, with placement into it often sapping a student’s chances of getting a degree.  Community colleges are working with comparatively few resource.  This area needs more attention.

Q: Joe Cornelli noted that open source projects are open to a very wide peer review system.  This works for both coding and also content (example: Wikipedia).

A: What about calibrated peer review, matching grades to those of a trained assessor?  This works by establishing a baseline.  In order to get a grade for yourself you first need to grade five others.  This works in a low-stakes settings, like a MOOC.

Q: Rachel M wanted to know what Anya thinks about badging.

A: Badging needs to work within a recognized brand or community.  It won’t succeed in a  decentralized ecosystem.  It needs a central authority such as LinkedIn, for example..  “Badges don’t travel alone.”

Rachel wondered if universities would issue badges, acting as that kind of authority.

Q: George S asked how does community college remedial education differ from public state university remedial education? 40-60% of new first-year students at California State University campuses take at least 1 remedial course.

A: Remedial classes are more common at community colleges.

Q: Ellen Borkowski: will we see schools linking test outcomes to teacher evaluations in Common Core states?

A: That’s actually very, very hard to do.  There isn’t a good match of data to outcomes.

Q: Mjcraw asked about feedback in testing.

A:  Can you extract something from the feedback process to tell us more about learning?  Examples of this being done include leaderboards; some software.  We can shift from snapshots of learning to a graph, focusing on trends over time.  What’s missing is linking those results to established measurements.

Q: What’s about Campbell’s Law?

A: cites Goodhart, to the effect that “when a measure becomes a target, it ceases to become a good measure.”  We distort the education process if everything becomes a score.  We need to keep updating testing.

Q: Pamela inquired about states or districts using funds for a more holistic approach to education like ASCD’s Whole Child initiative?

A: It’s good to see more experiments.

Q: Brett Boessen asked about  Quest to Learn? How is that perceived outside of games-based learning folks?

A: It’s very influential.  QtL people are highly connected within the worlds of informal learning, badging, and project-based learning.  They also influenced GlassLab (same origin in Institute of Play).

BOTH IN BOOK

Q: Matthew asked if Anya was familiar with Todd Rose and The End of Average. His research indicates there is no average and we should end standardization.

A: interviewed him on NPR.  Anya finds it to be an important book.  Rose also started an organization to really do individual learning.

I asked for any parting shots, and Anya told us that it is important to unite the worlds of K-12 and higher education.

During post-discussion discussion Mark Wilson mentioned an ecological fallacy, wherein systems become too complex for decent prediction.  In these cases aggregated data doesn’t yield good predictions.  You can identify macro trends, but can’t do this at the individual level.  As an example he cites this fascinating talk about predictive policing.

Thanks to Anya and participants for a very energetic and richly informative Forum!

Liked it? Take a second to support Bryan Alexander on Patreon!