Recently watching the NBA playoffs there was the jarring experience of back-to-back apology ads from Facebook and Wells Fargo. The theme running through both was: “We optimized for the wrong metrics! Oops!”
In Facebook’s case, they optimized for time spent in their app. But because we don’t post enough baby pictures, they filled our feeds with publisher content. Unfortunately the signal used to determine the reliability of publisher content Facebook chose was “engagement” rather than"truth.”
Startups and other businesses looking for intel on the school marketplace should check out EdWeek Market Brief, a service that provides news, analysis, and original data on the ever-changing needs and priorities of schools.
So you take all that and add in the motivation to increase the value of Facebook by sharing our data with anyone who might find value in it and that’s how you end up testifying to Congress.
For Wells Fargo, the issue was a result of an even more straightforward analysis: the more accounts we have the more money we make so let’s incentivize the opening of new accounts. Not so different from the model-driven disaster of 2008 (see The Big Short): “Home prices never go down, right?” Oops.
Bringing it back to education specifically, one comment I’ve heard from a lot of district leaders recently is “Well, we have this app and it claims to have some scientifically-proven predictive validity...but we just don’t trust it.” As we move from algorithmically-driven decision-making to artificial intelligence and machine-learning solutions, the “decision-making process” of the code will only become more inscrutable. An algorithm has a hand-coded set of human-readable logic statements (think “if this, then that”). But the innards of a neural network are just mathematical transforms that nobody can translate to helpful English sentences.
Meanwhile, the first thing a user wants to know upon receiving your big-data generated “prediction” is “How’d you come up with that?” If the best answer we’re giving is “science!” then you’ve skipped the step where a human interprets your notification and knows what to do with it. On the other hand, if a teacher has a limited scope and sequence and gives a quiz they expect the students to be able to master based on content they taught, then guess what? The little data that comes out of that assessment is going to be a lot more intuitive and actionable for that teacher.
You can optimize your metrics all you want but if humans can’t take helpful action as a result, you’re not going to end up where you hoped.
Follow EdWeek Market Brief on Twitter @EdMarketBrief or connect with us on LinkedIn.
See also:
