If, like me, you’ve disliked Klout for a while, this article may give you a warm glow - assuming you can beat back yr tears for the professional bloggers who’d staked their reputations on a “social influence” service.
But the logic of the piece is worth digging into because it has implications for the research business too - we are (quite often) also an industry which relies for backup on accumulated norms, and particular ways of measuring things.
The writer at first is cross with Klout because its score readjustment meant that people’s scores dropped and they suffered for it. But at no point does she say the score readjustment got things wrong - in fact, as she describes it, it solved a big problem with people gaming the metric. (Perhaps some of the people whose scores dropped had also gamed the metric - heaven forbid!)
But even so she claims Klout’s credibility is shot, because making this adjustment means it can’t be treated as a standard. In other words, the problem wasn’t its inaccuracy, it was that it admitted its inaccuracy and worked to improve it.
Now, I can’t get too upset about the widespread Klout backlash - on every level, from concept through implementation to small print, I’ve had problems with the site. But this has implications for what we do. Imagine you’d created a better way of running surveys, or measuring ad effectiveness. Now imagine you were trying to get this way adopted by people who’d been working for a long time with an existing system. The AdAge logic suggests - and reality confirms - that you face an uphill struggle. For all the rhetoric of an industry in constant change, existing datasets have a massive incumbent advantage.
Standard metrics become standards not necessarily because they’re better but because their accumulated data means that shifting away from them would cause huge problems for people who’ve embedded them into their businesses. “Normification”, I called it in a tweet, setting it against - yes - “Gamification”, which was the inspiration for one of my examples above. My response to the problem was that if you could demonstrate a really good reason for shifting methods and abandoning old datasets you should be able to overcome this.
But Normification is a tenacious enemy: like many an embedded power structure it would rather tear the playhouse down than leave quietly. What this article shows is that even if a change solves a real problem and creates an apparently better solution it may not be accepted. Rather than think “I’ve lost score but the algorithm is better so Klout will be more useful” (what the company wanted) people have thought “I’ve lost score so Klout was always junk”. Which is probably true. But it raises a nasty spectre for research: what if the response to a kind of survey which is better, but changes the data, isn’t “Great, this will help our data get better” but “Surveys were probably junk anyway”?
