2012 (1): New Years’ On A Rational Planet
I had planned to post this before new year, but was too busy doing actual research (how gauche). But like Christmas decorations and turkey leftovers, year-end predictions posts ought not to be showing up on January 5th. So let’s pretend this ISN’T the first in a series of ten (ten!) such things.
Even if I WAS about to write a predictions post, I would be using this second paragraph for caveats. In the ever-rolling sea of market research I am but one small coracle, and I tend to follow my nose after things which interest me, rather than sit on high and worry about the fate of the industry. And where better to start than on something I am almost completely uninformed about (I am hardly alone in this). Yes, it’s “Big Data” time!
Everything in research and marketing now is happening against the backdrop of encroaching “big data” – though this has been true for a few years now. What ought to be occurring at the moment is a shift away from swooning over the idea of big data and an interest in embracing its practicalities. But actually this isn’t really going on, partly because for non-specialists like us getting to grips with business intelligence concepts can be like reading Martian – let alone trying to use the software. All I can really offer is a read on “big data” as a cultural concept within research and business – big data as meme, if you like.
So in the market research industry “big data” operates in two quasi-mystical ways. One is as a singularity – a point not too far off at which everything about consumers can be known and used (there’s the crucial bit). The other is as a bogeyman – big data as a school of superhuman insight ninjitsu which will make us obsolete unless we researchers master its skills, which to be absolutely frank we’re not best placed to do. (An aside: “Learning how to analyse big data” has become the new “learning Chinese” – yes, we probably ought to do it; yes, it’s really bloody difficult.)
“Any sufficiently enormous dataset is indistinguishable from reality”
One of the most interesting things I read about big data in 2011 was to do with its density. The idea is that small data – like surveys – is high-density. It’s useful and manipulable because it’s relatively complete – you have a limited number of defined variables which are complete across a very limited number of cases. As datasets grow, they also begin to melt – the number of variables increases, the importance of each variable becomes harder to define, the gaps within the cases grow and this happens more and more the less structured data becomes. Ultra-unstructured data is basically gaseous. Now, I’m not very knowledgeable about business intelligence databases but this does ring true for the unstructured information (web text, social media, etc.) I have worked with, and I think it’s notable that hyped intelligence providers like Palantir are about fusing datasets as much as using them – increasing data’s density.
“Does my data look big in this?”
The other most interesting thing I read about big data last year was Jay Owens’ provocative insight that “big data” is the corporate equivalent of “central planning”, that dream of Western government in the 60s and 70s and its eventual nemesis. This rings true not so much on a “how it’s done” level as on a psychological one – both of them are dreams of control via understanding and model-building, “fantasies of rationality” as the Wall Street Journal perceptively put it. Except now these fantasies are coming true? Well, maybe: what struck me about the WSJ piece is how, well, drab its chosen success story was. A frozen food company builds an analytics system which allows it to squeeze 3-4% more sales from cross-selling. This was doubtless enormously welcome for said company but it’s not the thing legends are made of – a far cry from the invocations of Asimov’s fictional “psychohistory”, an infallible predictive mathematics of human behaviour.
What it inadvertently reveals is a future of warring algorithms all dedicated to making these incremental improvements in sales and logistics at the margin: big data as a twitchy nervous system for business, specialising not in insight but in endlessly reflexive micro-managing, more ant colony than agile. Even when the problems of data fusion, density, noise etc. are solved, in the B2C world all said algorithms are still competing for a sometimes growing, sometimes stagnant, but (at any given moment) finite resource – the money in your pocket. Is there a use in this future for research as a provider of wider (but vaguer) insights – ideas that might spark a counter-intuitive but profitable tweak to the model? We shall see.
Next: Cyborg Consumers, and why individuality is for suckers
