101 Uses For A Dead Discipline
Yesterday Faris Yakob put up a blog post, “All Research Is Wrong”. It’s worth reading partly because - like all the best blogs - there’s a lot of meat in the comments section (and Faris - like all the best bloggers - gives as good as he gets). The arguments in the main post aren’t unfamiliar and aren’t wrong: we don’t know why we do what we do, and there’s a huge gap between what people say they do and what they actually do.
Remarkably enough, market researchers are already aware of these things, and react with a certain amount of eye-rolling whenever we see them presented as astonishing revelations. When I was but a tiny researcher I remember learning how an awful lot of the techniques we use are attempts to get beyond direct questioning and towards deeper “truths” for exactly this reason.
Well, alright, when I say “truths”, I mean “norms”, but this is part of the point: when you’ve been asking people stuff for 60 years, and have been comparing those answers to behaviour for just as long, you have - as an industry - a pretty good notion of which answers match behaviour, which correlate to it, and which are completely batshit. Remember, a lot of the time - even in quant research - what you’re interested in is the comparison between two figures, which means that the relationship of those figures with reality doesn’t matter, as long as both have the same error.
So Faris’ conclusion is quite right: market research isn’t really trying to measure “what people do” or “what people think”, it’s constructing a model of what people say they do or think which we can then compare to actual behaviour. The more we run the model, and the more behaviour we have to compare it against, the better it ought to get. The straw man of market research Faris is building only really works if you assume MR never learns from its mistakes and treats every fresh piece of survey data as gospel truth.
Now admittedly, we don’t learn from them as much as we OUGHT to, but we’re not completely pigheaded. Most commercially available data is weighted - not just by census data, but by norms, by client data if available, and so on. The “triangulation” Faris talks about - the wisdom of the research ancestors - is baked in. Not as much as it could be, goodness knows, not as much as it needs to be in an era of real-time research and data glut. But the idea that research simply takes people’s self-reporting at face value and goes “gosh! how interesting” is off-base. (This is the real issue with DIY research, it seems to me: not just the survey design, but the depth of experience and knowledge needed to relate the results to behaviour and draw inferences from them.)
