The Technique That Should Not Be
On Twitter, @researchrants just posted some amazing screenshots from someone’s attempt to do an online survey using Magnitude Estimation (here’s the introductory one). I hadn’t seen a Mag.Est since the late 90s, when I was new to research, and I certainly haven’t seen anyone optimistic enough to try it online.
Magnitude Estimation is a glorious example of a research technique which would be absolutely terrific if it wasn’t for those cussed respondents. My trainers would discuss it with respect and sadness, a catch of a sigh in their voice as they covered its beauty, its elegance, its simple circumventing of all the issues which hinder rating scales. And yet I barely ever saw one deployed.
Like conjoint, Mag.Est is a method for measuring attribute preference. Unlike conjoint, it asks people directly what attributes they’d prefer, rather than figuring it out by making them trade off packages against one another.
The problem it’s designed to solve is that if asked, say, “What would you like in a car?”, people tend to tick everything. So you make them rank the attributes, right? But no! Because then you’re imposing an arbitrary value on each preference rank. If their top attribute (it has wheels) is a hundred times more important to them than the second attribute (comes with a nodding dog) you have no means of finding that out.
But with Mag.Est you do! You take an arbitrary attribute and give it a score of 100, and then ask people to score everything else relative to that. Like an attribute 10 times as much? Score it 1000. Like it one-tenth as much? Score it 10. It’s gorgeous, isn’t it? You get ranking AND relative preference with one question.
The only problem is that it is quite BOGGLINGLY unintuitive to explain and administer, because you’re forcing people to essentially use logarithmic, open-ended scales, which they are utterly unfamiliar with. Even in the era of Face To Face interviewing Mag.Est. was really hardcore, and if you included it you needed to make damn sure you had really good interviewers (so, er, nobody did). Using it online seems profoundly misguided.
But in a weird way I was pleased to see it - it’s a holdover from a more innocent age of statistician-led research, one of elegant and smart solutions to research problems which made the - surely trifling! - assumption that all our respondents were maths wonks too.
