Every market researcher’s favourite miracle is the feeding of the 5,000, when Jesus took a representative sample of loaves and fishes and extrapolated it to a population. But these core techniques of quantitative can always be cast in a shady light, as with this PC Pro report (linked above) on how many filesharers there are in the UK.
The report’s attack on the research rests on four things:
1. Forrester used a sample of 1,000 to make conclusions about the entire population.
2. Forrester upweighted the percentages of people to take into account the social undesirability of self-reporting illegal activity.
3. There’s a clash between Forrester’s figures and the UK government’s re. overall population size.
4. The research was commissioned by the BPI.
None of these are, by themselves, bad things. If you have a representative sample you can extrapolate it: there will be a margin of error, but that’s how sampling works. I suspect when PC Pro take their ABC figures to advertisers they don’t say “look, honestly, this is based on a tiny fraction of the population, don’t take it too seriously”. People don’t like the idea of extrapolation, it innately annoys them in fact because it says “you’re not different, and if you are, you don’t matter”. But that doesn’t mean it doesn’t work.
Similarly, people don’t answer survey questions honestly: if you ask people whether they smoke, a proportion of smokers will say “no”. If you ask people how they’ll vote, a bunch will lie. If you know what the proportion is who are likely to lie, you can weight the figures accordingly.
Thirdly, Forrester are entitled to use whatever figures they feel are most accurate for overall population size. These might or might not be the government’s: it’s a little odd to assume they would be more accurate in a piece designed to show how the government can spin figures.
And fourth, the job of a research company is to give its clients accurate information. If Forrester is doing its job, the research won’t be “partisan”.
So much for the defense. One problem is that there’s a lot of IFs piling up on top of one another. If the research isn’t influenced by client assumptions, and if the universe data is accurate, and if whatever weighting they’ve put on for lying respondents is accurate, then the figure is accurate - within a margin of error. Forrester have an incredibly good reputation so I think those 3 “if”s aren’t an issue here and I expect that 7 million filesharers is a pretty good shout.
But the job isn’t to persuade other market researchers, or to persuade a client behind closed doors, it’s to be a central statistic in forming government policy on a very hot-button issue. And by grabbing a statistic from a report, the government have left themselves - and Forrester - horribly exposed when the workings are shown. They should have done the bloody work themselves. At the very least they should have recalibrated the figure using their own population estimates (and the fact that they didn’t is hardly the researcher’s fault) and acknowledged the margin of error. As it stands this is yet more bad research PR.
