Blackbeard Blog: Archive

A Research Fable

This is the last Blackbeard Blog entry for a little while, because I go on paternity leave tomorrow and will be hip-deep in nappies from offspring #1 and #2. So here’s something a little more whimsical than usual…

A Research Fable

It’s May 1989, and you’re sitting at your desk in a research agency, filling in the labels on an overhead projector chart and thinking about the future. You’re enjoying your work, but is it really a long-term career? You’re young! There’s still time to change – do you really want to be doing market research in 20 years time?

Just as these thoughts bubble up in your brain, there’s a popping sound, and through the space-time continuum drops a letter from the future. “FIVE THINGS YOU NEED TO KNOW ABOUT THE YEAR 2009”, it says.

The world’s number one brand makes its money largely from the intelligent use of consumer data.

Hundreds of millions of people share their likes, dislikes and opinions, every day, linked to demographic data, and no incentives needed.

You can find out what consumers think about any subject you can imagine, without leaving your desk.

Marketers are buzzing about the idea of their customers having conversations with one another about their brand.

You can bring people from Oslo, Ohio, Osaka and Ouagadougou together in conversation, virtually for free.

“But this is fantastic!” you cry, “Surely for market researchers this is the best of all possible worlds! My goodness I’m glad I made such a sensible career choice: riches here we come.”

But then there’s another popping sound, and a second letter drops into your lap. “FIVE MORE THINGS YOU NEED TO KNOW ABOUT THE YEAR 2009”. Nervously, you open it.

More consumer data is going to be produced this year than in the rest of history put together – but only an infinitesimal fraction by market researchers.

Anyone who wants can put together their own surveys for free – and get respondents answering.

It will become generally accepted that most people can’t report their own behaviour or motives accurately.

If someone wants to talk to a brand, they’ll talk to a brand – and the brand will be desperate to listen, with no researcher required.

Response rates will be lower than ever, and the hot topic for researchers will be a perceived crisis in engagement and data quality.

“Whaaat?” you howl, “But this is catastrophic – it sounds as if research will be completely undermined! What a disaster! I wish I’d gone into something more solid, like journalism or the music industry.”

You hold the two letters in your hand. “Some help you are, The Future!” you curse, “Now I’ve got to decide which of you to trust. After all, you can’t BOTH be true.”

But they were.

May 28, 2009

Much-twittered post by Ian Delaney which I enjoyed this morning. I don’t really know my Foucault from my Elbault but I get the general thrust of the argument - it’s about the dangers of self-filtering content:

Facebook people get unfriended if they say the wrong thing; unsavoury Twitter followers are not followed or blocked. Bland self-approval of the group takes over. There are no racists on my spectrum right now. As far as I am concerned, they don’t exist. But that’s not the real story, clearly. Racists are poised to take Stoke in the next by-election. They don’t appear on my spectrum because I have deliberately blinded myself to their existence on a day-to-day basis. Diversity of opinion is purely opt-in (with strong incentives to opt-out) in socialmediaworld.

I’d love to see some stats or research done on triggers for de-friending/un-following across various social media platforms (Twitter, Facebook, LiveJournal are all very different here). My suspicion is that what Delaney calls “transgressive” opinions aren’t followed in the first place, though the outcome will of course be the same.

But his example is a little off. There’s a big logic gap between “There are no racists on my spectrum right now” and “As far as I’m concerned, they don’t exist”. I don’t have any BNP members in my Tweetstream either (I hope), but I do have a lot of people concerned about the BNP, and some people actually involved in anti-BNP activism around the UK. So even judging only on the tweetstream I’m well aware that racists exist: I just don’t want to hear what they have to say.

In other words, my stream isn’t cossetting me as to the existence of racism, it’s simply excluding people who might try and persuade me of the BNP’s case. Now, in this particular case that doesn’t bother me in the slightest - which is a danger with using racism as an example. There is a small but crucial difference between “blocks out the existence of” and “narrows the range of available opinion on”.

Even so, if social media tools did narrow the range of opinion, I’d consider that a pretty bad thing. But thinking about this I’m not sure they do - at least, I’m not sure Twitter does. (Facebook is a different matter.)

Let me explain. I now follow more than 500 people on Twitter. In general, I followed them either because I thought they had interesting things to say about pop music, market research or social media. The self-reinforcing effect Delaney talks about does indeed apply: “interesting things to say” tends to mean “things I at least partly agree with”.

But even within this set of topics, there’s no guarantee that someone I’m following because they’re sound on social media will be equally sound on pop music. And if they should tweet about pop music I’ll likely be exposed to some views I dislike. And none of my reasons for following have to do with, say, politics: so my market research posse includes plenty of conservatives and libertarians who I most certainly disagree with on those issues even while admiring their research skeez.

Now Delaney might say - well, none of that is transgressive or oppositional. And fair enough. But my point is that because Twitter users’ streams aren’t generally filtered by topic, you have a much greater potential opportunity for serendipitous encounters with differing opinions than you did in topic-grouped environments like USENET or IRC (held up as loci of transgression in Delaney’s post). I’ve seen a lot of stereotypical online behaviour on Twitter but I’ve not yet seen the kind of groupthink so common with tools which allow easier group creation.

May 27, 20091 note

From the link above:

“Brain scan technology supports Restak’s observation about the incompleteness of our knowledge of our motivations. More often than we’re inclined to admit, the reasons we give for doing something better fit the category of speculation than reality.

Yet researchers confidently present clients with statistical renderings of what consumers have told them, unmindful of the fact that motivations initially take root outside the realm of consciousness.”

Here are four things I might want to know about people:

How they behave
How they think they behave
What they think
What they say they think

If my game is trying to influence people, all four of these are useful. Oh, and I also want to know about all four at an individual and a group level, since being in a group changes behaviour dramatically.

Which of these do brain scans tell me? With the right interpretation, and with the right experimental controls, they are useful indicators of the gap between #3 and #4. Which is awesome - it’s giving us genuinely new information and a way to confirm stuff marketers and researchers have always suspected about people and their motivations. But this:

The problem is, there is too much economic interest in keeping to the old ways in the $6 billion research industry for change to happen overnight. This is one reason why the neurorevolution I talked about in the previous post is moving more slowly than it might.

is poppycock. The “neurorevolution” is moving “more slowly than it might” because it’s been hyped as a paradigm shift rather than a useful bit of kit.

May 27, 2009

One frustrating thing about working in marketing for a research company was the incredible difficulty we had in getting any good case studies out in the world. Because we were trading in confidential data, we had to get client permission, and the paradox was that the more useful the data or analysis had been, the less likely we were to get it.

So the few we DID manage to clear were passed around the company, from slide deck to slide deck, until they became as familiar as bedtime stories - and likely to induce the same soporific effect. One particular idea generation example - a novel way of serving children’s food - could induce Reggie Perrin like howls of existential despair among the marketing team whenever its cheeky pandas and tigers showed up at the end of an otherwise magnificently sculpted pitch. We knew we’d done better, more recent work than this - but our lips were sealed.

I get the same feeling looking at the examples of social media case studies that keep cropping up again and again - Dell’s Ideastorm. Cadbury’s Wispa revival. Crowdsourced dog treats. The link at the top of this post is to an Ad Age piece on the Del Monte dog snacks - a yarn that’s been doing the rounds for a good couple of years now. I remember stuffing it into a memo I wrote a year or so ago, having seen it in a Marketing Week piece. A week or two later I chatted to the journalist who’d put the MW piece together and the Snausages example came up: he ruefully admitted he’d been desperate to find a fresher one but good ROI examples are few and far between.

It seems they still are. The Snausages example is a fine one - it ably demonstrates the value of listening to your outlier customers. But I can’t help thinking that the credibility and cause of social media research and marketing won’t be helped by the same half-dozen examples circling around and around on a creaking carousel.

May 26, 2009

Trolls v Bores redux

Some great comments on my community bores post - thanks to all. I was expecting a rash of comments along the lines of “who are you to judge?” and am grateful not to have suffered it!

One clarification I feel I ought to make: an individual bore is of no harm or importance. You can route around him easily. On a one-on-one basis, a troll is considerably more harmful.

The peril of bores is that they’re hard to shift and tend to accumulate - and what matters isn’t the individual bore but the proportion of boring v non-boring content. A community attacked by multiple trolls will tend to take action: but communities infected by bores tend to be like the proverbial frog in boiling water.

(NB: Many trolls - the poor quality ones especially - are also bores, of course!)

May 25, 2009

“What we see play out through social network sites is the emergence of what Penny Eckert marked in her seminal text “Jocks and Burnouts.” Those who are drawn to Facebook are more likely to represent privileged, educated, stronger socioeconomic backgrounds. They are more likely to be respectful of adult society and more likely to connect with adults who hold power over them. Those drawn to MySpace are more likely to come from immigrant families and from poorer, urban communities. They are more likely to be resistant to normative value and affiliate with subcultures. Of course those divisions are not clean and a good number of teens straddle both worlds. But that’s precisely why Eckert noted that the hegemony of high school is not comprised solely of one group or the other, but the tension between them.”

Worth reading the whole thing. Couple of thoughts:

- There’s a definite parallel with the way the UK social media commentariat completely ignores Bebo (though there’s a big age skew there, but I think also a class one)

- Teens from “poorer, more urban” communities who “affiliate with subcultures” and are “resistant to normative value” have historically speaking been popcult gold. If MySpace embraced this, promoting itself as an edgier and more vibrant community than Facebook, could it regain some of its momentum?

May 23, 2009

Online Communities: Fight The Real Enemy

A post on Freshnetworks by Holly Seddon reminded me to write about what I think - beyond even trolls - is the number one menace on online communities.

Bores.

Here are the reasons why bores are the bane of a community.

1. They come in many forms: Depending on how your community works your bores might include verbose bores, bores who trot out received wisdom, “me too” bores, “First!” bores, bores who flog in-jokes to death, list bores, bores who ask obvious questions, and many many more.

2. They often appear to contribute signal: That said it’s hard to actually fault bores. Aside from off-topic bores (there’s another kind!) they tend to stick resolutely to their mundane point and you can’t really categorise it as “noise”, it’s just useless signal.

3. It takes less effort to make a boring contribution than a non-boring one: Thinking of something interesting to say takes time. Thinking of something someone else said and parroting it takes less. Saying the same thing you always say takes even less.

4. It’s hard to legislate around them: Because they’re not strictly speaking doing anything wrong, it’s very hard to kick bores out without damaging the fabric of your community (see point 7 below). User-rating of content might do the trick for some types of bore, but risks encouraging others (and carries its own set of problems). All you can really do is try and stop them arriving in the first place. Which isn’t easy. Partly because…

5. They attract other bores while putting off non-bores: People with dull ideas may or may not react well to people with interesting ones. But people with interesting ideas aren’t likely to want to hang out with people who have dull ones. So the more bores your community has, the harder it gets to recruit new non-bores.

6. Unlike trolls, they don’t realise they’re bores: Of course, almost nobody comes to a community and thinks “I’m too boring for this place”. Every bore thinks they have plenty to contribute. In fact, you might be a bore. So might I.* The definition is highly fluid and subjective, after all!

7. Even raising the problem makes you look like an arsehole: The ethos of most social media is highly participatory and built around welcoming people and being nice to them. When you start talking about bores being a problem, you open yourself up to looking conceited, elitist, bullying, envious, etc. And in a way you are, since it’s quite possible for bores to get less so.

So what can you do? If you throw up your hands and do nothing, the community reverts gradually to the mean. On the communities I’ve been on solutions have ranged from “ban them” (not fair or workable), “set a good example” (good idea but problematic because of point #6), limit contributions per person in some way (risky). Limiting or sidelining topics which are boredom red-flags - by pointing people to FAQs, setting up sub-boards for list threads, and so on - can be helpful, as can promoting and featuring interesting things.The best way forward is probably gentle encouragement.

*it’s worth noting that boring-ness is only an issue on communities, not social networks. Unless you’re related to my son I am an extremely boring Flickr user, but my boringness would only become more problematic if I joined groups: as an unconnected node in the netork I’m harming nobody’s experience.

(Thanks to Frank Kogan for raising some of these issues.)

May 22, 20097 notes

The Expectations Game

This set of slides by Pete Ashton is as good an intro to social media as you could want, because it tells his own story first and then gets into the theory. And his own story has a couple of very important lessons.

1. Anybody could do it.

2. Once you start doing it, not only do you get better (as with everything), but interesting things get progressively more likely to happen to you.

These are hardly new ideas, so it’s the way Pete tells ‘em that helps them feel fresh. They’ve also sparked a good discussion in the comments box to that post, which has started to focus on barriers to participation, and especially the fear of failure. (This post is actually a repost of my comment there, so just go there and ignore me waffling on!)

The web, it is said, removes the financial cost of failure. But what about the emotional cost?

My take is that the emotional cost is heavily influenced by expectations, and our expectations tend to be exaggerated by where we are on a social tool’s adoption curve.

F’rinstance, I started blogging at about the same time in Blogger history as Pete started in Flickr’s timeline - not one of the very first users but early enough that I didn’t have any expectations. So every bit of recognition I did get felt like a success and a surprise. Same with my first website - I remember, after 6 months, getting my 1,000th “hit” overall on Sitemeter and buying everyone who was around a drink to celebrate.

But if I was launching my first site, or starting my first blog, or joining Flickr or Twitter or whatever NOW my definitions of ‘failure’ would be different. If I participated and people didn’t notice I’d be disappointed - especially as my awareness of what it was “meant” to be like would be warped by media hype of blogs, Twitter, Flickr etc.

There’s an awful lot of encouragement of people into social media - which is great - but not an awful lot of helping them set realistic expectations.

May 21, 20091 note

Frank Kogan pointed me at this very interesting little NYT column by Steven Strogatz, about the economies of scale that apply within both cities and human bodies. The column talks about how the metaphor of city-as-organism may be more than just a metaphor: cities seem to organise themselves around the same structural principles.

The example given by Strogatz is the number of petrol stations per head in cities: this number increases with population, but not in proportion.

For instance, if one city is 10 times as populous as another one, does it need 10 times as many gas stations? No. Bigger cities have more gas stations than smaller ones (of course), but not nearly in direct proportion to their size. The number of gas stations grows only in proportion to the 0.77 power of population. The crucial thing is that 0.77 is less than 1. This implies that the bigger a city is, the fewer gas stations it has per person.

So one obvious question I have is: what kind of equivalent economies of scale would apply to the infrastructure of online networks and communities? Not just the actual physical infrastructures - servers, bandwidth, etc. - but the actual activity within the community?

May 21, 2009

Open-Source Research

If someone was ever lunatic enough to run a Top 100 Survey Questions special on Channel 4, Fred Reichheld’s Net Promoter Score would surely be near the top. There are a lot of people who loathe the Net Promoter Score, of course, but there are a lot of people who loathe “Bohemian Rhapsody” and that doesn’t seem to stop it polling well.

This entertaining Vovici Blog post on the NPS lays out the arguments against and for, and some of Reichheld’s own counters. This bit leaped out at me as worth thinking about:

“As he said: “NPS is an ‘open source’ system: you are all welcome to use it! You don’t have to pay me to use it.” (I could have done without his next statement: “Therefore its bad business for market research, and why they call me a liar.”) But if you’ve ever gone to implement the five questions of the Secure Customer Index® or the one question of the Customer Effort Score™ and realized you couldn’t because they are proprietary, you’ll appreciate that there are no such restrictions on the Net Promoter Score.”

I don’t know enough about NPS to know whether it’s actually “open source” (i.e. you can modify it) or simply “free to use”, but the concept of open source research intrigues me. At first it seems wildly counter-intuitive: most research organisations have methodologies they carefully protect, and tracking studies which require a consistent approach to asking questions.

But in the world of DIY research maybe open-source is the way to go. If you’re a non-profit with no research budget, say, you probably don’t have a huge amount of survey-writing expertise and left to your own devices you might well get things wrong - creating questions which don’t differentiate well or which are ambiguous. Why not open the study to your respondents (or present your discussion guide as a wiki)?

Using free/DIY survey products you could easily have a “beta” and “launch” phase - the former allowing users to modify your questionnaire and options, the latter with the (much improved) fixed questionnaire for analysis purposes. It might add a day or two to your timings but it could get round one of the main issues with DIY.

May 20, 2009

Losing My Edge 2.0

What’s the one bit of “social media” received wisdom you’d most like to eradicate?

Mine is the notion that there was a chronological divide between “web 1.0” and “web 2.0”, and that social interaction and user-generated content online only began with the latter. I can’t imagine anyone reading this blog would disagree, but offline - and in the sillier marketing blogs - I still run into this attitude surprisingly often.

It’s related to that other dangerous concept, the “digital native”, and it’s based in the conceit that the first decade of the web has nothing to teach us. It’s no lie that mass adoption changes the impact of a technology, but it would be crazy to assume the change invalidates everything learned about that technology before the adoption happened.

And it’s no accident that the people who I learn the most from in social media are often those with a deep background of online activity: they’ve put the hours in on USENET, BBS systems, MUDs, IRC, the blogosphere, online fandom, etc. They’ve probably built stuff or run stuff themselves, and they’ve got plenty of war stories. They’re not necessarily the most financially successful - though some are very successful - because once you’ve got into the habit of doing stuff online for love it’s an irritatingly hard one to break.

What this all adds up to is more than just experience, it’s what experience can give you - something I reckon is the single most important skill an analyst (or a critic!) can have: the ability to recognise when a precedent matters and when it doesn’t. If all you can see is the difference between then and now you’ll end up hyping everything, if all you can see if the similarity you won’t spot the important stuff. You have to be able to identify both - often in the same thing.

May 19, 20092 notes

A matter of perspective

In one of Douglas Adams’ Hitch-Hikers’ Guide books there’s a machine called the Total Perspective Vortex, a form of punishment which shreds its victim’s mind by exposing it to the actual scale of the universe and showing them how infinitesimally tiny and insignificant they are. The outrageously egotistical protagonist, Zaphod Beeblebrox, is flung into the machine and emerges smiling, the Vortex having apparently confirmed that yes, he was in fact enormously important.

Working in quantitative research generally turns you into a Total Perspective Vortex - you’re exposed not so much to how insignificant everyone is, but to how ordinary they are: rare indeed is the genuine outlier, the individual who can’t be clustered or segmented somehow. And you’re no different, buster! To some extent of course, this is a trick of the light - segments are caricatures, and the individual who is completely predictable is even rarer.

But something of this acknowledgement of perspective has gone into the traditional ‘research bargain’. We solicit opinions from people on the promise that someone high up in some company somewhere might use those opinions to improve their lives. A bit. Maybe. It’s all very cagey: we don’t say who might listen and we certainly don’t say that they will listen - the sense of distance acknowledges that you might find your answers in a distinct minority.

In fact it works a bit like democracy: the act of voting doesn’t give you the right to have what you voted for enacted. You might be on the losing side: thanks for playing, better luck next time.

“Better luck next time” isn’t a phrase that shows up much in this list of “10 Rules for Today’s Consumers” - a ‘new consumer manifesto’. Consumers want things fast, how they want, where they want; they expect to be listened to, responded to, partnered with as a matter of course. At once! Right now!

As the parent of a 2 year old boy I feel quite familiar with these “new consumers”. No Total Perspective Vortex for them - no ‘research bargain’ where they might get what they want, but no guarantees. They can demand the impossible and still think it reasonable.

If these new cosnumers do exist (and I do wonder if this isn’t just an enlarged, louder segment of Zaphods), having them as your customers will be an exhausting mix of servility, tactful diplomacy and gentle manipulation. Let’s not forget that “The customer is always right” became a mantra in an age when “The customer is mostly quiet”. The most urgent marketing question in a real-time world won’t be how to delight your customers, but how to get them to rediscover and accept the inevitable compromises between their desires and what you can deliver. (The music business seems already to have failed here.)

Luckily, as a researcher that’s not my direct problem. The issue for research is simpler - we used to sell ourselves to participants as bringing them closer to companies. Now we’re intermediaries, and the danger is we’ll be seen as making companies more distant. Part of our job as researchers is precisely to put things into perspective: that’s what our clients want, but is it in the interests of the people whose opinions we seek?

May 15, 20092 notes

Mapping The MR-Sphere: 20 Research Blogs To Follow

This post serves two purposes. It’s a follow-up to my thoughts about the benefits of a “research blogosphere”, and it’s also a means of assuaging the particular guilt I feel about never having sorted a blogroll out for Blackbeard Blog. Hopefully I’ll be able to do something about this soon.

Meanwhile here are 20 research blogs I’m following at the moment - suggestions of more delightedly received, as I’m sure I’ve left out one or two obvious ones. I’ve tried to include blogs which are more or less purely about market research - so excellent blogs like Verbatim (community building), FreshNetworks (social media) and Further And Faster (planning) aren’t on the list despite having plenty of research content. And I’ve only included blogs in English - please help me out with suggestions, non-Anglophones.

UPDATED: I’ve now listed the Twitter names of each blogger at the end of the post - thanks to @harrisonma1 for this idea!

There’s also a real mix of styles and topics here, and I’ve tried to indicate each blog’s approach with a brief write-up. So, in no particular order:

Straight Talk With Nigel Hollis: Nigel is Millward Brown’s chief global analyst, and his site is probably the most-read research blog around at the moment. As you’d expect from MB, the focus is very much on branding but Hollis is also a staunch champion of research in general. Not on Twitter.

More Than Market Research: Tom H C Anderson’s blog about his “next gen market research” ideas (combining traditional tools with data mining and text analysis). For me, what makes the blog really valuable are its series of insightful interviews with research, marketing and digital thinkers. @tomhcanderson

The Future Place: The blog of Ray Poynter’s consultancy firm, the Future Place is one of the longest running research blogs and is increasingly taking an agenda-setting role when it comes to promoting social media and new methodologies in research. (For instance, “The Dawn of New MR”) @raypoynter

Voice Of Vovici: Many research firms run a blog these days - Vovici’s is definitely one of the better ones, focusing on practical tips and examinations of the nuts and bolts of survey work. @jhenning contributes a lot of the posts.

Zebra Bites: I don’t always agree with Katie Harris (of Australian qual agency Zebra) but her excellent posts invariably make me think - she’s a voice of realism and occasional provocation when it comes to the advance of ‘online qual’. @zebrabites

Curiously Persistent: Simon Kendrick’s blog has been a bit quieter since he started a new job at qual agency Essential, but its posts are worth waiting for - the blog name says it all about his open-minded approach. And his occasional lists of links to good articles are some of the best around. @curiouslyp

Research Reinvented: Synovate’s Emiel Van Wegen is one of the most active researchers on Twitter, which might account for the relative scarcity of blog posts - but his list of researchers who tweet played a huge part in shaping the nascent MR ‘twittersphere’. @emielvanwegen

Bad Research: No Biscuit: One of a handful of anonymous research bloggers, BR:NB specialises in bringing horrible online survey practises to our scandalised attention. @researchrant

Kumeugirl: A TNS qual researcher based in Asia, Kumeugirl writes involved, intelligent posts which tend to be studded with useful follow-up links. @kumeugirl

MROC Talk: Another agency blog, this time of Plugged In, who specialise in creating MROCs (Market Research Online Communities). As you’d expect, they’re pro new methodologies and write shrewd posts about working with them. @mattpluggedin

Lovestats: One of the friendliest and most good-natured blogs on the MR circuit, Annie Pettit’s LoveStats mixes solid advice on stats and quant research with quirkier material (like her “Ode To A Pie Chart”) @lovestats

Market Research Deathwatch: And just to show the tonal range of the MR-sphere, here’s its most evil-natured inhabitant, the anonymous MR Heretic - who takes a militant stance to warn the industry of its impending demise. The vitriol alone is worth a follow, but MR Heretic also makes consistently good points. @mrheretic

FreshMinds Research Blog: Another agency blog, but far more than just a way to pimp FreshMinds work - intelligent and rich posts on online culture, measurement, customer acquisition and more. Like Nigel Hollis, the writers tackle the business issues research is designed to shed light on as much as they write about research itself. Can’t find Twitter accts for the posters.

The Human Element: Qual researcher Alison Macleod’s blog is perhaps my personal favourite - not only are all her posts well-written and well-thought-out, she’s got an enviable knack of inspiring high quality comment box discussion too. @alisonmacleod

CRO-ing About Research: A blog by the ARF’s Chief Research Officer, Joel Rubinson, setting out his attempts to transform the research game by moving ‘listening’ to the centre of the discipline. Like Ray Poynter, he’s an agenda-setter for the New MR. @joelrubinson

Yellow Submarine Qual: They haven’t updated since March but I threw YSQ in here because quite a few people have recommended them. There’s a lot of insight and real charm in their rambling, complex and - here’s a rarity! - funny posts on qual and its issues. @cristi_popa

Modern Metrix: One of the research world’s better kept secrets, MMX specialises in analytics and metrics, posting frequent and interesting links to bring these potentially dry topics to life. @modernmetrix

Research Rockstar: I admit it, I am allergic to the usage “rockstar” applied to anyone other than some sweaty old giffer strutting round an arena wearing leathers. Kathryn Korostoff’s blog is a lot more substantial than the word suggests, though - she emphasises the practicalities of MR and she’s a passionate advocate of the profession. @researchrocks

The Better Research Blog: The Better Research Blog mixes research observations with industry commentary, which makes it feel more real-world and grounded than some of the theory- or tip-laden blogs out there (love those though I do! But there’s room for both approaches.) @mendelj2

Datasets: A new blog (and I hope its writer hasn’t become discouraged already, good new blogs are rare in any space) with posts on scales, statistics and the effects of research participation on behaviour. @BretIG

Like I say, there are surely many more out there - if I’ve left you or a blog you read out, let me know and I can put together a follow up list.

And, naturally, I think you should include my blog on your subscription lists if you don’t already!

May 14, 2009

If you want to read my article on the 10 hottest thinkers in the consumer behaviour area, that’s a link to a PDF of it - massive thanks to Research magazine!

I will add only two comments:

i. Nassim Nicholas Taleb is an anagram of “Chaos salesman bin-lit”

ii. Apologies to danah boyd, whose “read this” was left off (my fault not the mag’s) - specially annoying cuz it’s FREE. (Her dissertation). AND the designers didn’t lower-case her name. Sorry :(

May 13, 2009

Research vs Tinkering

The latest hashtag swarm on Twitter is #fixreplies, which I’ll try and sum up quickly:

1. When someone you follow posts a tweet, you get to see it.

2. But if it’s an @ reply, you don’t see it unless it’s to someone you also follow. (This stops you seeing half-conversations).

3. There is an option in Twitter’s settings to remove this filter, so you see everything your follows post, @ or not.

#fixreplies is about Twitter taking away that option, which they said was confusing people. The upshot of this: if you didn’t tick that option, your experience of Twitter won’t have changed at all. If you did, your experience will have changed dramatically. We can presume that your experience has changed for the worse, in that you specifically opted in to seeing all replies and now you can’t. (Those are the people complaining, by and large. And for what it’s worth, I’m one of ‘em.)

In other words, among existing users, nobody sees an improved experience, the majority sees an unchanged one, and a minority see a worse one. So why has Twitter done this?

I can think of two reasons. Firstly, the option was a server-side resource drain which contributed to the downtimes Twitter sometimes experiences. Secondly, Twitter have been doing some investigation into what’s causing their apparently high churn rate, and have found that new users who ticked that option are much more likely to not come back. Both these would apparently justify the move by improving the experience of new users.

Twitter are leaving the door open for backtracking on this, and quite possibly they will - maybe restoring the option with a clearer explanation of what it does would be a good idea.

What’s interesting to me from a research point of view is this: did Twitter research this change, or simply enact it? It reminds me a little of the recent brouhaha around Facebook’s Terms Of Service - a change enacted, a vocal user response, then some compromise and confusion. In fact, it reminds me of a lot of things. If there’s one unbreakable law of social media it’s this: every service pisses its users off at some point.

I reckon this is because social media and web firms have a completely different attitude to their product than old school businesses: they’re tinkerers, interested in small improvements which they test live rather than pre-test. As Tim O’Reilly has pointed out, they exist in a state of perpetual beta, constantly making improvements. The release cycle, says O’Reilly, is dead.

The question then is, does the death of the release cycle mean the death of the research cycle, which is parasitic upon it? What can traditional market research offer a company like Flickr, which upgrades itself every half hour? One answer is a higher level strategic view, but in an environment changing this rapidly even that kind of information has a far shorter half-life.

In terms of tactical decision-making - the bread and butter of research - you might as well forget it. When decisions can be made and re-made on an hourly cycle there’s precious little room for it, and besides, live web optimization testing can sort out most of your problems.

With regards to #fixreplies, Twitter will either stick by its decision or reverse it in some way. If the former, research wouldn’t have helped (as users would have been angry anyway). If the latter, research might have helped avoid the situation entirely, but then Twitter loses the feelgood factor of the service “listening to its customers”.

I think this is a real dilemma for the research business: research happens more privately and more slowly than tinkering - what can it offer to businesses who are used to that mode of working?

May 13, 2009

Do we need a market research blogosphere?

I asked a question on Twitter today - does a market research blogosphere exist, and what use would it be?

I got a lot of replies saying “a what-o-sphere?”, which suggests my question writing skills aren’t too sharp, and that ‘blogosphere’ is jargon.

Here’s what I mean by it: a community of blogs in a loose network. Whether linking to each other, discussing one another’s posts, commenting on one another’s sites. By no means all agreeing, but with some common interest.

There are functioning blogospheres - by this definition - in politics, marketing, and ad planning, to name but three. There used to be several in music blogging - maybe there still are.

So, I’ll take a shot at answering my own question.

What good is a blogosphere? A blogosphere creates a type of decentred community. A lot of market research community initiatives exist, but they’re all centred on forums or sites: they’re destinations not networks. The blogosphere has an advantage for participants in that your blog is your own - you set its agenda, you can use it to promote yourself and your business, at the same time as taking some part in the wider blogosphere.

Aside from networking (which is probably better done on destination sites) a blogosphere has two main advantages, it seems to me. One is debate, the other is defense. Debate is the most important: ideas and discussions can spread quickly around a blogosphere, which accelerates the pace of new thinking. The blogosphere can act like a perpetual conference (OK, to some this will sound horrific, I admit!). Defense is a slightly more specialised case: when attacks on the validity or use of the profession surface and spread, a blogosphere can quickly bring together the best counter-arguments.

There are also issues with a blogosphere, especially in a commercial field like MR. Researchers are busy people: blogging is not going to be a priority for many. More importantly, for all that the industry is quite a chummy one, researchers are often competing against each other. The blogosphere to some extent encourages this competition, but it also creates a sense of co-operation which commercial context might strain. In my opinion the trade-off makes blogging worthwhile (or else I wouldn’t do it myself!)

What do you need for one to happen? Blogs, obviously. Interested readers. Bloggers who are each other’s interested readers, and willing to carry debates across individual blogs. I’d also add that you need a variety of approaches and viewpoints - to prevent a tedious “echo chamber” effect. And finally you need some kind of external pressure, stuff happening to blog about. With a lot of people suggesting that market research is changing rapidly, there would seem to be no shortage of that.

Is there a Market Research one? Not a large or well-established one, for sure. Market Research seems to be the poor relation of the marcomms industry when it comes to blogging. But I’d say there are definitely stirrings: as one commenter put it, an MR “twittersphere” is certainly emerging to link existing, small-readership blogs (like this one). Other prominent researchers are purposefully trying to set agendas, which is important to get debates started. Things are happening: whether all this turns into a “blogosphere” as I’d understand it is an open question.

To do my bit, in a post tomorrow I’ll list all the good research blogs I know, and give shout-outs to some of the people I think are doing most to pull conversation together.

The image in this post is of the Orrery built by the Long Now project.

May 12, 20094 notes

The authors of this paper (caution: moderately hardcore stats, though if I understood it you could too) call their finding a paradox. But it’s not really.

The finding is this: the more often you submit content to YouTube, the lower your chances of scoring a successful video become.

The authors - Fang Wu and Bernard A Huberman - profess bafflement that people continue to submit videos given their poor success record. Maybe they don’t understand the odds? But herein lies the problem with the paper: it’s working from an arbitrary definition of success.

Not that people whose videos hit the front page aren’t successful - but that any given uploader will be working from their own definition of what constitutes “success”. It may be to do with the exact demographics viewing their video, but even if it’s down to absolute numbers one person’s success is another’s failure.

Take my “Popular” feature on Freaky Trigger, for instance. A post in that series might rack up around 1000 views in its first week. These would be disastrous numbers for, say, Pitchfork, but for me they’re a triumph, and so I put much of my energy into writing those posts rather than other ones.

Now imagine that Pitchfork and Freaky Trigger do the same posts as YouTube videos. Pitchfork sees the figures and gives up quite quickly - its definition of success is closer to Wu and Huberman’s. Freaky Trigger sees the figures and thinks “fantastic!” and keeps on doing it, becoming more persistent (by the definition of the paper).

So what the paper is detecting is the propensity of people who want high levels of success to drop out when they don’t get it: the ones who didn’t realise the odds do give up. This isn’t surprising - producing content is a lot more time-consuming than playing the lottery. The ones who remain have already found their level and are presumably happy with it - they are succeeding by their definition, continuing to produce content which Wu and Huberman classify as “failure”, and confounding researchers with apparently quixotic behaviour.

May 11, 2009

Red Shift part 1: "Fly, All Is Known!"

This is part 1 of a 2 part entry: I’m aware of the risks in announcing same and if part 2 doesn’t materialise I suggest you call me rude names in the comments box.

Dalton Conley’s Elsewhere, USA - a sociologist’s overview of “knowledge economy” workers - is not an especially good book, for the reasons laid out nicely in this NYT review. It’s thin, it’s anecdotal, it’s weirdly outdated - for instance, it diagnoses the economic bubble but seems to assume it won’t burst, despite it already having done so some months before publication time.

One metaphor did stick with me, though. Conley discusses the knowledge worker’s constant, nagging fear of being found out - exposed as a fraud. Of course there’s an immediate and obvious explanation for this, which is that most knowledge workers are indeed frauds who should be found out. But setting that possibility aside, Conley locates this anxiety in two broad areas.

One is what he calls the “economic red shift”: income inequality has been rising in the US (and some other) economies for decades. Conley points out that the bulk of this inequality is happening in the top tier of incomes - i.e. the gap between the 80th and 100th percentile is increasing more rapidly than that between the 60th and 80th, which is spreading more than between the 40th and 60th, and so on.

Conley observes that for an affluent individual in this situation both the less and the more well-off appear to be accelerating away, making the stakes of ‘success’ or ‘failure’ alike seem larger.

His other explanatory factor is what he sadly doesn’t call a “digital blue shift”: the well-documented way in which network and portable technologies pull previously compartmentalised elements of professional lives together - home/office, work/leisure, local/global, personal/professional etc. (Not everyone approves.)

Conley suggests this has a shattering effect on the individual but I don’t think I buy that. Instead I see technology as allowing us to network our own lives, activities, and interests, making them if not quite interdependent at least harder to separate. This makes us more resilient but also increases our awareness of systemic risk - the catastrophic impact a major life change could have. Hence, more stakes-raising and more anxiety.

Anyway, the mental health of white-collar America isn’t really what prompted me to start writing this. I was more attracted to the red shift metaphor as a description for what being a statistic in a power-law distribution (like income!) feels like. Which has direct relevance to how people experience - and design for - social applications.

To be continued…

May 11, 20091 note

This is important, I reckon. I heard Orlando Wood of Brainjuicer talking about this at the recent MRS conference - presenting the idea in a well-formed nutshell. Here’s Mark Earls on it:

And yet much market research - thanks to its roots in our individualist 20th Century culture - assumes that things are otherwise: we ask folk what they do, what they’ve done, what they might do and why; we listen to what they say about their lives on the assumption that they might know.

So we’ve been playing around recently with a (new?) kind of research approach, which plays to human strengths, rather than this well documented human weakness: “We-research”, we call it.

While we may not be very good at ourselves, we turn out to be much better at other folk and what they do (and why)

Pushing this idea to the forefront is new and exciting, but I think the concept has been embedded in market research for ages, specially in qual projective techniques: “What kind of people like this brand?” and so on. Even in quant research you have question phrasing which is designed to distance the individual from their own perspective - “Here are some statements other people have made”.

Of course, a we-research approach is only helped by the fact that the wired portion of society has more information about what other people are doing available than at any other time!

May 8, 2009

Hot Stats

I used to do a little feature on my music site called the “Pop Music Focus Group”. It wasn’t a focus group, I only called it that to tease. It was more of a drinking session where we’d play a load of current records, argue about them, and give them a mark out of ten each. It was a lot of fun to do.

The people who took part enjoyed doing it, and they enjoyed learning what had done well or badly too. Even more, they enjoyed finding out what each record’s “Controversy Rating” was - the standard deviation of the scores. They liked finding out who the biggest “Pop Tart” was - the person handing out the highest mean score.

And best of all was the “Compatibility Score”: a simple correlation matrix which told everybody who else was closest to them or furthest away.

Now, there are no doubt far more valid ways of coming up with all these things, but this was the best I could do armed with a laptop, a six pack and a copy of Excel. The point is that all the participants really enjoyed learning the results, and the more statsy and fiddly and algorithmic it got the more they enjoyed it.

What does this have to do with research? This: quant research, and modelling, and statistics are REALLY AWESOME when you present them in a way that feels meaningful and participatory to people.

When you ask people something, tell them the results. Tell them in a way that entertains them. They’ll want you to ask them more things, honest!

May 8, 20093 notes

If you have 56 seconds to spare...

In an “attention economy” it’s useful to know how much attention your website is likely to get, right? A few days ago, I noticed a statistic going around the Twitterverse - “you have 56 seconds to make your point”. It turned out that 56 seconds was the average time spent on a site, according to Nielsen.

Not long, eh?

Now, I used to work for Nielsen, back in the NetRatings days, and one of my jobs was to do their press releases. So part of me is always pleased to see their stats get attention. And my first thought was - oh, good angle. We never used to make much fuss over the average time spent metric, because it never changed by more than a few seconds.

But then I thought, hold on. And I went back and checked the earlier data via Google cache, and lo and hehold the metric back in March 2000 was….53 seconds.

And that’s not all - the 56s and 53s metric refers to the average time spent on a page, not a site. The average number of domains visited has leaped UP in the intervening years and now stands at 111. The average number of pages is at 2,554, which means (checks Excel), the average amount of time a visitor spends in a domain each month is… almost 22 minutes.

“You only have 22 minutes to make your point” doesn’t sound quite so dramatic.*

But of course most sites don’t get two minutes’ attention, let alone 22. The original statistic feels more right because “average” in the sense of the mean is pretty much meaningless. Those 111 domains probably include a webmail account, a couple of social networks, and the Google homepage - all of which rack the page views up like nobody’s business and skew the mean. As with all power law stats, the median is more interesting than the mean. Facebook and Google get a lot more than 22 minutes in a month, your little site gets a lot less.

The insight behind the “56 seconds O NOES” stat is fine - people don’t spend that long on a webpage so either get to the point or cultivate an audience who will take longer. But that’s been common sense since the web started (as the March 2000 stat shows!).

So what do the Nielsen stats actually tell us?

At the page level, people’s attention spans seem much the same as ever: some of the commentary turned a data point into a trend and said “well, American attention spans have got shorter”. There’s no evidence for this in these stats: the attention Americans pay to most pages has always been short, because the nature of the medium is that a lot of things that qualify as “pages” don’t have much info on them. Not that “average” means anything useful anyway!

People go to a lot more sites than they used to: This is because power has shifted from the huge portals (MSN, Yahoo) which dominated the rankings 10 years ago and aggregated information. Instead, sites dominate which hog time and pageview share but also spray visitors out to other, more irregular, destinations.

It’s incredibly difficult to prove anything much from averages in a power-law distribution: This isn’t news, but the Nielsen stats show (once again) how it should be the number one thing every researcher remembers about the web.** For a fuller discussion of this, read Clay Shirky’s typically brilliant takedown of the Wall Street Journal’s ridiculous recent “Bloggers be earning” story.

I’ve now taken considerably more than 56 seconds of your time, and will stop.

*This isn’t Nielsen’s fault, by the way: they put some data out their in public, and if people misread it that’s the people’s lookout.

**Incidentally, back when I did work for Nielsen, I was a crap analyst because I hadn’t internalised this. Sorry. I did write an OK press release though.

May 7, 2009

“And a labelled scatterplot for the lovely lady?”

(Srsly though encountering this kind of thinking is why I love my job)

May 7, 2009

Newsbeat have framed this story of a Ronaldo page’s demise to make the owner look a bit of a sadcase - and certainly he seems to fall into the “obsessive” fan category. But, you know, that’s what having fans involves! As Pete Ashton pointed out on Twitter, Facebook’s fan page policy seems designed to avoid pages created by ACTUAL FANS.

(I think my reaction to this wouldn’t be such an eye-roll if Facebook hadn’t chosen the word ‘fan’ to describe its brand presence pages: the relationship they seem to want is something considerably more….feudal.)

May 7, 2009

The Think Tank

Just a quick note to say that I’m in Research Magazine this month (May 09), writing a piece about the Top 10 hottest research-relevant thinkers. It was written to be deceptively frothy. The mag is out now, but they’re very kindly doing a special PDF of the article to circulate via Twitter, since I used that service to help me crowdsource some of the names.

If you simply want to know the names, well, I can help you with that too, but you’ll have to work a little harder: here they are in ANAGRAM FORM. (Over half are even APPROPRIATE!)

BIN CHAOS SALESMAN LIT
CHART LIAR HERD
REND SIR NO CASH
A BUSH ERA BANK MOSAIC
HER REAL JOHN
ALIEN ID LAYER
NUT WANTS CAD
DAB HAND, YO
YAKS RICHLY
KNEW ERA END

May 5, 2009

The Significance Problem

Via Lovestats on Twitter comes this handy little reminder of what a peril the free use of the s-word can be. The use of “significant” to describe a statistical relationship that passes a particular test is a bind because, as the article points out, the word’s meaning among non-researchers is a) near-universal and b) quite different.

In fact there’s two different misunderstandings at work here.

Non-researchers tend to misread “significant” as “important” or simply “big”. Which isn’t the case - it can be trivial or small, it’s just unlikely to be fluke or coincidence.

Researchers tend to read “significant” as “interesting”. Which isn’t the case either - even big results can be utterly banal, especially if they simply confirm something you could have guessed, or if they repeat information you already have.

The former misreading is a problem because it skews how people interpret research findings in the news or in their daily lives.

The latter misreading is a problem because it’s a root cause of boring, valueless research presentations - which leads to research being used to justify safe decisions rather than interesting ones.

A researcher faced with a wedge of quantitative data tables, tested for significance, will often simply skim through them and highlight EVERYTHING that passes a significance test. In some cases, they’ve been TAUGHT to do this. This mass of “significant” data forms the core of the report.

But significant/insignificant is only one way of cutting survey data. It’s a very useful way, let nobody deny that! On its own though, “it’s significant” deserves little more than a “so what?”

What do you ask next?

Is it useful? Does it relate to what the project was trying to uncover? Given this information, could anyone actually make a better decision?

Is it surprising? Some research is designed to avoid surprise, but in general “finding out stuff you didn’t know already” is an excellent reason to do research.

Is it explicable? Can we have a stab at why something’s significant? The explanation might be elsewhere in the data set, or elsewhere in another data set, or it might just be another hypothesis. But it’s better to actually frame that hypothesis than let the “significant” data sit there like some horrible statistical Sphinx.

Is it spreadable? If you can imagine your buyers telling their friends (not just their colleagues) about a result, it’s probably worth making a fuss over.

All this - I would venture - ought to fall into the “bleedin’ obvious” category, and nothing would give me greater pleasure than people queueing up in the comments to say “yes, this is how all researchers do work”. It probably is how all the clever blogosphere researchers work. But in the wider business? Hmmm.

May 4, 2009

May 4, 20092 notes

In my other gig as a Pitchfork columnist, I’ve written about Twitter. Hopefully the format isn’t too exhausting.

May 1, 20092 notes

Blackbeard Blog

May 2009

27 posts