Five Ideas That Matter

Search tools are not infallible, as anyone can testify who has taken content metrics sourced from Google, Technorati or their peers at face value, and been burnt. Here is some evidence, from a recent internal research project.

Chart 1 plots the number of results reported by Google Blog Search in a simple search string, banded by month using the advanced search interface.

Chart 2 plots the same results if you throw away the number of results reported by Google, and count actual results by hand.

There is a critical issue here for metatrend analysis – the old business cliche of garbage in, garbage out applies: if data are junk, no valuable insights will be created. Researchers working within social media need to understand the limitations of the tools that they use, and incorporate an effective validation process into their workflow.

A Metatrend method paper is in draft form, so more on the implications of accuracy and reliability in tool selection later.

We’re interested in ways of gleaning new insights from online data, and also in new ways of delivering those insights. So, our ears pricked up when we heard about a recently-launched iPhone application – called “Outbreaks Near Me” – based on MIT’s HealthMap resource.

From HealthMap.com

HealthMap monitors and maps semantic references to various illnesses through news reports and social media channels, giving users a potential early warning of outbreaks. It’s one among a number of interesting health data mashups that have cropped up, including Google Flu Trends – a similar initiative, which bases its findings on search trends, rather than media or blog comment.

Some are sceptical about the utility of these initiatives as standalone tools – false positives are an issue for example in both comment monitoring and search data. This scepticism is often justified, but we’re mainly interested in their contribution to metatrend analysis. Their value really becomes apparent when you look at combining the outputs from many different data sources together.

We can measure flu trends now by a combination of social media listening and aggregated search trends, but what else can we add to the mix? There have been successful attempts in the past to identify outbreaks through pharmacy sales data, and in the future technologies like FLIR (forward looking infrared cameras) might also have a contribution to make. More data, tighter insights, better value.

We’re interested in Metatrends partly for their potential to deliver untapped insights, and partly because we’ve lost confidence that traditional quantitative market research can deliver bread and butter insights in a convincing way.

The frequency which which we read about badly applied primary research methodologies is depressing, and even more depressing is the fact that examples of bad methods – such as television audience measurement – have attained institutional acceptance through inertia.

Commercial market research is a mature discipline, but while methods and resources to collect and mine data have evolved, program design has barely changed at all.

There are two widespread problems with traditional commercial research, split between sampling errors and respondent errors.

Sampling errors are usually the result of a compromise between scope and budget. For example, prestigious and respected surveys such as Fortune’s Most Admired Companies – based on a survey of only around 4,000 people.

Respondent errors are a hangover from another age, when there were no other ways to gather opinion than to ask people what they thought. The problem: people won’t always tell you the truth – because of embarrassment, malice or any of a number of other reasons. Gaming surveys has even become popular as a form of subversive act. The best possible example of this problem are election exit polls – notoriously volatile, and consequently hard to rely on.

Our experiments in metatrend analysis are rooted in the belief that there could be another, better way to measure trends – built on opinion, attitude and intention signals that can be mined online.

We’re building our method as we go – testing out a range of open and licensed data sources, acquisition and analysis tools for effectiveness. A methodology white paper will follow in the next few weeks, but what is certain is that it will not be a sacred cow. We plan to avoid the same stagnant rut as traditional market research: as tools, resources and processing capabilities evolve, so will we.

Behavioural economics is the dismal science with people brought back in. I think it tells us that the more perfect a market, the easier it is to see human irrationality at work. The web is a perfect information market, full of glitches of course, but one that REALLY shows human irrationality and its consequences.

The definition of a perfect market would be something along the lines of information being freely and equally available in the same timely manner to all participants. Sounds like the web.

This is one of the reasons why we think the web acts in a bull and bear kind of way that stock markets also do. And why we think marketing has to think less rationally and change its expectations.

My vote for the most irrational market behaviour of people is the Apple iPhone. Apple dominates the market in information about mobile handsets, and specifically that of smart phones.

First a piece of external data from Hitwise: iPhone almost immediately outstripped interest in the iPod just about from the minute it was launched.

Then there was consolidation of that interest until by late 2008 the iPhone was unassailable in the market for smart phone information. We studied the iPhone and Android, the Google-led open source operating system for mobile phones, from December 2008 to March 2009 to compare the two, to understand the web as an information market for handsets.

If you want to get on top of the very latest in the iPhone bull market turning bear take a look at this post of yesterday from Michael Arrington at Techcrunch. Om Malik is also an iPhone sell advocate. Interesting point is both have lived with serious iPhone negatives for two years and chose not to write or mention this. And the sell recommendation is not based in performance negatives. It is based on new contract conditions that Apple and AT+T want to impose.

So let’s turn the clock back. During our study period there was no major iPhone initiative or announcement. There was however the announcement of the first Android handset, the HTC G1.

Under normal circumstances Android, with the infinite reach of Google, and HTC should have dominated the information market for mobile handsets during this period.

What happened? During that time iPhone garnered 12 times more information flow than the G1 and Android combined.

The normal way to look at this is to say the iPhone gets stronger information flow because it is more popular. But we don’t believe the web works that way.

Take Facebook as a parallel. There is little in the way of competition to Facebook. It has define a service that is similar on concept to many other social networks but no-one else tops 200 million users. There is a bull market in Facebook information. Its numerical success bears only an ill defined relationship to its inherent qualities.

In our view there is a tipping point in cases like iPhone and Facebook where the fact of information becomes more critical than their features, price or quality.

The iPhone is a poorly constructed product, with a long lock-in contract, arguably with poor performance characteristics compared to competitors.

If the only answer to its success is the user interface or usability then we’d have to conclude that all products should focus primarily in usability regardless of performance. In the case of the iPhone consumers choose to pay a higher than normal price to ignore poor performance.

Instead we believe iPhone addiction is the phenomenon we are looking at. As with Facebook. And other online bull markets. The kind of factors that are at work are confirmation bias – a tendency to take on only information that works as confirmation of our views; and availability bias – a tendency to use the latest information available rather than the balance of information.

Neither of these explains the peculiar web characteristic that we see with the iPhone, that people are generating information about it, all the time and that this information is crowding out other information about smart phones. To get at that we need to look at the sources of influence in web information markets. We’ll be digging out our Android study to do that in our next post.