The Bug in Google’s Flu Trend Data

First published March 20, 2014 in Mediapost’s Search Insider

Last year, Google Flu Trends blew it. Even Google admitted it. It over predicted the occurrence of flu by a factor of almost 2:1.  Which is a good thing for the health care system, because if Google’s predictions had have been right, we would have had the worst flu season in 10 years.

Here’s how Google Flu Trends works. It monitors a set of approximately 50 million flu related terms for query volume. It then compares this against data collected from health care providers where Influenza-like Illnesses (ILI) are mentioned during a doctor’s visit. Since the tracking service was first introduced there has been a remarkably close correlation between the two, with Google’s predictions typically coming within 1 to 2 percent of the number of doctor’s visits where the flu bug is actually mentioned. The advantage of Google Flu Trends is that it is available about 2 weeks prior to the ILI data, giving a much needed head start for responsiveness during the height of flu season.

FluBut last year, Google’s estimates overshot actual ILI data by a substantial margin, effectively doubling the size of the predicted flu season.

Correlation is not Causation

This highlights a typical trap with big data – we tend to start following the numbers without remembering what is generating the numbers. Google measures what’s on people’s minds. ILI data measures what people are actually going to the doctor about. The two are highly correlated, but one doesn’t not necessarily cause the other. In 2013, for instance, Google speculated that increased media coverage might be the cause for the overinflated predictions. More news coverage would have spiked interest, but not actual occurrences of the flu.

Allowing for the Human Variable

In the case of Google Flu Trends, because it’s using a human behavior as a signal – in this case online searching for information – it’s particularly susceptible to network effects and information cascades. The problem with this is that these social signals are difficult to rope into an algorithm. Once they reach a tipping point, they can break out on their own with no sign of a rational foundation. Because Google tracks the human generated network effect data and not the underlying foundational data, it is vulnerable to these weird variables in human behavior.

Predicting the Unexpected

A recent article in Scientific American pointed out another issue with an over reliance on data models –  Google Flu Trends completely missed the non-seasonal H1N1 pandemic in 2009. Why? Algorithmically, Google wasn’t expecting it. In trying to eliminate noise from the model, they actually eliminated signal coming during an unexpected time. Models don’t do very well at predicting the unexpected.

Big Data Hubris

The author of the Scientific American piece, associate editor Larry Greenemeier, nailed another common symptom of our emerging crush on data analytics – big data hubris. We somehow think the quantitative black box will eliminate the need for more mundane data collection – say – actually tracking doctor’s visits for the flu. As I mentioned before, the biggest problem with this is that the more we rely on data, which often takes the form of arm’s length correlated data, the further we get from exploring causality. We start focusing on “what” and forget to ask “why.”

We should absolutely use all the data we have available. The fact is, Google Flu Trends is a very valuable tool for health care management. It provides a lot of answers to very pertinent questions. We just have to remember that it’s not the only answer.

The Ill Defined Problem of Attribution

First published July 11, 2013 in Mediapost’s Search Insider

For the past few years, I’ve sat on the board of a company that audits audience for various publications. One of the challenges the entire audience measurement industry has faced is the explosion of channels traditional publishers have been forced to use. It’s one thing to tally up the audience of a single newspaper, magazine or radio station. It’s quite another to try to get an aggregate view of an audience of publishers that, in addition to their magazines, have a website, several blogs, various email newsletters, a full slate of webinars, a YouTube channel, multiple Twitter accounts, Facebook pages, other social destinations, digital versions of magazines and an ever-growing collection of tablet and smartphone apps. Consider, for instance, how you would estimate the size of MediaPost’s total audience.

The problem, one quickly realizes, is how you find a common denominator across all these various points of audience engagement. It’s the classic “apples and oranges” challenge, multiplied several times over.

This is the opposite side of the attribution problem. How do you attribute value, whether it’s in terms of persuading a single prospect, or the degree of engagement across an entire audience, when there are so many variables at play?

Usually, when you talk about attribution, someone in the room volunteers that the answer to the problem can be found by coming up with the right algorithm, with the usual caveat something like this: “I don’t know how to do it, but I’m sure someone far smarter than I could figure it out.” The assumption is that if the data is there, there should be a solution hiding in there somewhere.

No disrespect to these hypothetical “smart” data-crunchers out there, but I believe there is a fundamental flaw in that assumption. The problem behind that assumption is that we’re accepting the problem as a “well defined” one – when in fact it’s an “ill-defined” problem.

We would like to believe that this is a solvable problem that could be reduced to a simplified and predictable model. This is especially true for media buyers (who use the audience measurement services) and marketers (who would like to find a usable attribution model). The right model, driven by the right algorithm, would make everyone’s job much easier. So, let’s quit complaining and just hire one of those really smart people to figure it out!

However, if we’re talking about an ill-defined problem, as I believe we are, then we have a significantly bigger challenge. Ill-defined problems defy clear solutions because of their complexity and unpredictability. They usually involve human elements impossible to account for. They are nuanced and “grey” as opposed to clear-cut “blacks and white.” If you try to capture an ill-defined problem in a model, you are forced to make heuristic assumptions that may be based on extraneous noise rather than true signals. This can lead to “overfitting.”

Let me give you an example. Let’s take that essential human goal: finding a life partner. Our task is to build an attribution model for successful courtship. Let us assume that we met our own livelong love in a bar. We would assume, then, that bars should have a relatively generous attribution of value in the partnership “conversion” funnel. But we’re ignoring all the “ill-defined” variables that went into that single conversion event: our current availability, the availability of the prospect, our moods, our level of intoxication, the friends we were with, the song that happened to be playing, the time of night, the necessity to get up early the next morning to go to work, etc.

In any human activity, the list of variables that must be considered to truly “define” the problem quickly becomes impossible. If we assume that bars are good places to find a partner, we must simplify to the point of “over-fitting.”  It may turn out that a grocery store, ATM or dentist’s waiting room would have served the purpose equally well.

Of course, you could take a purely statistical view, based on backwards-looking data. For example, we could say that of all couples, 23.7% of them met in bars. That may give us some very high level indications of “what” is happening, but it does little to help us understand the “why” of those numbers. Why do bars act as a good meeting ground?

In the end, audience measurement and attribution, being ill-defined problems, may end up as rough approximations at best. And that’s OK. It’s better than nothing. But I feel it’s only fair to warn those who believe there’s a “smarter” whiz out there who can figure all this out: Human nature is notoriously tough to predict.

Seperating the Strategic Signal from the Tactical Noise in Marketing

First published April 4, 2013 in Mediapost’s Search Insider

It’s somewhat ironic that, as a die-hard Darwinist, I find myself in the position of defending strategy against the onslaught of Big Data. Since my initial column on this subject a few months ago, I’ve been diving deeper into this topic.

Here’s the irony.

Embracing Big Data is essentially embracing a Darwinist approach to marketing.  It resists taking a top-down approach (aka strategy) by using data feedback to enforce evolution of your marketing program. It makes marketing “antifragile,” in the words of Nassim Nicholas Taleb. In theory, it uses disorder, mistakes and unexpected events to continually improve marketing.

Embracing strategy — at least my suggested Bayesian approach to strategy — would be akin to embracing intelligent design. It defines what an expected outcome should be, then starts defining paths to get there. But it does this in the full realization that those paths will continually shift and change. In fact, it sets up the framework to enable this strategic fluidity. It still uses “Big Data,” but puts it in the context of “Big Testing” (courtesy Scott Brinker).

To remove the strategy from the equation, as some suggest, would be to leave your marketing subject to random chance. Undoubtedly, given perfect feedback and the ability to quickly adapt using that feedback, marketing could improve continually. After all, we evolved in just such an environment and we’re pretty complex organisms.  But it’s hard to argue that a designer would have designed such flaws as our pharynx, which is used both for eating and breathing, leading to a drastically higher risk of choking; our spinal column, which tends to become misaligned in a significant portion of the population; or the fact that our retinas are “inside out.”

Big Data also requires separating “signal” from “noise” in the data. But without a strategic framework, what is the signal and what is the noise? Which of the datum do you pay attention to, and which do you ignore?

Here’s an even bigger question. What constitutes success and failure in your marketing program? Who sets these criteria? In nature, it’s pretty simple. Success is defined by genetic propagation. But it’s not so clear-cut in marketing. Success needs to align to some commonly understood objectives, and these objectives should be enshrined in — you guessed it, your strategy.

I believe that if  “intelligent designers” are available, why not use them? And I would hope that most marketing executives should fit the bill. As long as strategy includes a rigorous testing methodology and honest feedback does not fall victim to egotistical opinions and “yes speak” (which is a huge caveat, and a topic too big to tackle here), a program infused with strategy should outperform one left to chance.

But what about Taleb’s “Black Swans”? He argues that by providing “top down” direction, leading to interventionism, you tend to make systems fragile. In trying to smooth out the ups and downs of the environment, you build in limitations and inflexibility. You lose the ability to deal with a Black Swan, that unexpected occurrence that falls outside of your predictive horizon.

It’s a valid point. I believe that Black Swans have to be expected, but should not dictate your strategy. By their very nature, they may never happen. And if they do, they will be infrequent. If your strategy meets a Black Swan head on, a Bayesian approach should come with the humility to realize that the rules have changed, necessitating a corresponding change in strategy. But it would be a mistake to abandon strategy completely based on a “what-if.”

A Benchmark in Time

First published September 13, 2012 in Mediapost’s Search Insider

That’s the news from Lake Wobegon, where all the women are strong, all the men are good-looking, and all the children are above average. — Garrison Keillor

How good are you? How intelligent, how talented, how kind, how patient? You can give me your opinion, but just like the citizens of Lake Wobegon, you’ll be making those judgments in a vacuum unless you compare yourself to others. Hence the importance of benchmarking.

The term benchmarking started with shoemakers, who asked their customers to put their feet on a bench where they were marked to serve as a pattern for cutting leather. But of course, feet are absolute things. They are a certain size and that’s all there is to it. Benchmarking has since been adapted to a more qualitative context.

For example, let’s take digital marketing maturity. How does one measure how good a company is at connecting with customers online? We all have our opinions, and I suspect, just like those little Wobegonians, most of us think we’re above average. But, of course, we all can’t be above average, so somebody is fudging the truth somewhere.

I have found that when we work with a client, benchmarking is an area of great political sensitivity, depending on your audience. Managers appreciate competitive insight and are a lot less upset when you tell them they have an ugly baby (or, at least, a baby of below-average attractiveness) than the practitioners who are on the front lines. I personally love benchmarking, as it serves to get a team on the same page. False complacency vaporizes in the face of real evidence that a competitor is repeatedly kicking your tushie all over the block.  It grounds a team in a more objective view of the marketplace and takes decision-making out of the vacuum.

But before going on a benchmarking bonanza, here are some things to consider:

Weighting is Important

It’s pretty easy to assign a score to something. But it’s more difficult to understand that some things are more important than others. For example, I can measure the social maturity of a marketer based on Facebook likes, the frequency of Twitter activity, the number of stars they have on Yelp or the completeness of their Linked In Profile, but these things are not equal in importance. Not only are they not equal, but the relative importance of each social activity will change from industry to industry and market to market. If I’m marketing a hotel, TripAdvisor reviews can make or break me, but I don’t care as much about my number of LinkedIn connections. If I’m marketing a movie or a new TV show, Facebook “Likes” might actually be a measure that has some value. Before you start assigning scores, you need a pretty accurate way to weight them for importance.

Be Careful Whom You’re Benchmarking Against

If you ask any marketer who their primary competitors are, they’ll be able to give you three or four names off the top of their head. That’s the obvious competition. But if we’re benchmarking digital effectiveness, it’s the non-obvious competition you have to worry about. That’s why we generally include at least one “aspirational” candidate in our benchmarking studies. These candidates set the bar higher and are often outside the traditional competitive set. While it may be gratifying to know you’re ahead of your primary competitors, that will be small comfort if a disruptive competitor (think Amazon in the industrial supply category) suddenly changes the game and blows up your entire market model by resetting your customer’s expectations. Good benchmarking practices should spot those potential hazards before they become critical.

Keep Objective

If qualitative assessments are part of your benchmarking (and there’s nothing wrong with that), make sure your assessments aren’t colored by internal biases. Having your own people do benchmarking can sometimes give you a skewed view of your market.  It might be worthwhile to find an external partner to help with benchmarking, who can ensure objectivity when it comes to evaluation and scoring.

And finally, remember that everybody is above average in something…

Marketing Physics 101

First published February 9, 2012 in Mediapost’s Search Insider

Physics has never been my strong suit, but I think I have a good basic grasp of the concepts of velocity and direction. In my experience, the two concepts have special significance in the world of direct marketing. All too often I see marketers that are too focused on one or the other. These imbalances lead to the following scenarios:

All Direction, No Velocity

As a Canadian, I am painfully familiar with this particular tendency. Up here, we call it a Royal Commission. For those of you unfamiliar with the vagaries of the Canadian political landscape, here’s how a Royal Commission works. It doesn’t. That’s the whole point. Royal Commissions are formed when you have an issue that you wished would simply go away, but the public won’t let it. So a Royal Commission deliberates over it for several months, issues a zillion-page report that nobody ever reads, and by the time the report comes out, everybody has forgotten why they were so riled up in the first place.

This is similar to a company’s strategists noodling for months, or even years, about their digital strategy without really doing anything about it. They have brainstorming sessions, run models, define objectives and finally, decide on a direction. Wonderful! But in the process, they’ve lost any velocity they may have had in the first place. Everyone has become so exhausted talking about digital marketing that they have no energy left to actually do anything about it. Worse, they think that because it lives on a shelf somewhere, the digital strategy actually exists.

All Velocity, No Direction

With some companies, the opposite is true. They try going in a hundred directions at once, constantly chasing the latest bright shiny object. Execution isn’t the problem. Stuff gets done. It’s just that no one seems to know which direction the ship is heading. Another problem is that even though velocity exists, progress is impossible to measure because no one has thought to decide what the right yardstick is. You can only measure how close you are to “there” when you know where “there” is.

Failing any unifying metrics grounded in the real world, people tend to make up their own metrics to justify the furious pace of execution. Some of my favorites: Twitter Retweets, Number One SEO rankings and Facebook Likes.  As in “our latest campaign generated 70,000 Facebook likes” — a metric heard in more and more boardrooms across America. Huh? So? How does this relate in any way to the real world where people dig out their wallets and actually buy stuff? Exactly what dollar value do you put on a Like? Believe me, people are trying to answer that question, but I’ve yet to see an answer that doesn’t contain the faint whiff of smoke being blown up my butt. I suspect those pondering the question are themselves victims of the “all velocity, no direction” syndrome.

Balanced Physics

The goal is to fall somewhere in between the two extremes. You need to know the general direction you’re heading and what the destination may look like. You will almost certainly have to make course adjustments on the way, but you should always know which way North is.

And if you have velocity, it’s much easier to make those course adjustments. Try turning a ship that’s standing still.

Look at the Big Picture in 2012

First published December 29, 2011 in Mediapost’s Search Insider

Another year’s pretty much in the can. And because I’m working on idle this week, trying to catch my breath with my family before plunging headlong into 2012, search marketing falls somewhere behind the recent releases on Netflix and trying out the new Wii game on the list of things preoccupying my mind. So, don’t expect any salient and timely search news from me!

When I look back on what has preoccupied me over the last 12 months, I will say that much of it has been spent “stepping back” and trying to look at the bigger picture. As online interactions have taken a bigger and bigger chunk of our lives (you’ll notice that both of the recreational options I mentioned have online components woven into them), trying to understand how our actions play out against a broader online backdrop has been the thing I think about most often.

We digital marketers tend to take that “bigger picture” and break it into pieces, trying to make sense of it by focusing on one small piece. Digital marketing lends itself to this minute focal depth because of the richness of each piece. Even the smallest chunk of an online interaction has a lot to explore, with a corresponding mound of data to analyze. We could spend hours drilling into how people use Linked In, or Twitter, or Google+ or Facebook.  We could dig into the depths of the Panda update or how local results show up on Bing and never come up for air.

But think back to what, at one time, was another holiday season pastime. Some of us remember when we used to get a jigsaw puzzle for Christmas. You’d dump out all 5,000 of those little photographic morsels and then begin to piece it together into a coherent image of something (usually a landscape involving a barn or a covered bridge). Success came not only from examining each piece, but also in using the image on the boxtop to help understand how each piece fit into the bigger picture. Without understanding what that bigger picture was supposed to look like, you could examine each piece until the cows came home (again, often a topic for jigsaw art).

So, much of my 2011 was spent trying to understand what the picture on the top of the puzzle box was supposed to look like. What would ultimately tie all the pieces together?  In physics terms, I guess you could say I’m been looking for the Unified Field Theory of online marketing. And you know what I realized? You won’t find it by focusing on technology, no matter how cool it is. Foursquare marketing or search retargeting or hyperlocal optimization are all just pieces of a much bigger puzzle. The real picture emerges when you look at how people navigate the events of their lives and the decisions they must make. It’s there where the big picture emerges.

A few weeks ago I was speaking to a group of marketers about the emerging role of mobile.  This was no group of digital slouches. They knew their mobile stuff. They had tested various campaign approaches and honed their tactics. But the results were uneven. Some were hits, but more were misses. They knew a lot about the pieces, but didn’t have the boxtop picture to guide them.

My message (for those who know me) was not a surprising one: understand how to leverage mobile by first understanding how people use mobile to do they things they intend to do.  Don’t jump on a QR code campaign simply because you read somewhere that QR codes are a red-hot marketing tool. First see if QR codes fit into the big picture in any possible way. If you do that, you might find that QR codes are a puzzle piece that actually belongs in another box.

After delivering my sermon about the importance of understanding their respective big pictures, I asked my favorite question: “How many of you have done any substantial qualitative research with your customers in the past year?” Not one hand went up. This was a group of puzzle assemblers working without any boxtop picture to guide them.

If you want to sum up my past year and fit it into one final paragraph for 2011, it’s this: Understand your customers! Spend a good part of 2012 digging deep into their decision process and their online paths. Make it personal. Stalk if necessary. Ask questions that start with “why.” Observe. Make notes. Broaden your online reading list to include blogs like Science Daily, Futurity, Neuroscience Marketing and Homo Consumericus. At some point, the bigger picture will begin to emerge. And I bet it will be much more interesting than a landscape with a barn and some cows in it.

We’re Looking in the Wrong Place for our Attribution Models

First published June 16, 2011 in Mediapost’s Search Insider

The online landscape is getting more complex. Speaking from a marketer’s perspective, there are more points of influence that can alter a buyer’s path. At the last Search Insider Summit, John Yi from Facebook introduced us to something he called Pinball Marketing. It’s an apt analogy for the new online reality.

Hoping for a Strike

In the past, marketing was like bowling. You would build a campaign with sufficient critical mass and aim it toward your target, hoping at the end of the campaign (or lane) your aim was good enough, and the ball/campaign had enough kinetic energy (measured in REACH X FREQUENCY X AD ENGAGEMENT) to knock down all the potential customers.  If you think about marketing in this perspective, it explains the massive amount of pain traditional marketers are feeling as they pull their bowling-shoe-clad feet from the old world and gingerly dip their toes in the new. The bowler was in control (theoretically) and the success or failure of the campaign lay in her hands alone. The paradigm was simple, clean and linear, just the way we marketers like it.

The new game of marketing is much more like pinball. The intersections between a buyer’s decision path and a product’s marketing presence are many, and each can send the buyer off in a different direction. Some of those intersection points are within the marketer’s control — and some aren’t. Marketers now have to try to understand engagement and buyer impact at each of these intersections and, in the process, try to piece together a map of the buyer’s journey, assigning value in the appropriate places.

Repealing Newton’s Law

But even though the frenetic path of a pinball gets us a little closer to today’s marketing reality, it still doesn’t get us all the way, because there’s one fundamental difference: pinballs don’t have brains. Nor do they have emotions, feelings, or needs. Pinballs are just little metal spheres that obey the laws of physics.

And therein lies the difference.  How much more challenging would pinball be if, rather than relying on Newtonian physics to set the path of a ball coming off a flipper, it could decide whether it wanted to go right, left or simply stop dead in its tracks, refusing to go one inch further until you showed it a little more respect.  As physicist Murray Gell-Mann once quipped, “Imagine how hard physics would be if particles could think.”

As we try to understand what influences our buyers, we tend to apply something like the laws of physics to unraveling attribution. We apply formulas to various touchpoints, mathematically weighting their respective values. We can weight it to the first click, the last click, or divvy up the value based on some arbitrary calculation. But, in the end, as we try to figure out the new rules of marketing, we tend to forget that these balls have brains.

Go to the Source

If we want to understand what makes buyers buy, we should ask them. We should base attribution models on decision paths, not arbitrary formulas. We should walk through the buying landscape with our prospects, seeing how they respond at each intersection point. And when we build our attribution models, we should base them on psychology, not physics.

Is this approach harder than the holy grail of a universal attribution formula (or even multiple variations of said formula)? Absolutely. It’s fuzzy and sometimes messy. It tends to squirm around a lot. And unlike Newtonian physics, it depends on context. What I’m proposing is riddled with “ifs” and “maybes.” In short, it’s human in its ambiguity, and that’s really the whole point. I would much rather have ambiguity that’s somewhat right than clarity that’s completely wrong.