Had a chance to chat with Shuman Ghosemajumder regarding click fraud. Shuman is Google’s point person on the click fraud issue. This follows up on the post Andy Beal made on MarketingPilgrim earlier this week. Most of what we chatted about was in my Search Insider column this week. However, not all of it made it into the column, as there is a cut off which I routinely ignore (thanks to MediaPost editor Phyllis Fine for keeping me in line).

Here’s some tidbits that didn’t make it into the column:

First of all, I wanted to take the media to task for crying the sky is falling around this issue. I know that’s what journalists do, but the portrayal of the click fraud issue has been very one sided to this point. That’s why I wrote the column. I think it’s important we get balancing viewpoints. In the absence of numbers universally regarded as accurate, one has to poll the extremes and guess that the true answer lies somewhere in the middle. Up to this point, all we’ve heard are the negative estimates, and these are based on some studies with methodolgy that’s questionable at best (i.e. the Outsell study)

Secondly, I believe it’s unfair that everyone seems to be taking aim at Google, and to a lesser extent, Yahoo on this issue. I know they’re easy targets, because the targets are so damned big, but when the real numbers finally do come out, I’d bet my 89 Mazda 626 (the car that just won’t die!) that it’s the 2nd and 3rd tier networks that are the hotbeds of click fraud.

I dealt with it briefly in the column, but one of the main sources of misrepresentation seems to be this question of what click fraud is. For me, the definition is pretty simple, fraudulent clicks that leave the advertiser financially impacted. But when it comes to most of the media portrayals, there are a number of clicks that get lumped together under the label “click fraud”, the majority of which don’t meet this definition. And Google’s point of contention with reports of click fraud that come from the media and various 3rd party fraud detection tools comes from this aggregation of questionable numbers. There’s no distinction made between actual fraud, the clicks that cost the advertiser, and attempted fraud, the ones that got caught. And often more benign clicks, i.e. multiple legitimate clicks coming from the same IP address, get mistakenly labelled as click fraud.

Another positive move by Google was the inclusion of invalid clicks in the advertiser’s reporting dashboard. Every move that Google makes towards greater transparency is a very positive one. And the best know Google evangelist for communication, Matt Cutts, indicated so on a blog post. By the way, Shuman also has a blog, where he goes into greater depth on this issue.

I can only imagine how frustrating this must be for Shuman and the Google Click Fraud team. They sit and listen to numbers be bandied about in the 15% plus range, knowing from first hand experience that the real number is likely much much lower (in the column, using assumptions that are probably on the high side, the actual amount of click fraud that an advertiser would have to challenge Google on is less than 0.18%). Yet, their tongues are tied, both by Google’s legal and corporate communications department.

Why is the media targeting click fraud and trying to scare the hell out of advertisers? In no other industry I can think of are reporters more prone to mix and match numbers without regard for accuracy. They do it, and get away with it, because there are no independent and reliable numbers to look at.

There are a number of reasons. Google is in the vanguard of disruptive change agents that are shaking the very ground of marketing. It’s somewhat defensive to look for an Achilles heel, and right now, click fraud seems to fit the bill. Google in particular is boldly stating they want to change everything. That scares people.

Part of it is that there is still a lot of people that would love to see Google be knocked down a few pegs. Much as we rever success, wildly successful companies or individuals generate jealousy and suspicion. Our society gets a nasty little thrill when the mighty fall.

But perhaps the biggest reason is the very strength of search and online marketing: it’s accountability. Nothing else is as measurable. So when something appears to be eating away at the cost effectiveness, we tend to go all forensic on it and analyze the hell out of it. Could you imagine the mainsteam press making a big deal out of a .18% hole in the accountability in television advertising, or radio, or print? Even a 10 to 15% hole? Of course not, because much bigger holes than that are accepted every day as being inherent in the channel. But search and online ad networks are apparently fair game.

Is click fraud happening? Absolutely. And if you switch the lens a bit, there are some sophisticated click fraud operations that are making a killing. In a response to my column, Chris Nielsen had this excellent observation:

The problem is not overt clicking on ads, competitors clicking on ads, or double-clicking on ads. The problem is with large-scale concerted efforts that are massive enough to to have enough variety of IP address, user agents, etc. and pose as “valid” user click activity.

Of course this activity varies some with the bid price of the clicks, but it’s really the old idea of stealing a penny from a million people. If anyone notices, who’s really going to care? The problem is that in some areas, there are hundreds or thousands of people stealing pennys, and it is noticible and it is a problem. The only real indication is the lack of bona fide conversions, and that’s hard to say for sure if it’s fraud or real factors with the marketing or web site.

But it comes down to which lens you look through. Do you look at those looking to profit from click fraud, some of them doing it very well? Or do you look at the scope of the problem over the big picture? The problem I have with the BusinessWeek report is that the reporting is trying to do both at the same time, and you can’t get a clear picture by doing so.

I just wanted to wrap up this post by mentioning some other initiatives on this front that Google is pursuing which didn’t make it into the original column. Obviously they’re working on proprietary techniques to filter out click fraud, but they’re also trying to attack the problem on an industry wide basis as well. They’re working with the IAB Click Measurement working group, in which SEMPO is also involved. And they’re calling for stringent and scientific independent auditing standards, so when we throw around terms like click fraud, we’re all dealing with a common reference framework. By the way, I also asked Shuman about impression fraud. We didn’t go into a lot of depth on the issue, but they feel they’re equally on top of that as well.