Author Archives: ghotchkiss

Matt Cutts: Personalization and the Future of SEO

mattcutts555I had the chance to interview Matt Cutts this week about personalization and it’s impact on the SEO industry. Excerpts from the interview and some additional commentary are in my Just Behave column on Search Engine Land today. As promised, here is the full transcript of the interview:

Gord: We’ve been awhile setting this up, and actually, this came from a discussion we had some time ago about geo-targeting of results in Canada, and we’re going to get to that a bit later. With this recent move by Google to move towards more personalization of the search results page, there’s some negative feedback and, to me, it seems to be coming from the SEO community. What’s your take on that?

Matt: I think that it’s natural that some people would be worried about change, but some of the best SEO’s are the SEO’s that are able to adapt, that are able to look down the road 4 or 5 years and say, “What are the big trends going to be?” and adjust for those trends in advance, so that when a search engine does make a change which you think is inevitable or will eventually happen, they’ll be in a good position. Personalization is one of those things where if you look down the road a few years, having a search engine that is willing to give you better results because it can know a little bit more about what your interests are, that’s a clear win for users, and so it’s something that SEO’s can probably predict that they’ll need to prepare for. At the same time, any time there’s a change, I understand that people need some time to adjust to that and need some time to think, “How is this going to affect me? How is this going to affect the industry? And what can I do to benefit from it?”

Gord: It seems to me, having a background in SEO, that the single biggest thing with personalization is the lack of a “test bed”, the lack of something to refer to when you’re doing your reverse engineering. You can’t look at a page of search results any more and say “that’s going to be the same page of test results that everyone’s seeing“. Given that, , more and more, we’re going to be seeing less of universal search results, is this the nail in the coffin for shady SEO tactics?

Matt: I wouldn’t say that it’s necessarily the nail in the coffin, but it’s clearly a call to action, where there’s a fork in the road and people can think hard about whether they’re optimizing for users or whether they’re optimizing primarily for search engines. And the sort of people who have been doing “new” SEO, or whatever you want to call it, that’s social media optimization, link bait, things that are interesting to people and attract word of mouth and buzz, those sorts of sites naturally attract visitors, attract repeat visitors, attract back links, attract lots of discussion, those sorts of sites are going to benefit as the world goes forward. At the same time, if you do choose to go to the other fork, towards the black hat side of things, you know you’re going to be working harder and the return is going to be a little less. And so over time, I think, the balance of what to work on does shift toward working for the user, taking these white hat techniques and looking for the sites and changes you can implement that will be to the most benefit to your user.

Gord: It would seem to be that there’s one sector of the industry that’s going to be hit harder by this, and I think it was Greg Boser or Todd Friesen who said, “You don’t take a knife to a gun fight.” So when you’re looking at the competitive categories, like the affiliates, where you don’t have that same site equity, you don’t have that same presence on the web to work with, that’s where it’s going to get hit, right?

Matt: I think one area that will change a lot, for example, is local stuff. Already, you don’t do a search for football and get the same results in the U.K. as you do in the U.S. So there are already a lot of things that return different search results based on country, and expect that trend to continue. It is, however, also the case that in highly commercial or highly spammed areas, if you are able to return more relevant, more personalized results, it gets a little harder to optimize, because the obstacles are such that you’re trying to show up on a lot of different searches rather than just one set of search engine result pages, so it does tilt the balance a little bit, yes.

Gord: I had a question about localization of search results, and I think being from Canada we’re perhaps a little bit more aware of it. How aware are American SEO’s that this is the case, that if  they’re targeting markets outside the U.S., they may not be seeing the same results that you’re seeing in the U.S.

Matt: I think that many SEO’s are relatively aware, but I’ve certainly talked to a few people who didn’t realize that if you do a search from the U.K., or from Canada, or from India, or from almost any country, you can get different results, instead of just the standard American results. And it’s definitely something that’s a huge benefit. If you’re in the United Kingdom and you type the query newspapers, you don’t want to get, necessarily, the L.A. Times or a local paper in Seattle, the Post-Intelligencer. Something like that. So I think it’s definitely started down that trend, and, over time, personalization will help a lot of people realize that it’s not just a generic set of results, or a vanilla set of results. You have to be thinking about how you’re going to show up in all of these markets, and personalization and localization complement each other in that regard.

Gord: Now one difference between localization and personalization is that personalization has the option of a toggle, you can toggle it on and off. Localization doesn’t have that same toggle, so as a Canadian, sometimes I may not want my results localized. Where does that put the user?

Matt: It’s interesting, because you have to gauge…and you talked to Marissa a couple times already, and from that you probably got a feel for the difficulty in making those decisions about just how much functionality to expose, in terms of toggles and advanced user preferences and stuff like that. So what we try to do is tackle the most common case and make that very simple. And a lot of the times, the functionality is such that you don’t even necessarily want someone that’s coming in from the U.K. to be able to search as if they’re coming in from Africa because it just makes things a lot more complicated. So, over time, I’d say we’re probably open to lots of different ways of allowing people to search. For example, you can select different countries for the advertisements. There’s a GL parameter I believe, where you can actually say, “now, show the ads as if I were searching from Canada. Okay, now I’m going to switch to Mexico”. Stuff like that. And that’s been very helpful, because if you giving Google money to buy ads, you want to be able to check and see what those ads would look like, in different regions. For search we haven’t historically made that as easy. It’s something that we’d probably be open to, but again, it’s one of those things where probably SEO’s are a lot more interested,you’re your regular user isn’t quite as interested.

Gord: And that gets to the ongoing problem. SEO’s have one perspective, users have another and arguably, yes, localization is good for the user, but for an SEO that deals with a lot of Canadian companies where the U.S. is their primary market. They’re looking at hitting that U.S. market. I guess this restricts them to making it look like their sites actually reside in the U.S. to get around it. So again, we’re trying to poke holes in the functionality, rather than live with it.

Matt: Well, one thing that should be possible is to indicate some sort of preference, or some sort of origin of location where you can indicate where you are. Historically Google has been ahead of the other search engines at the time by not just using the top level domain, so .ca, but also the I.P. address. So you can have .com hosted in Canada and that’s worked very well for many, many years. But we do continue to get feedback that people would like more flexibility, more options, so it’s a matter of deciding how many people that would help and just allocating resources on those types of things.

Gord: So we talked about personalization, we talked about localization. Are there other factors that are coloring the search results we should be aware of as we’re trying to consider all these aspects?

Matt: Once you’ve sort of “broken the mould” with different results for different countries, after that it’s good for people to move beyond the idea of a monolithic set of search results. If we had the ability to say someone is searching for Palo Alto or someone is searching for Kirkland or Redmond and give them local newspapers, truly local newspapers, that would be a good win for users as well.  So over time, I would expect search results to serve a broader and broader array of services.  The idea of a monolithic set of search results for a generic term will probably start to fade away, and you already see people expect that if I do a search and somebody else does the search, they can get slightly different answers. I expect that over time people will expect that more and more, and they’ll have that in the back of their heads.

Gord: Let’s take that “over time” and drill down a little more.  One of the things it was interesting for me when I was talking to Marissa with the fact that the Kaltix acquisition was made four years ago and it’s really taken four years for that technology to really show up in the search results.  Obviously a cautious approach to it.  And even with that we’re talking a couple of results being lifted into the top 10 and we’re talking one in five searches.  Also Marissa wasn’t exactly sure about this so I’ll clarify this with you.  She believed that it would never replace the number one organic result.

Matt: I believe that’s correct.  I’d have to double check to make sure.

Gord: So that’s a fairly tentative step in the direction of personalization, and you said over time we can expect this to continue to ship to be more of an individual experience.  Are we talking months, are we talking years, are we talking tomorrow?

Matt (chuckling): It’s usually not our policy to comment on exactly when stuff might roll out in the future, but personalization is an important trend and the ability to make search results better through personalization is really exciting to us here at Google.  I think if you look backwards over time, a lot of the reason why we might not have been able to personalize before was because Google was very much a “you come to the front page, you do a search, you get the results and you’re gone” type of model.  And there really weren’t that many opportunities to have a single sign on or some sort of Google account, where we could actually learn or know a little bit more about you to make your results more relevant.  So I think part of it involves getting all of the different ways of having an account together, so you can have personalized news, which rolled out a while ago, you could have a personalized homepage and those things give people a reason to sign in to Google.  Once you’re signed in to Google that helps us a lot more, by having your search history and the ability to offer personalization.  So at least looking backwards, I think some of the amount of time was just getting people ready to have a Google account and not just show up in Google, do a search and leave.

Gord: So part of it is that transition from a tool you use to more of a community you are engaged in.

Matt: Yes

Gord: That’s moving closer to your competition, notably Yahoo and Microsoft.  Google’s done very well as a tool.  Is this just the inevitable progression?

Matt: I think one nice thing is that Google adapts very well to what users want, and also the industry marketplace.  And so when our primary competition was a pure search engine, whether it be AltaVista or AlltheWeb or HotBot or Inktomi, then pure search mattered very much.  Search is still a part of everything we do.  It’s at the core of all the information that we organize and yet competing against sites like Yahoo and Microsoft involves a different set of strategies than competing against just a search engine for example.  So I think competition is very good for users, because it makes all of us work hard and it keeps us on our toes.  The one strength that Google has is that we do adapt and we look at the marketplace and we say, “What do we need to deliver next for our users to help them out and to encourage them to be more loyal to Google?”

Gord: So for your job, where you’re looking at the quality of the index and policing it, how does personalization change your job?

Matt: To some degree, it makes it easier, because it’s not one monolithic set of search results anymore.  But let me flip that around and say how we can make it easier for SEO’s as well.  I’m a computer graphics person, so if you go back to a concept called digital half toning, it’s this process where you have nothing but black and white, yet you are able to approximate different shades of gray. And if you look at the existing set of search results, a lot of people before had a very black or white mentality.  I’m ranking, or maybe I’m ranking number one or are not in the search results at all.  And that’s a very harsh step function, in terms of you not ranking where you think you should be, and maybe you’re not getting very much traffic at all.  If you are ranking number one, or very highly, you’re a very happy person.  And yet that monolithic set of search results may not serve users the best.  So now as we see that spread and soften, more people can show up at number one but for a smaller volume of queries.  And so individual users are happier because they’re getting more relevant search results and yet it’s not a winner take all mentality for SEOs anymore.  You can be the number one ranking set of results for your niche, whether it be a certain demographic or a certain locality, or something like that.  And I think that’s healthier overall, rather than having just a few people that are doing very well, you end up with a lot more SEO, and a lot more users who are happy and that’s softens the effect quite a bit.

Gord: What you’re talking about is a pretty fundamental shift in thinking on the part of a lot of SEOs…

Matt: yes

Gord: … a lot of SEOs are almost more engineers right now, where they’re looking at the algorithm and trying to figure out how to best it.  You’re asking them to become a lot of things, more marketing, PR, content developers, and know more about the user, more about user behavior online.  These are very different skill sets and often don’t reside in the same body.  What is this going to do to the SEO industry?

Matt: I think the SEO’s that adapt well to change an optimized for users are going to be in relatively good shape, because they’re trying to produce sites that are really pleasing and helpful to users.  It’s definitely the case that if all you care about is an algorithm than the situation grows more complicated for you with personalization.  But it’s also an opportunity for people to take a fresh look at how they do SEO.  So give you a quick example: we always say, don’t just chase after a trophy phrase.  There are so many people who think if I ranked number one for my trophy phase I win and my life will be good.  When, in fact, numerous people demonstrated that if you chase after the long tail and make a good site that can match many many different user’s queries you might end up with more traffic than if you had that trophy phrase.  So already the smart SEO, looking down the road, knows that it’s not just the head of the tail, it’s the long part of the tail and with personalization and the changes in how SEO will work, it will just push people further along the spectrum, towards looking at “it’s not just looking at a number one result for one query, how do we make it across a lot of queries.  What value do I deliver?  Am I looking at my server logs to find queries that I should be targeting?  And not just search engines, how do I target different parts of the search engine?  Like the local part of Google, the maps part of Google.  How do I target Google notebook and the other properties and how do I show up well across the entire portfolio of search properties?”  And that’s a healthy transition period that will push people towards delivering better value for their users and that’s better for everybody.

Gord: I get that and I’m an SEO.  My challenge comes in getting my clients, who in a lot of cases did their own SEO or worked with another SEO firm before they came to us and are used to that trophy phrase ranking.  How do we get them to get?  Because I see that being a challenge with a lot of SEOs. They will understand that, but getting the client to understand it could be a different matter

Matt: Sometimes I think you might have to do a demonstration like sign them into personalized search, do a query, sign them out, do query and show them, these are very different sets of results.  And sometimes the demonstration can be very visceral, you know, it can drive home the point that it’s not just going to be this one trophy phrase. People are going to have to think and look at the entire horizon of the space.

Gord: In Google there’s a very definite church versus state divide and traditionally the relationship with the advertiser was almost exclusively on one side of that divide.  But this could mark a fairly fundamental shift, and it will impact your advertisers, so as part of that community, will Google be doing anything to help those advertisers understand the organic part of their visibility on Google?  Will you be doing the same demonstration you just telling us we should be doing?

Matt: I think Google is always trying to communicate with the outside community, both with webmasters and advertisers.  So it’s really exciting to see some of the different techniques that we’ve used, everything from webinars to training materials to making videos available.  I would definitely say that every part of Google is going to keep their eyes open on how to best communicate how to stay on top of changes, because nobody wants anybody outside of Google to be unprepared for personalization or improvements in any of our technologies.

Katie (Katie Watson, Google PR representative who was sitting in on the interview) Something to actually cite there is that I know we recently just opened up our webmaster blog to outside comments, so that’s a good example of gradually moving forward to communicate even better.

Matt: You were couching the question in terms of advertisers, but if you look at the general story of webmaster communication and assume that that’s the leading edge, it’s pretty safe to assume that those smart ideas are percolating throughout the company and we’re trying to figure out all the different ways to communicate more.

Gord: So that’s the canary in the coal mine. Whatever’s happening in the webmaster community will act as a testbed for communication?

Matt: Exactly.

Gord: There is a debate raging right now about “is SEO rocket science”?  (Matt begins laughing) So what does personalization means for that debate?  Does it become more complicated?  You said it becomes easier in some ways and I countered that by saying that may be, but is also spreading out in a lot of different directions. Is there still a place for the pure SEO consultant out there?

Matt: I think there still is a place for you for a pure SEO consultant but it’s also true that over time those consultants have to keep adding to their skill set.  A few years ago no one would have even thought about the word Ajax and now people have to think about Ajax or Flash and how do I handle some of these new interfaces to still make sites crawlable?  So I definitely think there will still be places for consulting and improving crawlability of sites and advice on keywords and personalization will add some wrinkles to that, but I have faith that, over time we’ll see the benefit to users and if you make good site for your users, you will naturally benefit as a result.  Some people spend a lot of time looking at data centers and data center IP addresses and if people want to have that as a hobby they’re welcome to it but a lot of people don’t do that anymore and they’re just worried about making good results and yet, everything still comes out pretty well for them.

Gord: Some time ago I wrote a column along that line and said that, in many ways, the white hat SEO has helped clean up the Black hat side of the street because they enabled those good site to be crawled, to show up in the index and to assume their rightful place in the results.  It would seem to mean that personalization is just going to drive that process faster.

Matt: I think it will.  It’s making Black Hat tougher to do.  I think it’s interesting, it was designed primarily to improve the relevance for users but as a side effect, it definitely changes the game a lot more if you’re on the Black hat side of things then if you’re on the white hat side of things

Gord: Matt, I think that wraps things up for me..

Matt: Thanks, that was fun.

Should We Believe Google’s Click Fraud Numbers?

Today, Google finally came out with some solid numbers around the click fraud issue. The number of invalid clicks across the Google network? Less than 10%. The total amount of undetected click fraud that advertisers have reported and asked for a refund for? .02% I was briefed by Google little while ago about their plans around click fraud and so I had some time to digest the numbers and think about them. Google also passed my name along as an expert third-party that the media could contact to get more commentary about the numbers and Google’s product roadmap for dealing with click fraud. If you’re interested in what the numbers actually mean, I would suggest going to Danny Sullivan’s post this morning on Searchengineland. Danny does his usual thorough job of making sense of the announcement.

One question that I got from a couple reporters yesterday was, did I believe Google’s numbers? Although I should have anticipated this question, I was somewhat surprised. So last night I thought about. What would Google gain by fudging the numbers at this point? I think there’s a few points you have to consider when looking for the answer to this question. Based on the fact that I’ve already been asked it three times, by three different reporters. I believe it is a valid question and one that a number of people will probably be asking.

Maybe I’m naïve, after all, I am an Alberta farm boy at heart, but in all my interactions with Google I have to say, Google just doesn’t work this way. Google is a very cautious company when it comes to divulging information. I would think one of the biggest frustrations that Shuman Ghosemajumder has had in the past is having to keep his mouth shut while various inflated numbers around click fraud were thrown about. My belief is that it’s been Shuman lobbying inside of Google that finally convinced them to open the box a little bit on the scope of click fraud in the Google advertising network. Maybe the “don’t be evil” motto of Google sounds trite to some, but people at Google believe it and take it to heart.

What would Google have to gain by releasing false numbers about click fraud? The only possible motivation would be to; one, artificially inflate their stock price, and two; encourage more advertising revenue by falsely reducing the sensitivity around the click fraud issue.

Let’s deal with the first point. I talk to financial analysts all the time and frankly, it’s been a long time since any of them asked me about click fraud. As far as a sensitive issue, there are a lot of other factors that financial analysts are looking at much closer when it comes to making recommendations on buying or selling Google stock. I believe click fraud has been already factored into the valuation and analysts have moved on.

When it comes to advertisers, there still is sensitivity around the click fraud issue, but it has lessened in the last year. The recent SEMPO study shows that as a concern for advertisers it actually trended down from 2005 to 2006. Certainly it’s something we should be aware of and keeping our eye on, but I really don’t believe it’s preventing advertising revenue from flowing into Google at this particular point. So any short-term gain that might come to Google from falsely announcing numbers could potentially be a bit of a spike in their stock price. But within a day or a week other factors would smooth that out and it would basically become a nonissue. I really don’t believe it would have any impact on advertisers at all. Short-term gain would be minimal at best.

But, the long-term cost to Google could be tremendous if they were caught releasing false numbers around click fraud. It would just be a really, really dumb thing to do, and you can say what you want about Google, but one thing they’re not is dumb. So do I believe the numbers? Yes, I have no reason not to.

The other question that the reporters asked me was what I believed the number to be for undetected click fraud. One of the reporters was actually from BusinessWeek, and if you’ve read my blog you know that I’ve taken some exception to BusinessWeek’s reporting around click fraud and search in the past. I did happen to mention that to the reporter I was talking to. The way I answered that question for BusinessWeek was that obviously we don’t know what we don’t know. Potentially there could be a lot of click fraud that slips through all of Google’s filters and slips past the advertiser as well. But for it to do so it would have to be click fraud at a extremely sophisticated level. Let me explain why it’s highly unlikely that there’s a large percentage of undetected click fraud in Google’s advertising network.

First of all Google has a number of signals they can watch to determine if click fraud is happening. Shuman mentioned that there’s well over a hundred data points they look at, including overall ROI rates, impression rates, click stream activity, click patterns, IP detection and that’s just a few of them. Also, Google is very, very good at building systems. Their engineers are the best in the world. So if they throw their collective brain power at a problem, you can be pretty confident that they’re going to come up with a robust solution to that problem. Click fraud was one of the biggest threats that Google faced in the last few years. They knew they had to restore advertiser confidence around the click fraud issue. So they threw their full engineering horsepower at the problem to build the filters that they currently have in place. This is the first line of defense against click fraud. The vast majority of the invalid click activity that’s happening in the Google network is caught by the filters. That’s the first screen that this activity would have to pass through.

The second screen is Google’s post-click review screen. This is where they look at questionable activity that made it through the proactive filters, do further investigation, and if they feel it’s warranted, they will go back and make a refund to the advertiser without the advertiser having to take any action at all. Again this is a very robust program that Google has put in place. This represents the second screen that fraudulent activity would have to get through.

The third screen in the advertiser themselves. Think about this. We have a lot of very sophisticated advertisers who have put robust analytics in place and have a deep, inherent understanding of what their website traffic patterns should look like. These advertisers have also been exposed to the so-called reporting of click fraud, like the BusinessWeek expose. They have a heightened sensitivity to click fraud so they would be very vigilant, particularly on any traffic that was coming from Google. So undetected fraudulent activity would have to get past this screen as well.

Finally, as an overall metric, Google aggregates the conversion data from advertisers who have Google analytics in place and uses that as a baseline of what typical behavior across the network should look like. In aggregate form the data allows them to check out anomalies in the dataset that may indicate fraudulent activity. This level of detection is over and above all the other fraud detection I previously mentioned. It acts as a monitor on the overall activity that could potentially indicate undetected click fraud in the network. So the likelihood of there being a significant amount of undetected click fraud is very, very low. Once again, so low it’s probably not worth spending much time worrying about.

The gist of my column today in SearchInsider is advising advertisers to look at a much bigger picture than just focusing on click fraud. I realized Google had to release these numbers because everyone was asking for them. If we can accept those numbers than perhaps we can get on with looking at our overall campaign performance and really spending some time on the things that would have a much greater impact on our overall return on investment. For example, the drum that I will continue to beat as long as anyone is willing to listen is for advertisers to focus on their own conversion rates. Time after time, I see landing pages that aren’t optimized and aren’t aligned to the intent of the potential visitor. I see sloppiness in advertising messages with a lack of relevancy aligned to the queries that are used. If advertisers paid more attention to these things they’d realize far greater benefit than they would by fretting over click fraud.

Don’t Think Click Fraud, Think Negative ROI

First published March 1, 2007 in Mediapost’s Search Insider

The search engines have a dilemma on their hands when it comes to click fraud. We’re all clamoring for more information on the issue. We all want solid numbers to help us define the scope of click fraud. The very fact that we refer to it as click fraud is confusing. A lot of things get thrown in the click fraud “basket” that are in no way fraudulent. Thanks to sensationalist reporting by publications like BusinessWeek, click fraud is portrayed as the biggest scourge to threaten the Nirvana that is search marketing. A tremendous number of resources have been dedicated towards click fraud by the engines themselves, in response to the advertisers’ demand that the problem be stamped out.

But when you do an honest appraisal of the issue, the search engines would rather we get over our preoccupation with click fraud and start thinking of it as part of a much bigger whole, the return we get on our search marketing investment. This in no way negates the importance of click fraud as an issue. I don’t think there’s anyone more aware of click fraud than Shuman Ghosemajumder (Google), John Slade (Yahoo), and Brendan Kitts (Microsoft). They’re the first to say that click fraud does exist and that they’re each, in their own ways, actively policing it.

It’s more a question of proportional response, an appropriate amount of attention given the actual scope of the issue. And today, for the first time, Google is giving us concrete numbers on what that scope might be, at least for its network. Google is announcing a multiphase approach and product road map to handle the click fraud question. Accompanying the announcement are hard numbers, for the first time, about how much of Google’s traffic could actually be considered fraudulent. I’ll talk more about the numbers in a moment, but first, let’s explore the dilemma that presents itself to the engines.

Caught Between an Over-Hyped Threat and an Ignored Danger

The engines know that, as a factor that negatively impacts return on search marketing investment, click fraud represents a tiny percentage. There are far bigger drains on the performance of campaign that advertisers should be paying significantly more attention to, but thanks to doom and gloom “exposés,” there’s a disproportionate amount of attention focused on click fraud. So, although the engines would rather advertisers focus more on the big picture and consider all the factors, including fraudulent traffic, that are negatively impacting their return on that investment, they’re playing the game they have to and are keeping the focus on click fraud. Google’s announcement today may allay some of the “sky is falling” concerns that are being whipped up by journalists, but in the long run it may do the advertisers a disservice by diverting attention from more pressing campaign optimization issues.

I’ve talked about some of this before, but here are some of the issues I have with the current click fraud situation:

Just Because We Call It Fraud Doesn’t Make It Fraud

Click fraud seems to be the label that has stuck with this particular issue. There have been calls to try to put numbers around the occurrence of click fraud in search marketing. In reality, it’s not that cut-and-dried. First of all, fraud implies that someone loses money through the deliberate actions of someone else. For a click to be fraudulent, at least in the way that BusinessWeek tried to define it, advertisers have to lose money. They have to be paying for traffic that has no value.

Less than 10% are Invalid Clicks

The fact is, there are a number of factors that may result in traffic that the advertiser would probably prefer not to pay for. Fraudulent traffic is just one of them. Google puts all this traffic into a basket they call invalid clicks. This includes double clicks on ads, questionable activity from a single IP address, automated clicks, and yes, clicks from the nefarious click fraud perpetrator. In today’s release, Google said invalid clicks accounted for less than 10% of its total network traffic. The company didn’t want to get more specific than this, because the actual percentage can rise and fall with a fair amount of volatility, based on spikes in clickbot attacks and other factors. Google works to filter this traffic out proactively, so it’s as if the clicks never happened. The advertiser is never charged for this traffic. In most cases, the publisher of the site from which the traffic is generated is never paid for the traffic. No money changes hands, so no fraud has been committed. If anyone is out of pocket, it’s Google, not the advertiser.

The Bottom Line for Advertisers? .02%!

The traffic that the advertiser should be concerned about is the fraudulent traffic that slips through the cracks. This is truly click fraud. It’s not caught by the Google filters and it’s up to the advertiser to come back and report it and request a refund. In this case, money has changed hands and fraud has been perpetrated. Today, Google announced that this represents .02% of its total traffic. Some time ago I did a column after a talk with Shuman at Google, and after making some assumptions and extrapolating the number, I came out with a “worst case” estimate of .2%. It appears that my worst case was much higher than reality, by a factor of 10X.

I don’t know about you, but frankly, if something is only making a .02% impact on my advertising campaign, I’ve probably got better places to be spending my time. One place you might want to look? The conversion rates of your landing page. If you can bump your conversion rates by .5%, you’ve just made 25 times more impact on your overall campaign performance than by continuing to fret about click fraud on Google.

Google’s announcement today was more than just releasing numbers on the occurrence of click fraud. It is also announcing the creation of a Click Fraud Resource Center, a streamlined reporting process, the ability for advertisers to filter out questionable IPs, more details in its nvalid click reporting and some other initiatives. I believe all these things are good and are needed by advertisers, if only to put to bed the perceptions of click fraud as a major issue. But do me a favor, will you? Take some of the time you may be spending worrying about click fraud, and start looking at all the other places where your return on investment may be slipping through the cracks. My guess is there a lot bigger cracks you should be looking at than the click fraud one.

Webpronews Video: Who Said What?

I happened to be browsing through Webpronews on the weekend and saw one of their new video news updates. The clips are well produced, professional looking and even have their own attractive newscaster, Nicole Eggers. One I happened to pick, however, left a little to be desired on the accuracy front. As you’re probably aware, I just did a series of interviews with the top usability people at each of the three engines for Search Engine Land and a couple weeks ago I did a recap talking about the differences I saw between each of their philosophical approaches. The blurb on the video appeared to be on the same topic so I decided to give it a watch. If it, Webpronews indicated that search expert Danny Sullivan had talked to each of the three usability people at the engines and had come to the following conclusions:

  • That relevancy was almost a religion for Google
  • Yahoo had a heightened sensitivity to the needs of their advertising community
  • Microsoft was still finding their competitive niche

Huh? That’s exactly what my recap said. They even pulled a few quotes from it and attributed them to Danny. I quickly e-mailed Danny to see if we were doing some kind of weird Cyrano de Bergerac thing but Danny was apparently as out of the loop on this as I was. Anyway a quick e-mail to Webpronews seems to have got it straightened out. They’ve pulled the clip and apparently they’re redoing it.

Not that I mind being mistaken for Danny, but I just hate to be putting words in his mouth. By the way, does anyone else feel like they’re being scolded by Nicole? Again, not that I mind.

A Caffeine Fueled Vision of the Future

This week, for some reason (largely to do with thinking I could still handle caffeine and being horribly wrong), a number of pieces fell into place for me when it came to looking at how we might interact with computers and the Internet in the future.  I began to sketch that out in my SearchInsider column today (more details about the caffeine episode are in it) , but quickly found that I was at the end of my editorial limit and there were a lot of pieces of the vision that I wasn’t able to draw together.  So I promised to put a post on this blog going into a little more detail.

The ironic thing about this vision was that although I’d never seen it fully described before, as I thought about it I realized a lot of the pieces to make this happen are already in development.  So obviously, somewhere out there, somebody also seen the same vision, or at least pieces of it.  The other thing that struck me was: it all made sense as a logical extension of how I interacted with computers today.  Obviously there’s a lot of technology being developed but if you take each of those vectors and follow it forward into the future, they all seem to converge into a similar picture.

Actually, the most commonly referenced rendering of the future that I’ve seen is the world that Spielberg imagined in his movie Minority Report.  Although anchored in pop culture, the way that Spielberg arrived at his vision is interesting to note. He took the original short story by Philip K. Dick and fleshed it out by assembling a group of futurists, including philosophers, scientists and artists, and putting them together in a think tank.  Together they came up with a vision of the future that was both chilling and intriguing.

I mention Minority Report because there are certain aspects of what I saw the future to be that seem to mirror what Spielberg came up with for his future.  So, let me flesh out the individual components and provides links to technology currently under development that seem to point this way.

The Cloud

First of all, what will the web become?  There’s been a lot of talk about Web 2.0 and Web 3.0, or the Semantic Web envisioned by Tim Berners Lee.  Seth Godin had a particularly interesting post (referenced in my column) that he called the Web4.  All these visions of the Web’s future share common elements. In Godin’s version, “Web4 is about making connections, about serendipity and about the network taking initiative”. This Web knows what we’re doing, knows what we have to do in the future, knows where we are at any given time, knows what we want and works as our personal assistant to tie all those pieces together and make our lives easier.  More than that, it connects us a new ways, creating the ad hoc communities that I talked about in my earlier post, Brain Numbing Ideas on Friday afternoon.

For the sake of this post, I’m calling my version of the new Web “the Cloud”, borrowing some language from Microsoft. For me the Cloud is all about universal access, functionality, connection and information.  The Cloud becomes the repository where we put all our information, both that which we want to make publicly accessible and that which we want to keep private.  Initially this will cause some concern, as we wrestle with the change of thinking required to understand that physical ownership of data does not always equal security of that same data.  We’ll have to gain a sense of comfort that data stored in online repositories can still remain private. 

Another challenge will be understanding where we, ourselves, draw the line between the data we choose to make publicly accessible and the data we want to keep for our own personal use.  There will be inevitable mistakes of an embarrassing nature as we learn where to put up our own firewalls.  But the fascinating part about the Cloud is that it completely frees us physically. We can take all the data we need to keep our lives on track, stored in the Cloud, and have it accessible to us anywhere we are. What’s more, everyone else is doing the same thing.  So within the Cloud, we’ll be able to find anything that anyone chooses to share with us. This could include the music they create, the stories they write, or on a more practical level, what our favorite store currently has in stock, or what our favorite restaurant has on for it’s special tonight.  Flight schedules, user manuals, technical documentation, travel journals…the list is endless.  And it all resides in the Cloud, accessible to us if we choose.

The other really interesting aspect of the Cloud is the functionality it can offer as we begin to build true applications into the web, through Web 2.0 technology. We start to imagine a world where any functionality we could wish for is available when we need it, and where we can buy access as required.  The Cloud becomes a rich source of all the functionality we could ever want.  Some of that functionality we use daily, to create our own schedules, to communicate, to connect with others and to manage our finances.  Some of that functionality we may use once or twice in a lifetime.  It really doesn’t matter because it’s always there for us when we need it.

The functionality of the Cloud is already under development.  The two most notable examples can be found in Microsoft’s new Office Live Suite and in the collection of applications that Google is assembling.  Although both are early in their development cycles, one can already see where they could go in the future.

The final noteworthy aspect of the Cloud is that it will create the basic foundation for all communication in the future.  Our entertainment options will be delivered through the Cloud.  We will communicate with each other through the Cloud, either by talking, writing or seeing each other.  We will access all our information through the Cloud.

For the Cloud to work, it has to be ubiquitous.  This represents possibly the single greatest challenge at the current time.  The Cloud is already being built, but our ability to access the Cloud still depends on the speed of our connection and the fact is right now, our wireless infrastructure doesn’t allow for a robust enough connection to really leverage what the Cloud has to offer.  But universal wireless access is currently being rolled out in more and more locations, so the day is drawing near when access will cease to be a problem.

So, when the Cloud exists, the next question is how do we access it?  Let’s start with the two access points that are most common today: home and at work.

The Home Box

The Home Box becomes the nerve center of our home.  It acts as a control point for all the functionality and communication we need when we’re not at work.  The Home Box consists of a central unit, which doubles as our main entertainment center, and a number of “smart pods” located throughout the home, each connected to a touch screen.

So, what would the Home Box do?  Well first of all, it would inform and entertain us.  The pipeline that funnels our entertainment options to us would be directly connected to the Cloud.  We would choose what we want to see, so the idea of channels becomes obsolete.  All entertainment options exist in the Cloud and we pick and choose what we want, when we want.

Also, the Home Box makes each one of those entertainment options totally interactive.  We can engage with the programming and shape it as we see fit.  We can manipulate the content to match our preferences.  The Home Box can watch four or five sporting events and assemble a customized highlight reel based on what we want to see.  The Home Box can scan the Cloud for new works by artists, whether they be visual artists, music artists or video artists, notifies us when new content is ready for us to enjoy.  If we have an interest that suddenly develops in one particular area, for instance a location that we want to visit on an upcoming vacation, the Home Box assembles all the information that exists, sorted by our preferences, and brings it back to us.  And at any time, while watching a video about a particular destination, we can tag items of interest within the video for further reference.  As soon as they’re tagged, a background application can start compiling information on whatever we indicated we were interested in.  Advertising, in this manifestation, becomes totally interwoven into the experience.  We indicate when we’re interested in something and the connection to the advertiser is initiated by us with a quick click.

But the Home Box is much more than just a smarter TV set or stereo.  It also runs our home.  It monitors energy consumption levels and adjusts them as required.  It monitors what’s currently in our fridge and our pantry (by the way, computers are already being built into fridges) and notifies us when we’re out of something.  Or, if there’s a particular recipe we want to make, it will let us know what we currently have and what we need to go shopping for.

Microsoft already has the vision firmly in mind.  Many of the components are already here.  The limited success of Microsoft’s Windows Media Center has not dissuaded them from this vision of the future.  Windows Media Center is now built into premium versions of the Vista operating system. And the is Smart Pods I refer to?  Each Xbox 360 has the ability to tap right into windows XP Media Center.  The technology is already in place.

The Work Box

Probably the least amount of change that I see in the future is in how we access the Internet at work.  For those who of us who work in an office environment, we’re already fairly well connected to the Internet.  The primary difference in this case would be where the data resides.  Eventually, as we gain comfort with the security protocols that exists within the Cloud, we will feel more comfortable and realize the benefits that come with hosting our corporate data where it’s accessible to all members of the organization, no matter where they are physically located.

But consider what happens for the workers who don’t work in an office environment.  Access to the Cloud now allows them to substantially increase their connectivity and functionality while they’re mobile.  You could instantly access the inventory of any retail location within the chain.  You can see if a parts in stock at the warehouse.  You can access files and documents from anywhere, at any time.  And, you can tap into the core functionality of your office applications as you wish, where ever you happen to be.

Once again, much of the functionality that would enable this is already in place or being developed.  In the last year we at Enquiro have started to realize the capabilities of Microsoft Exchange Server and Sharepoint services.  Just today, Google announced new enterprise level apps would be available on the web. Increasingly, more and more collaborative tools that use the Internet as their common ground are being developed.  The logical next step is to allow these to reside within the Cloud and to free them from the constraints of our own internal hardware and software infrastructure.

The Mobile Device

When we talk about tangible technology that will enable this future; hardware that we can see and touch, the mobile piece of the equation is the most critical.  For us to truly realize the full functionality of the Cloud, we have to have universal access to it.  It has to come with us as we live our lives.  The new mobile device becomes a constant connection to the Cloud.  Small, sleek, GPS enabled, with extended communication capabilities, the new handheld device will become our computing device of choice.  All the data and the functionality that we could require at any time exists in the Cloud.  The handheld device acts as our primary connection to the Cloud  We pull down the information that we need, we rent functionality as required, we do what we have to do and then we move on with our lives.

Our mobile device comes with us and plugs into any environment that we’re in.  When we’re at work, we plug it into a small docking station and all the files that we require are interchanged automatically.  Work we did at home is automatically uploaded to the corporate section of the Cloud, our address books and appointment calendars are instantly updated, new communications are downloaded, and an accurate snapshot of our lives is captured and is available to us.  When we get home again we dock our mobile device and the personal half of our lives is likewise updated.

Consider some practical applications of this:

When we go to the gym, our exercise equipment is now “Cloud” enabled.  Our entire exercise program is recorded on our mobile device.  As we move from station to station we quickly plug it into a docking station, the weights are automatically adjusted, the number of reps is uploaded, and as we do our exercises, appropriate motivating music and messages are heard in our ear. At the same time, our heart rate and other biological signals are being monitored and are being fed back to the exercise equipment, maximizing our workout.

When we’re at home, we quickly plug our mobile device into the Smart Pod in the kitchen, and everything we need to get on our upcoming shopping trip is instantly uploaded.  What’s more, with the functionality built into the Cloud, the best specials on each of the items is instantly determined, the best route to pick up all the items is send to our GPS navigation module, and our shopping trip is efficiently laid out for us. While we’re there, the built in bar code scanner allows us to comparison shop on any item, in the geographic radius we choose.

As I fly back from San Francisco, a flight delay means that I may miss my connecting flight in Seattle.  My mobile device notes this, adjusts my schedule accordingly, automatically notifies my wife and scans airline schedules to see if an alternative flight might still get me home without an unexpected layover near SeaTac Airport. It there’s no way I can make it back, it books me a room at my prefered hotel.

The Missing Pieces

I happen to think this is a pretty compelling vision of the future.  And as it started to come together for me, I was surprised by how many of the components already exist or are being currently developed.  As I said in the beginning, it seems like a puzzle with a lot of the pieces already in place.  There are some things, however, we still need to come together for this vision to become real.  Here are the challenges as I see them.

Computing Horsepower

For the mobile device that I envisioned to become a reality, we have to substantially up the ante of the computing horsepower.  The story that led to my writing of the SearchInsider column was one about the new research chip that is currently under development at Intel.  Right now the super chips are being developed for a new breed of supercomputer, but the trickle-down effects are inevitable.  Just to give you an idea of the quantum leap in performance we’re talking about, the chip is designed to deliver teraflops performance.  Teraflops are trillions of calculations per second.  The first time teraflops performance was achieved was in 1997 on a supercomputer that took up more than 2000 square feet, powered by 10,000 Pentium Pro processors.  With the new development, that same performance is achieved on a single multi-core chip about the size of a fingernail. This opens the door to dramatic new performance capabilities, including a new level of artificial intelligence, instant video communications, photorealistic games, multimedia data mining and real-time speech recognition.

A descendent of this prototype chip could make our mobile device several orders of magnitude more powerful than our most powerful desktop box today.  And when implanted in our Home Box, this new super chip allows us to scan any video file and pick up specific items of interest.  You could scan the top 100 movies of any year to see how many of them reference the city of Cleveland, Ohio (not exactly sure why you’d want to do this), or included a product placement for Apple.

Better Speech Recognition

One of the biggest challenges with mobile computing is the input/output part of the problem.  Small just does not lend itself to being user-friendly when it comes to getting information in and out of the device.  We struggle with tiny keyboards and small screens.  But simply talking has proven to be a remarkably efficient communication tool for us for thousands of years.  The keyboard was a necessary evil because speech recognition wasn’t an option for us in the past.  We can talk much faster than we can talk.

I recently was introduced to Dragon Naturally Speaking for the first time.  I’ve been trying it for about three weeks now.  Although it’s still getting to know me and I’m still getting to know it, when it works it works very well.  I found it a much more efficient way to interact with my computer.  It would certainly make interacting with a mobile device infinitely more satisfying.  The challenge right now with this is that speech recognition requires a fairly quiet environment, you’re constantly speaking to yourself, and mobile devices just don’t have enough computing power to be able to handle it.

We’ve already dealt with the computing horsepower problem above.  So how do we deal with the challenge of being able to get our vocal commands recognized by our mobile device? Let me introduce you to the subvocalization mic.  The mic actually picks up the vibrations from our vocal cords, even if we’re only whispering, and renders recognizable speech without all the background noise.  New prototype sensors can detect sub vocal or silent speech.  We can speak quietly (even silently) to ourselves, no matter how noisy the environment, and our mobile device would be able to understand what we’re saying.

Better Visual Displays

The other challenge with a mobile device is in freeing ourselves from the tiny little 2.5″ x 2 .5″ screen.  It just does not produce a very satisfying user experience.  One of the biggest frustrations I hear about the lack of functionality with many of the mobile apps comes just because we don’t have enough screen real estate.  This is where a heads-up display could make our lives much, much easier.  Right now they’re still pretty cumbersome and make us look like cyborgs but you just know we’re not far from the day where they could easily be built into a pair of non-intrusive eyeglasses.  Then the output from our mobile device can be as large as we wanted to be.

Going this one step further, let’s borrow a scene from Spielberg’s Minority Report.  We have the heads-up display which creates a virtual 3-D representation of the interface.  We could also have sensors on our hands that would turn that display into a virtual 3-D touchscreen experience.  We could “touch” different things within the display and interact with our computing device in this way.  Combined with sub vocalization speech commands, this could create the ultimate user interface.  Does this sound far-fetched?  Microsoft has already developed much of the technology and has licensed it to a company called eon reality.  Like I said no matter what the mind can envision, it’s probably already under development. As I started down this path, it particularly struck me how many of the components under development had the Microsoft brand on them.

If you can fill in other pieces of the puzzle, or you have your own vision of the future, make sure you take a few moments to comment.

Marissa Mayer Interview on Personalization

marissa-mayer-7882_cnet100_620x433Below is the full transcript of the interview with Marissa Mayer on personalization of search results. For commentary, see the Just Behave column on Searchengineland.

Gord: It’s a little more than two weeks ago since Google made the announcement that personalization would become more of a default standard for more users on Google.  Why did you move towards making that call?

Marissa: We’ve had a very impressive suite of personalized products for awhile now: personalized homepage, search history, the personalized webpage and we haven’t had them integrated, which I think has made it somewhat confusing for users. A lot of people didn’t know if they had signed up for search history or personalized search; whether or not it was on.  What we really wanted to do was move to a signed in version of Google and a signed out version of Google.  So if you’re signed in you have access to the personalized home page, the personalized search results and search history.  You know all three of those are working for you when you’re signed in.  And if you’re signed out, meaning that you don’t see an email in the upper right hand corner that personalized search isn’t turned on.  If anything, it’s a cleaning up of the user model, to make it clearer to users what services they’re using them and when they’re using them.

Gord: But some of the criticism actually runs counter to that.  One of the criticisms is that it used to be clearer, as far as the user went, when you were signed in and when you are signed out.  There were more indicators on the Google results page whether you were getting personalized results or not.  Some of those have seemed to disappear, so personalized results have become more of a default now, rather than an option that’s available to the user.

Marissa: If you think about it as default-on when you’re signed in, I think that it’s still as clear on the search results page.  We removed the “turn off the personalized search results” link, but you still see very clearly up in the upper right-hand corner whether or not you’re signed in, your e-mail address appears, and that’s your clue Google has personalized you and that’s why that e-mail address is there.  I do think, based on our user studies and our own usage at Google, that we’ve made the model clearer.  We were actually ended up at the stage with our personalized product earlier this year where, at one point, Eric (Schmidt) asked “am I using personalized search?”  And the team’s answer as to whether or not he was currently using it was so complicated that even he couldn’t follow it.  You’d have to go to “my account”, see whether or not he was signed up for personalized search, make sure that your toggle hadn’t been turned off or on, and there was no way to just glance at the search results page and easily tell whether or not it was invoked.  So now it’s very easy, if you see your username and e-mail address up in the upper left-hand corner, you’re getting personalized results and if you don’t, you’re not.  So effectively there are two parallel universes of Google, per se.  One if you’re signed out where you see the classic homepage and the classic search results and one where you’re signed in, where you get the personalized home page and…you’ll be able to toggle back and forth, of course…and then the personalized search results page and the search history becomes coupled with all that because that’s how we personalize your search.

Gord: So, to sum up, it’s fair to say that really the search experience hasn’t changed that dramatically, it’s just cleaning up the user experience about whether you’re signed in or signed out and that’s been the primary change.

Marissa: That’s right.  Before you could be signed in and be using one of the three products or two of the three products but not all and, of course, because people like to experiment with a new product, they forget whether they signed up for personalized search.  Had they signed up for search history?  This just makes it cleaner.  If you’re signed in you’re using and/or have access to all three, if you’re signed out, you’re on the anonymous version of Google that doesn’t have personalization.

Gord: We can say that it cleans up the user experience because it makes it easier to you know when you’re signed in or signed out, but having done the eye tracking studies, we know that where the e-mail address shows is in a location that’s not prominently scanned as part of the page.  Do the changes mean that more people are going to be looking at personalized search results, just because we’ve made that more of a default opt in and we’ve moved the signals that you’re signed in a little bit out of the scanned area of the page.  Once people fixate on their task they are looking further down the page.  This should mean at a lot more people are looking at personalized search results than previously.

Marissa: Actually, I don’t think it will change the volume of personalized search all that much, not based on what we’ve seen on our logs and usage.  It makes it cleaner to understand whether or not you’re using it and I do think that over time, what it does is it pushes the envelope of search more such that you expect personalized results by default.  And we think that the search engines in the future will become better for a lot of different reasons, but one of the reasons will be that we understand the user better.  And so when we think about how we can advance towards that search engine of the future that we’re building, part of that will be personalization.  I do think that when we look five years out, 10 years out, users will have an expectation of better results.  One of the reasons that they have that expectation is that search engines will have become more personalized.  I think that in the future, working with the search engine that understands something about you will become the expectation.  But you’re right in that we believe that for users that are signed in, who find value in the personalized search results, over time as those users know they are signed in and that there search history is being kept track of, that their search results are being personalized, and they don’t need to look at every single search task to see whether or not they are signed in because that’s what their expectation is and they’re expecting personalized results.  So I do think we won’t see a drastic increase of volume right now of the use of personalized search but that it will hopefully change the user’s disposition over time to become more comfortable that personalization is a benefit for them and it’s something they come to expect.

Gord: There are a number of aspects of that question that I’d like to get into, and leave behind the question of whether you’re signed in or signed out of personalized search, but I have one question before we move on.  We’ve been talking a lot about existing users. The other change was where people were creating a new Google account and they got personalized search and search history by default.  The opt-out box is tucked into an area where most users would go right past it.  The placement of that opt-out box seems to indicate that Google would much rather have people opting into personalized search.

Marissa: I think that falls in with the philosophy that I just outlined. We believe that the search engines of the future will be personalized and that it will offer users better results.  And the way for us to get that benefit to our users is to try and have as many users signed up for personalized search as possible.  And so certainly we’re offering it to all of our users, and we’re going to be reasonably aggressive about getting them to try it out. Of course, we try to make sure they’re well-educated about how to turn it off if that’s what they prefer to do.

Gord: When this announcement came out I saw it as a pretty significant announcement for Google because it lays the foundation for the future.  I would think from Google’s perspective the challenge would be knowing what personalized search could be 5 to 10 years down the road,  what it would mean for the user experience and how do you start adding that incrementally to the user experience in the meantime?  From Google’s side, you have invested in algorithmic work to categorize content online. I would think the challenge would be just as significant to introduce the technology required to disambiguate intent and get to know more about users. You’re not going to hit that out of the park on the first pitch. That’s going to be a continuing trial and error process.  How do you maintain a fairly consistent user experience as you start to introduce personalization without negatively impacting that user experience?

Marissa: I will say that there are a lot of challenges there and a lot of this is something that’s going to be a pragmatic evolution for us.  You have to know that this is not a new development for us. We’ve been working on personalized search now for almost 4 years. It goes back to the Kaltix acquisition. So we’ve been working on it for awhile and our standards are really high.  We only want to offer personalized search if it offers a huge amount of end user benefit.  So we’re very comfortable and confident in the relevance seen from those technologies in order to offer them at all, let alone have them veered more towards the results, as we’re doing today.  We acquired a very talented team in March of 2003 from Kaltix.  It was a group of three students from Stanford doing their Ph.D, headed up by a guy named Sep Kamvar, who is the fellow who cosigned the post with me to the blog. Sep and his team did a lot of PageRank style work at Stanford.  Interestingly enough, one of the papers they produced was on how to compute PageRank faster.  They wrote this paper about how to compute page rank faster and it caused a huge media roil around the web because everyone said there are these students at Stanford who created an even faster version of Google.  Because the press obviously doesn’t understand search engines and thinks that we actually do the PageRank calculation on the fly on each query, as opposed to pre-computing it.  Their advance was actually significant not because it helps you prepare an index faster, which is what the press thought was significant.  Interestingly enough, the reason they were interested in building a faster version of PageRank was because what they wanted to do was be able to build a PageRank for each user.  So, based on seed data on which pages were important to you, and what pages you seemed to visit often, re-computing PageRank values based on that. PageRank as an algorithm is very sensitive to the seed pages.  And so, what they were doing, was that they had figured out a way to sort by host and as a result of sorting by host, be able to compute PageRank in a much more computationally efficient way to make it feasible to compute a PageRank per user, or as a vector of values that are different from the base PageRank.  The reason we were really interested in them was: one, because they really grasped and cogged all of Google’s technology really easily; and, two, because we really felt they were on the cutting edge of how personalization would be done on the web, and they were capable of looking at things like a searcher’s history and their past clicks, their past searches, the websites that matter to them, and ultimately building a vector of PageRank that can be used to enhance the search results.

We acquired them in 2003 and we’ve worked for some time since to outfit our production system to be capable of doing that computation and holding a vector for each user in parallel to the base computation.  We’ve been very responsible in the way that we’ve personalized Search Labs and we also did what we called Site Flavored Search on Labs where you can put a search box on your page and that is geared towards a page of interests that you’ve selected. So if you have a site about baseball you can say you want to base it on these three of your favorite baseball sites and have a search box that has a PageRank that’s veered in that direction for baseball queries.

So, the Kaltix team has been really successful at integrating all these Google technologies and taking this piece of theoretical research and ultimately bringing it to life on the Web.  And as it’s growing stronger and stronger and our confidence around the Kaltix technology grew, we’ve been putting it forward more and more.  We started off on Labs through a sign-up process, then we transitioned it over to Google.com and now we are in effect leaning towards a model where for people who use Google.com and have a Google account, they get personalized search basically by default.  If you look at the historical reviews of the Kaltix work it’s gotten pretty rave reviews.  The users that have noticed it and have been using it for a long time, like Danny (Sullivan), they’ll say that they think it’s one of the biggest advances to relevance that they’ve seen in the past three years.

Gord: So when you the Kaltix technology working over and above the base algorithm, obviously that’s going to be as good as the signals you’re picking up on the individual.  And right now the signals are past sites they visited, perhaps what they put on their personalized homepage and sites that they’ve bookmarked. But obviously the data that you can include to help create that on-the-fly, individual index improves as you get more signals to watch.  In our previous interview you said one thing that was really interesting to you was looking at the context of the task you are engaged in, for example, if you’re composing an e-mail in Gmail. So is contextual relevance another factor to look at.  Are those things that could potentially be rolled into this in the future?

Marissa: I think so.  I think that overall, we really feel that personalized search is something that holds a lot of promise, and we’re not exactly sure of the signals that will yield the best results.  We know that search history, your clicks and your searches together provide a really rich set of signals but it’s possible that some of the other data that Google gathers could also be useful. It’s a matter of understanding how.  There’s an interesting trade off around personalized search for the user which is, as you point out, the more signals that you have and the more data you have about the user, the better it gets.  It’s a hard sell sometimes, we’re asking them to sign up for a service where we begin to collect data in the form of search history yet they don’t see the benefits of that, at least in its fullest form, for some time.  It’s one of those things that we think about and struggle with. And that’s one reason why we’re trying to enter a model where search history and personalized search are, in fact, more expected.  And I should also note that as we look at reading some of the signals across different services we will obviously abide by the posted privacy policies.  So there are certain services where we’ve made it very clear we won’t cross correlate data. For example on Gmail, we’ve made it very clear that we won’t cross correlate that data with searches without being very, very explicit with the end user.  You don’t have to worry about things like that.

Gord: One of the points of concern seems to be how smart will that algorithm get and do we lose control?  For example, when we’re exploring new territory online and we’re trying to find answers we’ve refine our results based on our search experience.  So, at the beginning, we use very generic terms that cast a very wide net and then we narrow our search queries as we go. Somebody said to me, “Well, if we become better searchers, does that decrease the need for personalization?”  Do we lose some control in that?  Do we lose the ability to say “No, I want to see everything, and I will decide how I narrow or filter that query.  I don’t want Google filtering that query on the front end”?

Marissa: I think it really depends on how forcefully we’re putting forth personalization.  And right now we might be very forceful in getting people to sign up to it, or at least more forceful than we were. The actual implementation of personalized search is that as many as two pages of content, that are personalized to you, could be lifted onto the first page and I believe they never displace the first result, in our current substantiation, because that’s a level of relevance that we feel comfortable with.  So right now, at least eight of the results on your first page will be generic, vanilla Google results for that query and only up to two of them will be results from the personalized algorithm.  We’re introducing it in a fairly limited form for exactly the reason that you point out.  And I think if we tend to veer towards a model where there are more results that are personalized, we would have ways of making it clearer: “Do you want to explore this topic as a novice or with the personalization in place?” So the user will be able to toggle in a different filter form.  I think the other thing to remember is, even when personalization happens and lifts those two results onto the page, for most users it happens one out of every five times.  When you think about it, 20% of the queries are much better by doing that, but for 80% of the queries, people are, in fact, exploring topics that are unknown to them and we can tell from their search history that they haven’t searched for anything in this sphere before. There’s no other search like it. They’ve never clicked on any results that are related to this topic, and, as a result, we actually don’t change their query set at all because we know that they need the basic Google results.  The search history is valuable not only because it can help personalize the results but they’re also valuable because we can tell when not to.

Gord: There’s two parts to that: one is the intelligence of the algorithm to know when to push personalization and when not to push personalization, and two, as you said, right now this is only impacting one out of five searches where you may have a couple of new results being introduced into the top 10 as a result of personalization.  But that’s got to be a moving target.  As you become more confident in the technology and that it’s adding to the user experience, personalization will creep higher and higher up the fold and increasingly take over more of the search results page, right?

Marissa: Possibly.  I think that’s one of many things that could possibly happen, and I think that’s a pretty aggressive stance.  I look at our evolution and our foray into personalization, where we’re sitting here three or four years in, with some base technology that several years old already and it still has been very slight in a way that we have it interact with the user experience.  Mostly because we think that base Google is pretty good.  As it becomes more aggressive, certainly I would be pushing for an understanding of the ability of the user to know that these results are, in fact, coming from my personalization and not background and if I want to filter them out and get back to basics, that that would be possible.  One thing that we’ve struggled with is if we should actually mark the results are entering the page as a result of personalization but because team is currently and frequently doing experiments, we didn’t want to settle on a particular model or marker at this exact moment.

Gord: The challenge there is as you roll more personal results into the results page and get feedback from some users that they would want more control over what on the page is personalized and the degree of personalization and introduce more filters or more sophisticated toggles, it complicates the user experience. And as we know, that user experience needs to be very simple. Is it a delicate balance of how much control you give the user versus how much do you impact the 95% of the searches that are just a few seconds in duration and have to be really simple to do?

Marissa: There are two thoughts there.  One, even if we introduce them to filtering on the results page, it wouldn’t be any more complicated than what you had two weeks ago, so we already have that filter.  Two, we put the user first, and people have varying opinions about whether their search results page is too complicated, but the same people who designed that user experience will be the people who will be tackling this for Google, so I think you can expect results of a similar style and direction.

Gord: In the last few weeks, Google has introduced some new functionality, related searches and refine search suggestions, that are appearing at the bottom of the page for a number of searches.  To me that would seem to be a prime area that could be impacted by personalization opportunities that are coming.  As you make suggestions about other queries that you could be using, using that personalization data to refine those. Is that something you’re considering? And how long before personalization starts impacting the ads that are being presented to you on a search results page?

Marissa: Refinement is an interesting but a neophyte technology from our perspective.  We are finally now just beginning to develop some refining technologies that we believe in enough to use on the search results page.  A lot of people have been doing it for a lot longer. When you look at the overall utility, probably 1 to 5% of people will click those query refinements on any given search, where most users, probably more than two thirds of users, end up using one of our results. So in terms of utility and value that is delivered to the end user, the search results themselves and personalizing those are an order of magnitude more impactful then personalizing a query refinement.  So part of it is a question of, it’s such a new technology that we really haven’t looked at how we can make personalization make it work more effectively.  But the other thing is on a “bang for the buck” basis, personalizing those search results get us a lot more.

And as to ads, I think there are some easy ways to personalize ads that we’ve known for some time, but we’ve chosen at this point to focus on personalizing the search results because we wanted to make sure to delivered the end-user value on that, because that’s our focus, before we look at personalizing ads

Gord: So, no immediate plans for the personalization of ads?

Marissa: That’s right

Gord: Thank you so much for your time Marissa.

I Have Seen the Future (Thanks to Regular Coffee)

First published February 22, 2007 in Mediapost’s Search Insider

Why do epiphanies always happen in the middle of the night? Why can’t they be more conveniently scheduled during regular business hours, say between 10 and 11 in the morning or right after afternoon break at around 3: 30? But no, they usually occur somewhere between 2 and 4 in the morning. The fact that I was in a semiconscious state for this particular epiphany has everything to do with the fact that we ran out of decaf at the office yesterday, and I figured I could squeeze in just one cup of regular coffee without serious side effects. I was wrong.

Intel’s New Super Chip

This particular epiphany was catalyzed by a short news story about the new research processor chip that Intel is working on. It promises to be a performance breakthrough of breathtaking proportions and while it’s destined for supercomputers, the trickle-down effect to our everyday computing requirements is inevitable. Moore’s Law just keeps rolling along.

So, I asked myself, sometime between 2:45 and 2:49 a.m., with processing power set to take another leap forward, where would this new technology change our lives the most? The answer: mobile computing.

More Horsepower for Mobile

Some time ago I wrote a column about my frustrations with the limitations of mobile computing as it currently sits. But if you can pack enough horsepower into your average mobile device to facilitate things like speech recognition and more robust support for virtual displays, the mobile computing experience becomes much less frustrating. And when that happens, our entire interaction with the Web changes with it.

Right now the majority of our access probably happens in two places: at work or at home. Mobile access is generally limited to checking e-mails right now, and even that is a truncated experience where we’re scanning subject lines to see if there’s any fires we have to put out.

Godin’s Web4

Another thread that went into the weaving of this epiphany was a post I read on Seth Godin’s blog about a month ago, a post he called Web4. In it, Seth talked about the Web as our personal assistant that helps shuffle our schedule, introduces us to new interests and businesses, and generally makes our lives better in a number of helpful ways. For the Web4 that Godin envisions to happen, our computers have to know where we are, always be connected to the Internet, have a quick and easy way for us to communicate with it, and generally fit our lifestyle much better than the current boxes on our desks, whether they be at home or at work.

Living in the Wireless “Clouds”

Here’s another thread. Microsoft’s Live suite has one purpose: to put the functionality of Microsoft apps at your fingertips no matter where you are, no matter what your connection to online is. It “unhooks” you from the desktop and lets you move around and live your life with wireless freedom.

Computing and online access have to fit us, not the other way around. There are times during the day when we tend to stick in one spot for a while. When that happens, it makes sense for us to have a static access point and computing platform with some of the advantages that a little more elbow room could offer. Two places that come to mind immediately: our workplace, and when we sit down at home to be entertained. The rest of the time our computer should move with us.

The Home Box

At home our computers could become the oft-predicted convergent box that provides our entertainment options, but does more than that. It plugs into our home-based activities and keeps them organized for us. It becomes a communications center, our security system, an energy usage monitor, a recipe book and shopping, but most important, it’s our primary link to all our information and entertainment alternatives, allowing us to interact with those alternatives in ways never previously possible.

The Work Box

If we tend to stay in one place at work, it also makes sense to have a static access point to our corporate networks and the Internet. But the minute we get up from our seat, a mobile device would become the access point and computing platform of choice. All the data and functionality that defines us, the things we want at our fingertips, have to travel with us. When you get home you quickly plug it into your home system and the required information would be quickly transferred and the necessary updates would be done. When you get to work, you plug it in to your corporate network and again the required work-related information would be seamlessly transferred. The rest of the time, this little engineering marvel that knows where you are, what you like and what you have to do today would become your primary connection to the wired world.

Search as the Common Thread

When you look at this always-on, always-wired lifestyle, one can only imagine the dramatic uptick that would happen in all types of search activity. Once again, search becomes the common thread that runs through all that. It’s what allows us at home to search through all our entertainment options and find precisely what we would like to watch or listen to right now. At work, it’s what allows us to sift through the mountain of corporate data that resides either on our internal network or on vast online data repositories to find the file we need right now. And when we’re out there, interacting with the real world, it’s our trusted shortcut to the relevant content on the Web.

I happen to think this vision of the future is pretty darn cool. Unfortunately I’m already pushing the editorial boundaries of this column. There still seems to be a fair amount of regular coffee coursing through my veins, so check out my blog for some additional posts on the topic.

Bill Wise Leaving Did-It

Apparently Bill Wise and the Did-It Frog have had a difference of opinion.  The story this morning says that Bill Wise will be stepping down from the CEO position at Did-It but future plans haven’t been announced.  The split seems to be amicable at this point, but the two spokespeople are Bill and Kevin Lee and they’re both very diplomatic gentleman.

The split seems to come about from differing views of where Did-It should go in the future.  This is the challenge facing many SEMs and SEOs now as the industry continues to evolve at breakneck pace.  Of course, Did-It is also at the eye of the storm regarding the SEO/SEM controversy.

I had the privilege of being able to speak on a panel once with Bill Wise and was impressed with his clear view of search marketing strategy and the industry in general.  He’ll do very well no matter where he ends up. Good luck Bill.

Brain Numbing Ideas on a Friday Afternoon

I can’t help but get the feeling that when we look at online marketing, we tend to get blinded by the technology and lose sight of what’s really important: how it affects people.

Right now there’s a flurry of attention surrounding YouTube because of copyright issues and other factors.  And YouTube isn’t alone in this.  The majority of things I did in my in box focus on technology.  What will be the next killer platform?  I see mobile search, I see online video, I see social networking. It’s hard to keep your finger on the pulse of what’s really important.  I find it useful to step back a little bit and see how these things affect real people: people not like you and I, who are caught up in the promise of technology, but people like my daughter’s principal, people like my mom, people like my next-door neighbor.  People who are wary about technology and who will only embrace it if it makes their life better in some way.  This is not to discount the importance of technology, because it truly has turned our lives inside out in the last decade.  But there’s a distillation, a time when we have to get comfortable with change.  The dotcom boom and bust was not because of the lack of technology or its inadequacy.  To technology all things are possible.  But to people, it’s all about what’s in it for me.  And that, ultimately, is the success factor that has to be considered in all this.

So, is YouTube hot?  Is online video hot?  Is social networking hot?  All these things are, but not because of the technology that lies beneath, but rather because of the social change that they empower.  Consider online video for example.  A couple of items in my in box talked about how, at this point, we won’t watch television online.  Even the person at Google who was responsible for online video admitted that at this point, even with Google’s tremendous resources, online video at the quality that we’ve come to expect is not a scalable proposition. 

We interact with video in a far different way online.  For example, YouTube is all about the viral spiral.  It’s all about that cute little two to three minutes of video: something that is either funny or outrageous or awful.  There’s no tremendous requirement for engagement for this.  YouTube is the repository for a million different “in” jokes.  It’s the basket where we collect what titillates the fancy of our collective consciousness at any given time.  It gives us an easy reference point so we can take what interests us and forward it to others if we think they are interested as well.  We’re not ready to watch a one or two hour documentary on the web, simply because we’re not used to interacting with our computer screen in that way.  Our computers are things we do things on, not things we watch passively.  A commitment of two to three minutes to watch a little video screen is fine, but we don’t look to the Web for passive entertainment.  That’s not to say we won’t, some day, as connectivity and convergence moves our channels beyond the current paradigm and as we evolve and learn to interact with them in new ways. 

And it’s there that we start to pick apart at what truly makes technology, at least as far as it’s manifested on the web, really interesting.  It stitches together the fabric of our society.  It’s a synapse that allows our collective brain to fire more effectively than it did before. Communications can zing back and forth between us at a far faster rate.  What we find interesting, what we find intriguing, what we find funny, what we find painful to watch is now available for anyone to see.  It’s cataloged and categorized for our convenience.  It occupies a finite space in the virtual world that we can point to and say, “Look at this, it impacted me and I think it will impact you to.”

I recently had the opportunity to watch Dr. Gary Flake from Microsoft talk.  He started his presentation with the claim that the information technology revolution that we’re currently in will be more significant, as far as the change factor for our society, than anything that has gone before.  More important than the Industrial Revolution, more important than the invention of the printing press, more important than television.  To me the real power of the Internet is that it’s rewiring our society in ways we could never dream of and in ways we never anticipated.  To focus on the wiring or the technology of the Web is to take the mechanic’s view of the world.  To a mechanic or a car buff, a vehicle is a wonderful thing because of the internal combustion engine, because of the horsepower and how fast it can go from zero to 60.  They focus on what it is.  But when you look at how the automobile has affected our society, it’s not about what it is, it’s about what it does.  The automobile brought the world closer.  It allowed us to travel and see new things.  It allowed us to live in one place and work in another.  The macro change that the automobile engendered had nothing to do with how an internal combustion engine worked, it came from moving people from one place to another quickly, cheaply and efficiently.  It mobilized our society in a way that never existed before.

Likewise, the Web is not powerful because of Web 2.0 technologies, or speed of connection, or the ability to host video.  It’s important because it connects us in new and different ways.  It moves power from where it was stuck before into new hands.  It breaks down existing power structures and distributes that power amongst all of us.  It puts the individual in control and allows one individual to connect with another, freely and without paying a poll to the previous power brokers.

The really interesting thing about the Internet is the underlying social current, the groundswell of change that is redefining us and how we live together.  These fundamental factors are exerting a tremendous force within our day-to-day lives.  They’re precipitating change so fast that we haven’t been able to step back and see what the full impact to us will be.  We can’t see the trickle down effect of the things that are happening to us today.  The Internet is changing the very DNA of our society, and we are unable to take a long-term view of what those current mutations will mean for us.  One only has to look at the generational difference between the 45-year-old parent, myself, and my 13-year-old daughter, the first generation that has been fully immersed in online technology.  She interacts with the world in a completely different way.  She searches for information in a different way and evaluates it differently.  She takes these things for granted because she’s never known any other way.  What happens when this entire generation emerges as the shapers of our society?  What happens when they take control from us, with their innate understanding of what the Web makes possible, and redefine everything?

Here are three things that I believe are the foundations of social change being pushed by the Internet:

Access to Information

The amount of information we currently have access to is mind-boggling.  Never has so much raw information lived so close to us.  You can now think about any given topic in the universe of our consciousness, and that information exists just a mouse click away.  And, as the saying goes, information is power.  It empowers each one of us to take a more active role in our destiny.  This information has completely changed how people buy things.  It’s completely changed the relationship between vendors and buyers.  More and more, we go direct to the source, as educated, knowledgeable buyers who know exactly what we want and what we will pay for it.  The challenge on the Internet is that not all information is created equal.  There’s good information and there’s bad information.  However, we are becoming extremely good at being able to differentiate between the two.  We’re becoming amazingly adept at being able to recognize authenticity and we can sniff out BS.  In picking through the multiple threads of information that are available to us out there, we can recognize the scent of truth and quickly discount hype, spin and sheer lies. 

Again, as we begin to recognize the shifting of power to the consumer, the full impact has not shaken out yet.  When we can buy anything online, quickly, easily and confidently, will what will that mean for the entire bricks and mortar retail world out there?  Will there be shopping malls in 20 years?  Will there be stores at all?  Will we buy directly from the manufacturers, cutting out distributors, wholesalers and retailers?  Or will distribution of products to the world of consumers lie in the hands of a few mega, long tail retailers such as Amazon?  I certainly don’t know, the future is far too murky to be able to peer down this path.  And I don’t think it’s important to be able to predict the future, but I do think it’s vitally important to consider the quantum change that is likely in the future.

Searchability

As the amount of information available to us continues to multiply exponentially, the ability to connect with the right information at the right time becomes more and more important.  I’ve always maintained that search is the fundamental foundation of everything that will transpire online.  It is the essential connector between our intent, and the content we’re looking for.  But more than just the connector, the sheer functionality of search, both as it is today and as it will be in the future, creates another catalyst for change in our society. 

We are becoming used to having the answers just a few mouse clicks away.  We are becoming a society of instant gratification.  In the past, we accepted that we couldn’t know everything.  In divvying up the world’s knowledge, some of us were experts in one area and some of us were experts in another.  Some of us were experts in nothing.  But we held no pretensions that we would become experts in areas where we had no previous experience.  There was no path to follow so there was no reason to start the journey. 

But today, you can become an instant expert in anything, depending on how you define the scope of that expertise.  Within 30 seconds I can tell you every movie that Uma Thurman ever appeared in.  I can look up a medical condition and have access to the same information, likely more information, that a doctor 20 years ago would have access to, based on his own experience, education and reference materials.  But again, what is the impact of this?  Does having access to the information about a medical condition makes me an expert in treating that condition?  I have the information but I have no context in which to apply it.  As we gain access to information, will we use that information wisely without the experience and domain expertise that used to accompany that information?

And how will instant access to information alter education in the future?  I remember hearing an observation that if we had a modern day Rip van Winkle, who had gone to sleep 20 years ago and suddenly woke up today, the one place he would feel most comfortable would be in the elementary classroom.  While the outside world is changed dramatically in the past 20 years, the classroom in which your child spends the majority of their day has changed very little.  When I help my children do their homework, there isn’t much difference between the textbooks and the worksheets I see today and the ones I saw 30 years ago.  I recently had to explain to my daughter’s principal the difference between a Web browser and a search engine.  The classroom is like a backwater eddy in the rushing torrent of technological change that typifies the rest of the world.  And it’s not just elementary school where this is an issue.  We often speak to students who are currently going through marketing programs at the university level and are always aghast at how little they’re learning about this new world of marketing and the reality of consumer empowerment.  They’re learning the rules of a game that changed at least a decade ago.

So to bring the point home once more, what will the organization of the world’s information mean for our society?  As search gets better at connecting us to the content that we are looking for, what are the ripple effects for us?  Will our children’s and grandchildren’s brains be wired in a different way than ours are?  Will they assimilate information differently? Will they research differently? Will they structure their logic in a different way?

Creation of Ideological Communities

The Web has redefined our idea of community.  It used to be the communities were defined along geographic lines.  You need a physical proximity to people in order to create a community because physical proximity was a prerequisite for communication.  Communities could exist if there was two way communication.  That’s the reason why community and communication are extensions of the same root word and concept. 

Perhaps the most powerful change introduced by the Internet has been the enabling of real, two way communication between people where physical proximity was not required.  Consider the chain of events that typifies online interaction.  You become aware of someone who shares an ideological interest, usually through stumbling upon them somewhere online.  You initiate communication.  Depending on the scope of your shared interest, you may create the core of the community by inviting others into it.  The Internet gives us the platform that allows for the creation of ideological communities.  We see this happen all the time on properties such as YouTube or MySpace.  Ideological communities are created on the fly, flourish for awhile, and then fade away as interest in the idea that engendered them also fades away.  The Internet, at any given point in time, is a snapshot of thousands, or perhaps millions, of these ad hoc ideological communities.  They form, they flourish and then they disappear.

But in our real world there was physicality to the concept of community.  The way our world is built, our political boundaries, come from physical considerations.  There are distinct geographic boundaries like mountain ranges, oceans and rivers that, in the past, prevented the flow of people across them.  Because of the restricted ability to move, people spent long enough together to share ideals and create communities.  As time moved on these communities became larger and larger.  Transportation allowed us to share common ideals over a greater expanse and nations became possible.  The more efficient the transportation, the larger the nation became.  But throughout this entire process, the concept of geography defined communities and defined nations.  Our entire existing political structure was built around this geographic foundation.

With the Internet, geography ceases to have meaning.  It’s now a virtual world, and I can feel closer to someone in China with whom I share one particularly strong mutually held belief then I might with my next-door neighbor.  More fundamentally, I can belong to several different communities at the same time.  Again, the restraint of the physical world usually restricted the number of interests we had that we could share with those immediately around us.  Our sphere of interest as an individual was somewhat dictated by the critical mass each of those interest areas had within the community in which we lived.  If we thought particularly strongly about one interest we could physically move to a community where there were more people who shared that interest.  So we tended to move to communities that felt “right” ideologically as well as physically.  But with the Internet, does that need for ideological “sameness” where we live eventually disappear?  Does our physical need for community decrease as our ideological need for community is fulfilled through the Internet?

And, if this physical definition of community begins to erode, what does that do for the concept of nationhood and all the things that come along with it? Increasingly, communication and commerce travel along lines not defined by geography.  The idea of a nation, as we currently understand it, is inextricably bound to the realities of geography.  Politics, trade, laws and defense are all concepts that are rooted in thinking developed over the past several centuries.  In the past 30 years we’ve seen the erosion of the concept of nationhood through the creation of common markets and free trade areas.  The very breakdown of the Soviet Union comes from the inability to isolate the population from the concepts which flourished in the free world.  And that was before the Internet ever became a factor.  What happens when we take this movement, already afoot, and add the tremendous catalyst that is the Internet?

It’s in these macro trends that the true power of the Internet can be seen.  It’s not about an individual technology or even the cumulative power of all the technology.  It’s about how the sum of all that affects us as individuals, how we interact with the world around us and how we connect with other individuals.  The seeds have been planted, we can’t turn back, and we can’t foresee what will be.  The world is evolving and truly becoming a global community.  We are entering a time when change will accelerate faster than our society may be able to keep up.  There will be costs, certainly, but my hope and belief is that the rewards will far outweigh the costs.

The Inevitability of Personalized Search

First published February 15, 2007 in Mediapost’s Search Insider

Google’s announcement a little more than a week ago that it would be showing personalized search results to more people through a change in the sign-in/sign-out default signaled perhaps the most significant change in search marketing in the past few years. Fellow Search Insider David Berkowitz dealt with some of the SEO implications in his column on Tuesday. Today I’d like to deal more with the user side of the story. Although Google’s announcement heralds a relatively minor change in terms of user experience, at least for the present time, it represents a step down a path from which there is no return. This path marks a dramatically different direction for search that will have far-reaching implications, both for advertisers and users.

Google Gets Personal

First, a brief recap of Google’s announcement and what it means to users right now. Here are the details: Now, everyone signing up for a Google account gets Search History enabled by default. The opt-out box is positioned so that most people would likely not even notice it during the sign-in process.

Whether or not you have Search History enabled, you get personalized search turned on by default. This means that Google will subtly change your results, based on various “signals,” like what you have on your personalized Google Homepage and what sites you’ve bookmarked as Google favorites. Of course, if you have Search History enabled, this is the main “signal” for personalized search

 

Finally, and probably least significantly, everyone gets his or her own Google Home Page when s/he signs up for a Google account.

The End of One Page for All

Let’s leave aside the privacy issues of Search History right now. That’s a topic that deserves a column by itself. It’s the end of the universal search results page that I want to touch on today.

There has been significant dissent voiced about Google’s move to personalized search, and it’s coming primarily from one source: search engine optimizers. In opposing personalized search, they’re saying it degrades the user experience. I responded by saying that it was the wrench that personalized search throws into their SEO plans that was raising their ire. But let me set aside my jaundiced view of the search world for a moment and chronicle its concerns (excluding privacy issues), as near as I can understand them:

 

  • Taking control away from the user by making personalized search a default and making it more difficult to toggle on and off 
  • Fear of anomalous browsing patterns (i.e. going to visit a number of humor sites on a whim or the invite of a friend) unnaturally biasing search results 
  • The “machine learning” algorithms that power personalized search not being smart enough to really provide more relevant resultsI’ve come out as saying that personalized search is inevitable; the day when all of us see the same page of search results is rapidly coming to a close. To me, this just seems obvious. But still, there are those that protest. Here’s one example from Michael Gray, a well know SEO Blogger: “I’ve never met a business owner who’s said, ‘Man, you know what, I wish the search engines could create anarchy by making sure no two people got the exact same results for the exact same search — that would be the best thing since sliced bread.'”

    In fact, Michael’s beef seems to be a consistently recurring theme among the dissenters, that a move to personalization suddenly seems to open the door for chaos on the results page. I believe the opposite is true.

    Every Search is an Island

    I am an individual, with unique interests, experiences, values and goals. My intent when I search for hybrid vehicles, or New York hotels, or Smart Phones, or any of the hundreds of other things I search for monthly, will be significantly different than all the other people that launch those same searches. I want a search engine smart enough to know that. I’ve always said that humans are complex, far too complex for a simple search box to get it right. That’s why personalized search is inevitable. If we want search to move to the next level, to get smarter, more intuitive, more relevant, we need to leave standardized search results behind.

    Does this mean Google will get it right out of the box? No. It will take baby steps towards what personalization eventually needs to become (although I believe those steps will be in rapid succession, because Google can hear the competition hard on its heels). Yes, there will be many who find that in the early stages, personalization may be more frustrating than it is useful. But for search to mature, these are growing pains we’ll have to endure.

    I’ve been labeled as an early proponent of personalization. I’m not sure this is necessarily the case. To me, it’s not a question of liking or disliking the recent moves by Google. To me, fighting search personalization is as pointless as refusing to accept today’s weather.