Category Archives: Search Behaviors

Personalization: Google’s Defensible Trump Card?

A thought that came up in a conversation with Michael Ferguson, Ask’s usability guy (which is probably why I like talking to him. He always greases the mental machinery) was Google’s defensible position that personalization offers.

Google is betting the farm on personalization. And really, they’re possibly the only search engine that can make this work. Here are the required components:

  • A high enough degree of additional user value to convince people to opt in to personalization. As I’ve talked about before, that’s why it’s being rolled out in organic search first. Expect a slew of other value adds in the near future, all powered by personalization and all aimed and getting you to hit the opt in box.
  • An extensive network so you can maintain multiple touch points for the delivery of targeted advertising. Nobody has a bigger network that Google’s AdSense network
  • Critical mass amongst users. With Google’s almost 65% market share and the highest penetration of installed tool bars (42% plus in a recent B to B study we did), Google also has the required components to tap into a significant slice of the available market. And future Gadgets and tools will likely either require personalization to be turned on, or will provide an enhanced level of functionality when they are. Expect Google to get aggressive with forcing adoption in the next year or so.

It came to light when I was talking to Michael about Ask’s algo and whether personalization will play a part (by the way, this is part of an interview that will be on Search Engine Land next week). After the interview, I realized it’s not an option for Ask, at least not at the level that Google’s contemplating. Even if they did move to personalization, they just don’t own enough of the total online user experience to push them to opt into personalization. They’d never gain the critical mass needed to make it work.

Microsoft has an outside chance through Messenger, but it would be a long shot. Yahoo also has a long shot at it (although better than Microsoft’s) but they’d have to start gaining market share, and there are a number of huge obstacles in their way. Google is by far the best bet to force personalization on the market and have it be adopted at significant rates.

So what are the options for the other engines? Well, again, there’s an interesting twist there as well. One thing that’s touted heavily by the contenders is social search. I have severe doubts about the scalability of anything that requires a human element, and I’ve written about this in the past. But then I realized that personalization gives Google social search in a way that others just can’t touch.

If Google is collecting both web and search history, they’re collecting implicit votes for the quality of every property on the web. They create their own community, and with every click, that community votes for the quality and relevance of every site they visit. It’s social search in a very powerful and completely transparent form. In this form, social search requires no additional action on the part of the user (one of the critical risk areas of social search) and is completely scalable, because there’s no human bottleneck (the other critical risk area).

The more I think about personalization, the more I think that Google has just trumped the entire search space…again.

The Three Cs of Search

First published May 10, 2007 in Mediapost’s Search Insider

Since most of the Search Insiders are in Bonita Springs this week, chances are that you’ll be hearing a lot of what’s happening down here in the Florida Everglades (other than the brush fires which appear to have us surrounded). Aaron Goldman shared his Buzz-o-meter with us on Tuesday, where he measures the words that seem to be dropped with the greatest frequency. It appears that my opening remarks set a tone that has been picked up in a number of sessions, and two words breaking into the top 10 are “connection” and “community.” Aaron added a third “c”: “content.”

To me, these words sum up a transition that’s happening in search. Expect the activity of searching on a search engine to gradually disappear, to be replaced with the functionality of search as an underpinning to the workings of many things on the Web. Search will become the engine that drives the semantic web, which Esther Dyson talked about in her keynote. She’s looking for search to move beyond “search and fetch” to her ideal, “deliver, act and transact.”

Search will be the connector between what we want and what best matches our want out there on the Web. And rather than a singular task (i.e. go look for this query) it will become a self-guided series of tasks, with intelligent agents in between to set search on its new direction. An entire trip, include flight reservations, hotel bookings, ground transportation, notifications of friends in the area and restaurant reservations, could be booked by intelligent Web agents, powered by search. And as came up in a panel discussion with the Search Insiders, when the presentation of commercial messaging appears in this context, it’s not advertising, it’s a helpful recommendation.

The piece that drives this is personalization, and that’s why Google’s moves are potentially so important. They take us much closer to the semantic web that Dyson envisions. This is the first “c”: connections.

Redefining Community

The second “c” speaks to the very transformation of our society: community. The way we relate to each other is being totally rewired by the Internet. By sheer physical necessity, communities have previously been defined by geography. We shared a common space, which enable communication, which created community. But today, the Internet has made physical distance irrelevant. Our communities are now defined by commonly held ideas or interests. Communities form around ideas, and search connects us to those communities. Every time we do online research for a product or service, we step into a community. In the course of a day, we can belong to several different communities. They are constantly shifting, as people move in and out of them, depending on the longevity of the engagement with the idea that forms the community.

Content Trails

And a third “c,” content, is the trail that the other members of that community leave behind through their conversations. These are the telltale signs that someone has already gone this way, and left a permanent record of his or her engagement with the community. Every Wikipedia entry is part of a community, as are many MySpace pages, blog posts and other virtual outposts. Search is the thread that loops them together at the user’s initiative. In fact, the algorithm of the engine is the de facto definer of community with each given search. The engine goes out, defines the landscape of community, and connects you with the citizens of that community and the content trails they leave behind.

It’s a fascinating world, which is being born as we speak. It’s a sociological experiment of vast magnitude in the making, and I don’t think we know what the repercussions will be. Whatever they are, it’s too late to turn back now. Technology moves fast, but people move slowly, and not in one mass. Small degrees of technological change can create seismic shifts in the sociological landscape. And we’re subjecting ourselves to a degree of technological change unparalleled in history. Who knows what we’ve unleashed?

 

Thoughts on Yahoo and Microsoft Merging

Note: This was actually written on Friday, but I haven’t had a chance to post it til now. I’ve been travelling and access has been an issue. But I just came back from the opening reception at the MediaPost Search Insider Summit and the latest seems to be that the hype of this deal is far ahead of any actual discussions. That said, I think my comments are still valid, because as we’ve learned, things can happen fast in this industry.

Friday, May 4

The big news this morning as I was burning off some calories on the stair climber was the possible acquisition of Yahoo by Microsoft. I was actually in New York when I heard the story break, and one of my meetings today was at the Microsoft New York office, so I thought it would be interesting to ask my contact there what she thought. She indicated that this story has been going on for years now, but apparently they’re going back to the table. As we were chatting in a conference room, someone walked by outside asking somebody else if they had bought Yahoo stock. The media speculation was good news for Yahoo stock, not so for Microsoft.

Obviously, there’s a lot to mull over here. Rumor has it that Steve Ballmer is not taking Google’s DoubleClick scoop lightly. In fact, he’s downright pissed. And he may be preparing to make Terry Semel an offer he can’t refuse. Semel’s played hard to get before, but this time the shotgun marriage just might take.

The obvious question is how the two search properties will combine. In this case, it might be a case of two wrongs not making a right. Yahoo has managed to keep their search share from eroding too badly with Google’s domination, but Microsoft has been sputtering out of the starting gate from day one. The problem is that Yahoo and Live search duplicate each other in many ways, rather than complement each other. The biggest problem with both engines is too much focus on revenue generation and not enough on user experience. They each have their different flavors, but the combined Microhoo (or YahSoft) is in no way a Google killer. In fact, with the turmoil of a merger and the inevitable awkwardness of combining search teams, I see the focus on the user suffering even more. Both engines desperately need a clearly focused user champion to revamp the search experience (ala Google’s power usability troika, Larry, Sergey and Marissa) and this deal just doesn’t produce that.

I think the rationale of the deal has much less to do with search and more to do with a rather petulant online land grab. Yahoo does bring some interesting assets into the Microsoft fold. Microsoft is definitely eyeing the Asian market, and Yahoo has dominates in most of these markets, with the exception of China, and that’s a whole other story. Yahoo also brings a lot of users and online real estate as well, with roughly double Microsoft’s user base. This move looks like a strategy to bolster the front line for a head to head confrontation with Google in the ad serving space. Of course, it could just be the Ballmer has a lot of cash burning a hole in his pocket and everytime he goes to spend it, Google snatches the acquisition away from him. Steve wants to buy a ball he can actually take home.

One really interesting aspect of this is what it will do in the search space. While I really don’t think Yahoo’s search assets are the impetus for the deal, the potential combining of Live Search and Yahoo cleans up the search landscape a bit, and my guess is there will be significant user fall out from this. This will not be good news for the users of these two engines in the short term. But it could be extremely good news for Ask.

I just did an interview with Michael Ferguson, Ask’s usability point person (coming in Search Engine Land next week) and the IAC team are doing some really smart and relatively innovative things with their engine. And they’re probably the least aggressive in jamming ads on the page right now. Diller has provided a big enough bankroll to allow Jim Lanzone and his team to take a long run at capturing marketshare and this just may be the break they need. Based on what I’ve seen, Ask is paying a lot of attention to the user experience, and they may well pick up some converts and some pretty significant marketshare lift because of that. Perhaps Microsoft employees should be eyeing IAC stock. Or perhaps Steve Ballmer is starting to jot them down on his shopping list. After all, Google will probably scoop Yahoo out from underneath him at the last minute anyway!

More Food for Thought on Google’s Web History Announcement

Yesterday’s announcement from Google about including Web history in search personalization marks a fairly significant development in disambiguating intent on Google.  Consider the implications.  One of the issues I had with the initial implementation of search personalization was that it really only worked when there was existing search history.  That really only covered one in five searches for most of us.  That also meant that personalization showed up most often in areas where you tended to do a lot of searching.  For example, if you search within your industry a lot and tend to go to the same sites over and over again, you would find the site lifted on to your top page of search results.  Of course, if you were doing the typical “vanity” search to see where you rank and you end up clicking on your own site, this would have the effect of lifting your site into the top 10 results.  If anything, this implementation of personalization works to make navigation search a little more efficient.  But I’m not sure it went too far in disambiguating intent, which is the holy grail for any search engine.

With the introduction of Web history, it’s a whole new ballgame in disambiguating intent.  This allows Google to move far beyond the well tred search path and actually taps into your current browsing behavior to try to determine what’s on your mind right now.  If Sep Kamvar’s personalization algorithm is as powerful as I suspect it is, this could dramatically alter the results that you’re seeing.  The promise of personalization is greatest when it can be applied in areas that are new territory for you.  It helps Google interpret just the kind of site you want to see, given your behavior at the present time.

Let me give you an example.  Let’s say you’re looking at buying a new vehicle.  Let’s further say that you’re fairly early in the consideration phase and your visit a lot of sites like Edmunds.com and Autobytel.  This tells Google that you’re looking for information and you’re probably looking at sites that could be comparing your alternatives.  If you’ve already visited sites like Edmunds.com, Google would probably lift those sites into the first search results page.  If Google’s algorithm truly makes a move towards a recommendation engine, what it can then do is find similar sites you may have never considered, based on the characteristics of the sites you have been visiting and make you aware of these sites.  That’s where the real win for the user comes in personalization.  It’s not just providing you a shortcut to sites you are already aware of, it’s in making you aware of new sites you never knew existed, ranked and prioritized according to the PageRank algorithm.  With Web history, Google can track your progress through the buying cycle to be able to match the information site you’re looking for to where it believes you are, based on your current click stream data.

There are other implications that are very interesting to advertisers.  Click paths tend to indicate the life events that you’re currently in the middle of it.  The life event could be a major purchase, planning a holiday, buying a new house, planning for a wedding, or graduating from university.  In each of those instances, there are a number of linked consumer needs that tend to go together.  There’s been a significant amount of research done on how life events generate predictable consumer patterns.  Web history gives Google a window into exactly what is happening in your life right now.  I had written a column about how surprised I was with the glimpse that search history provided into my mindset at any given time.  If you combine that with Web history, you would have a very finely detailed snapshot of both big and small events in my life for any time period.  It gives Google the ability to precisely target search results based on exactly what’s happening to me right now.

But let’s face it, it’s not the search results that Google is focusing on.  Google is altruistic enough to make organic search results the testbed to play with the personalization algorithm, but the monetization opportunities in this are mind-boggling to say the least.  When you combine the ability to precisely target and interpret the mindset of any given consumer with the multiple touch points that Google now owns to provide advertising messaging to that prospect, you have a marketer’s dream scenario.  When I asked Marisa Mayer about this she made it clear that organic results are what they are working on now, but they don’t want their advertising network to be too far behind the curve.  I’m still working my way through the interview making notes but I did want to get this post up because I think from a user perspective there’s some important information here.  For me, the promise of personalization is moving Google to be a true recommendation engine when it gets confident in disambiguating my intent based on my current behavior.  Folding Web history into search history moves Google a quantum leap forward in being able to do this reliably and consistently.

The interesting question will be to see what kind of user pushback comes from the privacy concerns.  Danny Sullivan touched on this a little bit in his post.  Will the trade-off of increased search accuracy be enough to have lots of users opt in? Obviously this is what Google is counting on and that’s why they’re introducing the enhancement in the organic results first.  If they can provide a clear win to the user, than the trade-off seems a lot less formidable.  And when they’re introducing that usability lift in something as benign as organic search results, it seems a little less ominous and invasive.  If they can get us using Web history by giving us a win-win on our search functionality, is a greater likelihood that we’ll leave Web History turned for when they do decide to start rolling it in to their advertising presentation algorithms. Enough users will have it turned on it will give them the critical mass they need to appeal to the early adopter advertisers who want a take it for a spin.

Google Adds Your Click Path to Search Personalization

You know how when you install the Google toolbar and enable the PageRank feature, it gives you the warning on the EULA that this is not your typical legal  Yada yada?  Ever wondered what they were doing with all that information that’s being streamed back to a Google server somewhere?  Well, today Google announced just what they intend to do with it.  They’re going to use it to personalize your search results.  At least that’s what they’re going to do today. Tomorrow, who knows?

I just had a walk-through with Marissa Mayer and Sep Kamvar of Google’s new plans for personalization.

google1

 

 

In a nutshell, it will take the information gathered through the Google toolbar and use that, in combination with your search history, to personalize your search results.  Up till now only your past search history was used.  Enabling Web history, which is what Google is calling this, is very much a “opt in” process.  Google wants to get a small beta test bed of users so they can get a data set large enough to let Sep Kamvar, the person behind Google’s personalization algorithm, see what he can do with this additional rich set of data. Marissa indicated that this would increase the transparency of the data that Google was collecting about you. Based on the below screenshot you should be able to see exactly the sites that you visited in the past.

google2

If you want to see the nitty-gritty on what it means to sign in  or sign out of web history and what it will look like on the search results page, I would suggest checking out Danny Sullivan ever growing blog post on Searchengineland.  This is breaking news so I haven’t had much of a chance to put my thoughts together.  Like Danny, I’m “growing the post” as I find out more.  I’m going to be going back over my notes with Marissa because I think there’s some pretty significant implications for both users and advertisers in this.

In a column earlier this month I talked about Google moving towards behavioral targeting across their network and this is a huge missing piece.  I asked Marissa specifically about whether behavioral targeting of advertising based on the data collected through the toolbar would take place.  In her words, they want their organic search results to be “a little bit ahead of the curve” but overall, they want their search ads and their search results to be aligned in relevance, which means they need to be listening to the same signals.  Increasingly these will be coming from Google’s personalization algorithm.

Until I get a chance to blog more, here’s a little food for thought.  I had previously posted about Google moving towards behavioral targeting.  For me this makes all kinds of sense.  And there I speculated about what Google might be doing with all that data it collects through the PageRank toolbar.  Again, I seem to be reading my crystal ball into the future because just a few weeks later I got the call that Marissa wanted to talk to me.  And when Marissa Mayer wants to talk to you, you should listen, because it generally means something important is happening at the Googleplex.  So when you start putting pieces together, including this new move to personalization with including Web history, Google’s recent acquisition of DoubleClick, and the ability to behaviorally target people across both are contextual and DoubleClick network, not to mention the search results page, you start to get a picture of where they may be taking this.  There’s a lot more thought required here and I’m going to be trying to carve some time aside tomorrow morning to do another post on this.  I obviously want to look at this from the user perspective.  I think, although they’re limiting this to a beta and it’s very much an opt in process, this will renew the calls against personalization that have been coming from critics around the Web.  Marissa indicated that right now they’re going to be sticking to their threshold of two personal results per page, never knocking out the number one organic results, but she made it clear that that’s a “for now” call and will likely change in the future.  Google will move more aggressively towards personalization on more types of searches and they will impact more results.  A few months ago when I did the original post I said that once the gates are open on personalization the dam will burst and there will be no holding it back.  Today’s announcement ups the ante significantly.

Shari Thurow Talking Smack about Eye Tracking

You know, if I didn’t know better I’d say that Shari Thurow had issues with me and eye tracking. I ran across a column a couple of weeks ago where she was talking about the niches that SEO’s are carving out for themselves and she mentioned eye tracking specifically. In fact she devoted a whole section to eye tracking. Now, it’s pretty hard not to take it personally when Enquiro is the only search marketing company I know that does extensive eye tracking. We’re the only ones I’m aware of that have eye tracking equipment in-house. So when Shari singles out eye tracking and warns about using the results in isolation…

That brings me to my favorite group of SEO specialists: search usability professionals. As much as I read and admire their research, they, too, often don’t focus on the big picture.

…I’m not sure who else she might be talking about.

I’ve been meaning to post on this for awhile but I just didn’t get around to it. I’m on the road today and feeling a little cranky so what the heck. It’s time to respond in kind. First, here’s Shari’s take on on eye tracking and SEO.

Eye-tracking data is always fascinating to observe on a wide variety of Web pages, including SERPs (define). As a Web developer, I love eye-tracking data to let me know how well I’m drawing visitors’ attention to the appropriate calls to action for each page type.

Nonetheless, eye-tracking data can be deceiving. Most search marketers understand the SERP’s prime viewing area, which is in the shape of an “F.” Organic or natural search results are viewed far more often than search engine ads are, and (as expected) top, above-the-fold results are viewed more often than the lower, below-the-fold results. Viewing a top listing in a SERP isn’t the same as clicking that link and taking the Web site owner’s desired call to action.

Remember, usability testing isn’t the same as focus groups and eye tracking. Focus groups measure peoples’ opinions about a product or service. Eye-tracking data provide information about where people focus their visual attention. Usability testing is task-oriented. It measures whether participants complete a desired task. If the desired task isn’t completed, the tests often reveal the many roadblocks to task completion.

Eye-tracking tests used in conjunction with usability tests and Web analytics analysis can reveal a plethora of accurate information about search behavior. But eye-tracking tests used in isolation yield limited information, just as Web analytics and Web positioning data yield limited (and often erroneous) information.

Okay Shari, you didn’t mention me or Enquiro by name but again, who else would you be talking about?

Actually, Shari and I agree more than we disagree here. I agree that no single data source or research or testing approach provides all the answers, including eye tracking. However, eye tracking data adds an extraordinarily rich layer of data to common usability testing. When Shari says eye tracking is not the same as usability testing, she’s only half right. As Shari points out, eye tracking combines very well with usability testing but in many cases, can be overkill. Usability testing is task oriented. There’s no reason why eye tracking studies can’t be task oriented as well (most of ours are). The eye tracking equipment we use is very unobtrusive. It virtually like interacting with any computer in a usability lab. In usability testing you put someone in front of the computer with the task and asked them to complete the task. Typically you record the entire interaction with software such as TechSmith’s Morae. After you can replay the session and watch where the cursor goes. Eye tracking can capture all that, plus capture where the eyes went. It’s like taking a two dimensional test and suddenly making it three-dimensional. Everything you do in usability can also be done with eye tracking.

The fact is, the understanding we currently have of interaction with the search results would be impossible to know without eye tracking. I’d like to think that a lot of our current understanding of interaction with search results comes from the extensive eye tracking testing we’ve done on the search results page. The facts that Shari says are common knowledge among search marketers comes, in large part, from our work with eye tracking. And we’re not the only ones. Cornell and Microsoft have done their own eye tracking studies, as has Jakob Nielsen, and findings have been remarkably similar. I’ve actually talked to the groups responsible for these other eye tracking tests and we’ve all learned from each other.

When Enquiro produced our studies we took a deep dive into the data that we collected. I think we did an excellent job at not presenting just the top level findings but really tried to create an understanding of what the interaction with the search results page looks like. Over the course of the last two years I’ve talked to Google, Microsoft and Yahoo. I’ve shared the findings of our research and learned a little bit more about the findings of their own internal research. I think, on the whole, we know a lot more about how people interact with search than we did two years ago, thanks in large part to eye tracking technology. The big picture Shari keeps alluding to has broadened and been colored much more extensively thanks to those studies. And Enquiro has tried to share that information as much as possible. I don’t know of anyone else in the search marketing world who’s done more to help marketers understand how people interact with search. When we released our first study, Shari wrote a previous column that basically said, “Duh, who didn’t know this before?” Well, based on my discussions with hundreds, actually, thousands of people, almost everyone, save for a few usability people at each of the main engines.

There are some dangers with eye tracking. Perhaps the biggest danger is that heat maps are so compelling visually. People tend not to go any further. The Golden Triangle image has been displayed hundreds, if not thousands of times, since we first released it. It’s one aggregate snapshot of search activity. And perhaps this is what Shari’s referring to. If so, I agree with her completely. This one snapshot can be deceiving. You need to do a really deep dive into the data to understand all the variations that can take place. But it’s not the methodology of eye tracking that’s at fault here. It’s people’s unwillingness to roll up their sleeves and weed through the amount of data that comes with eye tracking, preferring instead to stop at those colorful heat maps and not go any further. Conclusions on limited data can be dangerous, no matter the methodology behind them. I actually said the same for an eye tracking study Microsoft did that had a few people drawing overly simplified conclusions. The same is true for usability testing, focus groups, quantitative analysis, you name it. I really don’t believe Enquiro is guilty of doing this. That’s why we released reports that are a couple hundred pages in length, trying to do justice to the data we collected.

Look, eye tracking is a tool, a very powerful one. And I don’t think there’s any other tool I’ve run across that can provide more insight into search experience, when it’s used with a well designed study. Personally, if you want to learn more about how people interact with engines, I don’t think there’s any better place to start than our reports. And it’s not just me saying so. I’ve heard as much from hundreds of people who have bought them, including representatives at every major search engine (they all have corporate licenses, as well as a few companies you might have heard of, IBM, HP, Xerox..to name a few). I know the results pages you see at each of the major engines look the way they do in part because of our studies.

Shari says we don’t focus on the big picture. Shari, you should know that you can’t see the big picture until you fill in the individual pieces of the puzzle. That’s what we’ve been trying to do. I only wish more people out there followed our example.

Marissa Mayer Interview on Personalization

marissa-mayer-7882_cnet100_620x433Below is the full transcript of the interview with Marissa Mayer on personalization of search results. For commentary, see the Just Behave column on Searchengineland.

Gord: It’s a little more than two weeks ago since Google made the announcement that personalization would become more of a default standard for more users on Google.  Why did you move towards making that call?

Marissa: We’ve had a very impressive suite of personalized products for awhile now: personalized homepage, search history, the personalized webpage and we haven’t had them integrated, which I think has made it somewhat confusing for users. A lot of people didn’t know if they had signed up for search history or personalized search; whether or not it was on.  What we really wanted to do was move to a signed in version of Google and a signed out version of Google.  So if you’re signed in you have access to the personalized home page, the personalized search results and search history.  You know all three of those are working for you when you’re signed in.  And if you’re signed out, meaning that you don’t see an email in the upper right hand corner that personalized search isn’t turned on.  If anything, it’s a cleaning up of the user model, to make it clearer to users what services they’re using them and when they’re using them.

Gord: But some of the criticism actually runs counter to that.  One of the criticisms is that it used to be clearer, as far as the user went, when you were signed in and when you are signed out.  There were more indicators on the Google results page whether you were getting personalized results or not.  Some of those have seemed to disappear, so personalized results have become more of a default now, rather than an option that’s available to the user.

Marissa: If you think about it as default-on when you’re signed in, I think that it’s still as clear on the search results page.  We removed the “turn off the personalized search results” link, but you still see very clearly up in the upper right-hand corner whether or not you’re signed in, your e-mail address appears, and that’s your clue Google has personalized you and that’s why that e-mail address is there.  I do think, based on our user studies and our own usage at Google, that we’ve made the model clearer.  We were actually ended up at the stage with our personalized product earlier this year where, at one point, Eric (Schmidt) asked “am I using personalized search?”  And the team’s answer as to whether or not he was currently using it was so complicated that even he couldn’t follow it.  You’d have to go to “my account”, see whether or not he was signed up for personalized search, make sure that your toggle hadn’t been turned off or on, and there was no way to just glance at the search results page and easily tell whether or not it was invoked.  So now it’s very easy, if you see your username and e-mail address up in the upper left-hand corner, you’re getting personalized results and if you don’t, you’re not.  So effectively there are two parallel universes of Google, per se.  One if you’re signed out where you see the classic homepage and the classic search results and one where you’re signed in, where you get the personalized home page and…you’ll be able to toggle back and forth, of course…and then the personalized search results page and the search history becomes coupled with all that because that’s how we personalize your search.

Gord: So, to sum up, it’s fair to say that really the search experience hasn’t changed that dramatically, it’s just cleaning up the user experience about whether you’re signed in or signed out and that’s been the primary change.

Marissa: That’s right.  Before you could be signed in and be using one of the three products or two of the three products but not all and, of course, because people like to experiment with a new product, they forget whether they signed up for personalized search.  Had they signed up for search history?  This just makes it cleaner.  If you’re signed in you’re using and/or have access to all three, if you’re signed out, you’re on the anonymous version of Google that doesn’t have personalization.

Gord: We can say that it cleans up the user experience because it makes it easier to you know when you’re signed in or signed out, but having done the eye tracking studies, we know that where the e-mail address shows is in a location that’s not prominently scanned as part of the page.  Do the changes mean that more people are going to be looking at personalized search results, just because we’ve made that more of a default opt in and we’ve moved the signals that you’re signed in a little bit out of the scanned area of the page.  Once people fixate on their task they are looking further down the page.  This should mean at a lot more people are looking at personalized search results than previously.

Marissa: Actually, I don’t think it will change the volume of personalized search all that much, not based on what we’ve seen on our logs and usage.  It makes it cleaner to understand whether or not you’re using it and I do think that over time, what it does is it pushes the envelope of search more such that you expect personalized results by default.  And we think that the search engines in the future will become better for a lot of different reasons, but one of the reasons will be that we understand the user better.  And so when we think about how we can advance towards that search engine of the future that we’re building, part of that will be personalization.  I do think that when we look five years out, 10 years out, users will have an expectation of better results.  One of the reasons that they have that expectation is that search engines will have become more personalized.  I think that in the future, working with the search engine that understands something about you will become the expectation.  But you’re right in that we believe that for users that are signed in, who find value in the personalized search results, over time as those users know they are signed in and that there search history is being kept track of, that their search results are being personalized, and they don’t need to look at every single search task to see whether or not they are signed in because that’s what their expectation is and they’re expecting personalized results.  So I do think we won’t see a drastic increase of volume right now of the use of personalized search but that it will hopefully change the user’s disposition over time to become more comfortable that personalization is a benefit for them and it’s something they come to expect.

Gord: There are a number of aspects of that question that I’d like to get into, and leave behind the question of whether you’re signed in or signed out of personalized search, but I have one question before we move on.  We’ve been talking a lot about existing users. The other change was where people were creating a new Google account and they got personalized search and search history by default.  The opt-out box is tucked into an area where most users would go right past it.  The placement of that opt-out box seems to indicate that Google would much rather have people opting into personalized search.

Marissa: I think that falls in with the philosophy that I just outlined. We believe that the search engines of the future will be personalized and that it will offer users better results.  And the way for us to get that benefit to our users is to try and have as many users signed up for personalized search as possible.  And so certainly we’re offering it to all of our users, and we’re going to be reasonably aggressive about getting them to try it out. Of course, we try to make sure they’re well-educated about how to turn it off if that’s what they prefer to do.

Gord: When this announcement came out I saw it as a pretty significant announcement for Google because it lays the foundation for the future.  I would think from Google’s perspective the challenge would be knowing what personalized search could be 5 to 10 years down the road,  what it would mean for the user experience and how do you start adding that incrementally to the user experience in the meantime?  From Google’s side, you have invested in algorithmic work to categorize content online. I would think the challenge would be just as significant to introduce the technology required to disambiguate intent and get to know more about users. You’re not going to hit that out of the park on the first pitch. That’s going to be a continuing trial and error process.  How do you maintain a fairly consistent user experience as you start to introduce personalization without negatively impacting that user experience?

Marissa: I will say that there are a lot of challenges there and a lot of this is something that’s going to be a pragmatic evolution for us.  You have to know that this is not a new development for us. We’ve been working on personalized search now for almost 4 years. It goes back to the Kaltix acquisition. So we’ve been working on it for awhile and our standards are really high.  We only want to offer personalized search if it offers a huge amount of end user benefit.  So we’re very comfortable and confident in the relevance seen from those technologies in order to offer them at all, let alone have them veered more towards the results, as we’re doing today.  We acquired a very talented team in March of 2003 from Kaltix.  It was a group of three students from Stanford doing their Ph.D, headed up by a guy named Sep Kamvar, who is the fellow who cosigned the post with me to the blog. Sep and his team did a lot of PageRank style work at Stanford.  Interestingly enough, one of the papers they produced was on how to compute PageRank faster.  They wrote this paper about how to compute page rank faster and it caused a huge media roil around the web because everyone said there are these students at Stanford who created an even faster version of Google.  Because the press obviously doesn’t understand search engines and thinks that we actually do the PageRank calculation on the fly on each query, as opposed to pre-computing it.  Their advance was actually significant not because it helps you prepare an index faster, which is what the press thought was significant.  Interestingly enough, the reason they were interested in building a faster version of PageRank was because what they wanted to do was be able to build a PageRank for each user.  So, based on seed data on which pages were important to you, and what pages you seemed to visit often, re-computing PageRank values based on that. PageRank as an algorithm is very sensitive to the seed pages.  And so, what they were doing, was that they had figured out a way to sort by host and as a result of sorting by host, be able to compute PageRank in a much more computationally efficient way to make it feasible to compute a PageRank per user, or as a vector of values that are different from the base PageRank.  The reason we were really interested in them was: one, because they really grasped and cogged all of Google’s technology really easily; and, two, because we really felt they were on the cutting edge of how personalization would be done on the web, and they were capable of looking at things like a searcher’s history and their past clicks, their past searches, the websites that matter to them, and ultimately building a vector of PageRank that can be used to enhance the search results.

We acquired them in 2003 and we’ve worked for some time since to outfit our production system to be capable of doing that computation and holding a vector for each user in parallel to the base computation.  We’ve been very responsible in the way that we’ve personalized Search Labs and we also did what we called Site Flavored Search on Labs where you can put a search box on your page and that is geared towards a page of interests that you’ve selected. So if you have a site about baseball you can say you want to base it on these three of your favorite baseball sites and have a search box that has a PageRank that’s veered in that direction for baseball queries.

So, the Kaltix team has been really successful at integrating all these Google technologies and taking this piece of theoretical research and ultimately bringing it to life on the Web.  And as it’s growing stronger and stronger and our confidence around the Kaltix technology grew, we’ve been putting it forward more and more.  We started off on Labs through a sign-up process, then we transitioned it over to Google.com and now we are in effect leaning towards a model where for people who use Google.com and have a Google account, they get personalized search basically by default.  If you look at the historical reviews of the Kaltix work it’s gotten pretty rave reviews.  The users that have noticed it and have been using it for a long time, like Danny (Sullivan), they’ll say that they think it’s one of the biggest advances to relevance that they’ve seen in the past three years.

Gord: So when you the Kaltix technology working over and above the base algorithm, obviously that’s going to be as good as the signals you’re picking up on the individual.  And right now the signals are past sites they visited, perhaps what they put on their personalized homepage and sites that they’ve bookmarked. But obviously the data that you can include to help create that on-the-fly, individual index improves as you get more signals to watch.  In our previous interview you said one thing that was really interesting to you was looking at the context of the task you are engaged in, for example, if you’re composing an e-mail in Gmail. So is contextual relevance another factor to look at.  Are those things that could potentially be rolled into this in the future?

Marissa: I think so.  I think that overall, we really feel that personalized search is something that holds a lot of promise, and we’re not exactly sure of the signals that will yield the best results.  We know that search history, your clicks and your searches together provide a really rich set of signals but it’s possible that some of the other data that Google gathers could also be useful. It’s a matter of understanding how.  There’s an interesting trade off around personalized search for the user which is, as you point out, the more signals that you have and the more data you have about the user, the better it gets.  It’s a hard sell sometimes, we’re asking them to sign up for a service where we begin to collect data in the form of search history yet they don’t see the benefits of that, at least in its fullest form, for some time.  It’s one of those things that we think about and struggle with. And that’s one reason why we’re trying to enter a model where search history and personalized search are, in fact, more expected.  And I should also note that as we look at reading some of the signals across different services we will obviously abide by the posted privacy policies.  So there are certain services where we’ve made it very clear we won’t cross correlate data. For example on Gmail, we’ve made it very clear that we won’t cross correlate that data with searches without being very, very explicit with the end user.  You don’t have to worry about things like that.

Gord: One of the points of concern seems to be how smart will that algorithm get and do we lose control?  For example, when we’re exploring new territory online and we’re trying to find answers we’ve refine our results based on our search experience.  So, at the beginning, we use very generic terms that cast a very wide net and then we narrow our search queries as we go. Somebody said to me, “Well, if we become better searchers, does that decrease the need for personalization?”  Do we lose some control in that?  Do we lose the ability to say “No, I want to see everything, and I will decide how I narrow or filter that query.  I don’t want Google filtering that query on the front end”?

Marissa: I think it really depends on how forcefully we’re putting forth personalization.  And right now we might be very forceful in getting people to sign up to it, or at least more forceful than we were. The actual implementation of personalized search is that as many as two pages of content, that are personalized to you, could be lifted onto the first page and I believe they never displace the first result, in our current substantiation, because that’s a level of relevance that we feel comfortable with.  So right now, at least eight of the results on your first page will be generic, vanilla Google results for that query and only up to two of them will be results from the personalized algorithm.  We’re introducing it in a fairly limited form for exactly the reason that you point out.  And I think if we tend to veer towards a model where there are more results that are personalized, we would have ways of making it clearer: “Do you want to explore this topic as a novice or with the personalization in place?” So the user will be able to toggle in a different filter form.  I think the other thing to remember is, even when personalization happens and lifts those two results onto the page, for most users it happens one out of every five times.  When you think about it, 20% of the queries are much better by doing that, but for 80% of the queries, people are, in fact, exploring topics that are unknown to them and we can tell from their search history that they haven’t searched for anything in this sphere before. There’s no other search like it. They’ve never clicked on any results that are related to this topic, and, as a result, we actually don’t change their query set at all because we know that they need the basic Google results.  The search history is valuable not only because it can help personalize the results but they’re also valuable because we can tell when not to.

Gord: There’s two parts to that: one is the intelligence of the algorithm to know when to push personalization and when not to push personalization, and two, as you said, right now this is only impacting one out of five searches where you may have a couple of new results being introduced into the top 10 as a result of personalization.  But that’s got to be a moving target.  As you become more confident in the technology and that it’s adding to the user experience, personalization will creep higher and higher up the fold and increasingly take over more of the search results page, right?

Marissa: Possibly.  I think that’s one of many things that could possibly happen, and I think that’s a pretty aggressive stance.  I look at our evolution and our foray into personalization, where we’re sitting here three or four years in, with some base technology that several years old already and it still has been very slight in a way that we have it interact with the user experience.  Mostly because we think that base Google is pretty good.  As it becomes more aggressive, certainly I would be pushing for an understanding of the ability of the user to know that these results are, in fact, coming from my personalization and not background and if I want to filter them out and get back to basics, that that would be possible.  One thing that we’ve struggled with is if we should actually mark the results are entering the page as a result of personalization but because team is currently and frequently doing experiments, we didn’t want to settle on a particular model or marker at this exact moment.

Gord: The challenge there is as you roll more personal results into the results page and get feedback from some users that they would want more control over what on the page is personalized and the degree of personalization and introduce more filters or more sophisticated toggles, it complicates the user experience. And as we know, that user experience needs to be very simple. Is it a delicate balance of how much control you give the user versus how much do you impact the 95% of the searches that are just a few seconds in duration and have to be really simple to do?

Marissa: There are two thoughts there.  One, even if we introduce them to filtering on the results page, it wouldn’t be any more complicated than what you had two weeks ago, so we already have that filter.  Two, we put the user first, and people have varying opinions about whether their search results page is too complicated, but the same people who designed that user experience will be the people who will be tackling this for Google, so I think you can expect results of a similar style and direction.

Gord: In the last few weeks, Google has introduced some new functionality, related searches and refine search suggestions, that are appearing at the bottom of the page for a number of searches.  To me that would seem to be a prime area that could be impacted by personalization opportunities that are coming.  As you make suggestions about other queries that you could be using, using that personalization data to refine those. Is that something you’re considering? And how long before personalization starts impacting the ads that are being presented to you on a search results page?

Marissa: Refinement is an interesting but a neophyte technology from our perspective.  We are finally now just beginning to develop some refining technologies that we believe in enough to use on the search results page.  A lot of people have been doing it for a lot longer. When you look at the overall utility, probably 1 to 5% of people will click those query refinements on any given search, where most users, probably more than two thirds of users, end up using one of our results. So in terms of utility and value that is delivered to the end user, the search results themselves and personalizing those are an order of magnitude more impactful then personalizing a query refinement.  So part of it is a question of, it’s such a new technology that we really haven’t looked at how we can make personalization make it work more effectively.  But the other thing is on a “bang for the buck” basis, personalizing those search results get us a lot more.

And as to ads, I think there are some easy ways to personalize ads that we’ve known for some time, but we’ve chosen at this point to focus on personalizing the search results because we wanted to make sure to delivered the end-user value on that, because that’s our focus, before we look at personalizing ads

Gord: So, no immediate plans for the personalization of ads?

Marissa: That’s right

Gord: Thank you so much for your time Marissa.

Brain Numbing Ideas on a Friday Afternoon

I can’t help but get the feeling that when we look at online marketing, we tend to get blinded by the technology and lose sight of what’s really important: how it affects people.

Right now there’s a flurry of attention surrounding YouTube because of copyright issues and other factors.  And YouTube isn’t alone in this.  The majority of things I did in my in box focus on technology.  What will be the next killer platform?  I see mobile search, I see online video, I see social networking. It’s hard to keep your finger on the pulse of what’s really important.  I find it useful to step back a little bit and see how these things affect real people: people not like you and I, who are caught up in the promise of technology, but people like my daughter’s principal, people like my mom, people like my next-door neighbor.  People who are wary about technology and who will only embrace it if it makes their life better in some way.  This is not to discount the importance of technology, because it truly has turned our lives inside out in the last decade.  But there’s a distillation, a time when we have to get comfortable with change.  The dotcom boom and bust was not because of the lack of technology or its inadequacy.  To technology all things are possible.  But to people, it’s all about what’s in it for me.  And that, ultimately, is the success factor that has to be considered in all this.

So, is YouTube hot?  Is online video hot?  Is social networking hot?  All these things are, but not because of the technology that lies beneath, but rather because of the social change that they empower.  Consider online video for example.  A couple of items in my in box talked about how, at this point, we won’t watch television online.  Even the person at Google who was responsible for online video admitted that at this point, even with Google’s tremendous resources, online video at the quality that we’ve come to expect is not a scalable proposition. 

We interact with video in a far different way online.  For example, YouTube is all about the viral spiral.  It’s all about that cute little two to three minutes of video: something that is either funny or outrageous or awful.  There’s no tremendous requirement for engagement for this.  YouTube is the repository for a million different “in” jokes.  It’s the basket where we collect what titillates the fancy of our collective consciousness at any given time.  It gives us an easy reference point so we can take what interests us and forward it to others if we think they are interested as well.  We’re not ready to watch a one or two hour documentary on the web, simply because we’re not used to interacting with our computer screen in that way.  Our computers are things we do things on, not things we watch passively.  A commitment of two to three minutes to watch a little video screen is fine, but we don’t look to the Web for passive entertainment.  That’s not to say we won’t, some day, as connectivity and convergence moves our channels beyond the current paradigm and as we evolve and learn to interact with them in new ways. 

And it’s there that we start to pick apart at what truly makes technology, at least as far as it’s manifested on the web, really interesting.  It stitches together the fabric of our society.  It’s a synapse that allows our collective brain to fire more effectively than it did before. Communications can zing back and forth between us at a far faster rate.  What we find interesting, what we find intriguing, what we find funny, what we find painful to watch is now available for anyone to see.  It’s cataloged and categorized for our convenience.  It occupies a finite space in the virtual world that we can point to and say, “Look at this, it impacted me and I think it will impact you to.”

I recently had the opportunity to watch Dr. Gary Flake from Microsoft talk.  He started his presentation with the claim that the information technology revolution that we’re currently in will be more significant, as far as the change factor for our society, than anything that has gone before.  More important than the Industrial Revolution, more important than the invention of the printing press, more important than television.  To me the real power of the Internet is that it’s rewiring our society in ways we could never dream of and in ways we never anticipated.  To focus on the wiring or the technology of the Web is to take the mechanic’s view of the world.  To a mechanic or a car buff, a vehicle is a wonderful thing because of the internal combustion engine, because of the horsepower and how fast it can go from zero to 60.  They focus on what it is.  But when you look at how the automobile has affected our society, it’s not about what it is, it’s about what it does.  The automobile brought the world closer.  It allowed us to travel and see new things.  It allowed us to live in one place and work in another.  The macro change that the automobile engendered had nothing to do with how an internal combustion engine worked, it came from moving people from one place to another quickly, cheaply and efficiently.  It mobilized our society in a way that never existed before.

Likewise, the Web is not powerful because of Web 2.0 technologies, or speed of connection, or the ability to host video.  It’s important because it connects us in new and different ways.  It moves power from where it was stuck before into new hands.  It breaks down existing power structures and distributes that power amongst all of us.  It puts the individual in control and allows one individual to connect with another, freely and without paying a poll to the previous power brokers.

The really interesting thing about the Internet is the underlying social current, the groundswell of change that is redefining us and how we live together.  These fundamental factors are exerting a tremendous force within our day-to-day lives.  They’re precipitating change so fast that we haven’t been able to step back and see what the full impact to us will be.  We can’t see the trickle down effect of the things that are happening to us today.  The Internet is changing the very DNA of our society, and we are unable to take a long-term view of what those current mutations will mean for us.  One only has to look at the generational difference between the 45-year-old parent, myself, and my 13-year-old daughter, the first generation that has been fully immersed in online technology.  She interacts with the world in a completely different way.  She searches for information in a different way and evaluates it differently.  She takes these things for granted because she’s never known any other way.  What happens when this entire generation emerges as the shapers of our society?  What happens when they take control from us, with their innate understanding of what the Web makes possible, and redefine everything?

Here are three things that I believe are the foundations of social change being pushed by the Internet:

Access to Information

The amount of information we currently have access to is mind-boggling.  Never has so much raw information lived so close to us.  You can now think about any given topic in the universe of our consciousness, and that information exists just a mouse click away.  And, as the saying goes, information is power.  It empowers each one of us to take a more active role in our destiny.  This information has completely changed how people buy things.  It’s completely changed the relationship between vendors and buyers.  More and more, we go direct to the source, as educated, knowledgeable buyers who know exactly what we want and what we will pay for it.  The challenge on the Internet is that not all information is created equal.  There’s good information and there’s bad information.  However, we are becoming extremely good at being able to differentiate between the two.  We’re becoming amazingly adept at being able to recognize authenticity and we can sniff out BS.  In picking through the multiple threads of information that are available to us out there, we can recognize the scent of truth and quickly discount hype, spin and sheer lies. 

Again, as we begin to recognize the shifting of power to the consumer, the full impact has not shaken out yet.  When we can buy anything online, quickly, easily and confidently, will what will that mean for the entire bricks and mortar retail world out there?  Will there be shopping malls in 20 years?  Will there be stores at all?  Will we buy directly from the manufacturers, cutting out distributors, wholesalers and retailers?  Or will distribution of products to the world of consumers lie in the hands of a few mega, long tail retailers such as Amazon?  I certainly don’t know, the future is far too murky to be able to peer down this path.  And I don’t think it’s important to be able to predict the future, but I do think it’s vitally important to consider the quantum change that is likely in the future.

Searchability

As the amount of information available to us continues to multiply exponentially, the ability to connect with the right information at the right time becomes more and more important.  I’ve always maintained that search is the fundamental foundation of everything that will transpire online.  It is the essential connector between our intent, and the content we’re looking for.  But more than just the connector, the sheer functionality of search, both as it is today and as it will be in the future, creates another catalyst for change in our society. 

We are becoming used to having the answers just a few mouse clicks away.  We are becoming a society of instant gratification.  In the past, we accepted that we couldn’t know everything.  In divvying up the world’s knowledge, some of us were experts in one area and some of us were experts in another.  Some of us were experts in nothing.  But we held no pretensions that we would become experts in areas where we had no previous experience.  There was no path to follow so there was no reason to start the journey. 

But today, you can become an instant expert in anything, depending on how you define the scope of that expertise.  Within 30 seconds I can tell you every movie that Uma Thurman ever appeared in.  I can look up a medical condition and have access to the same information, likely more information, that a doctor 20 years ago would have access to, based on his own experience, education and reference materials.  But again, what is the impact of this?  Does having access to the information about a medical condition makes me an expert in treating that condition?  I have the information but I have no context in which to apply it.  As we gain access to information, will we use that information wisely without the experience and domain expertise that used to accompany that information?

And how will instant access to information alter education in the future?  I remember hearing an observation that if we had a modern day Rip van Winkle, who had gone to sleep 20 years ago and suddenly woke up today, the one place he would feel most comfortable would be in the elementary classroom.  While the outside world is changed dramatically in the past 20 years, the classroom in which your child spends the majority of their day has changed very little.  When I help my children do their homework, there isn’t much difference between the textbooks and the worksheets I see today and the ones I saw 30 years ago.  I recently had to explain to my daughter’s principal the difference between a Web browser and a search engine.  The classroom is like a backwater eddy in the rushing torrent of technological change that typifies the rest of the world.  And it’s not just elementary school where this is an issue.  We often speak to students who are currently going through marketing programs at the university level and are always aghast at how little they’re learning about this new world of marketing and the reality of consumer empowerment.  They’re learning the rules of a game that changed at least a decade ago.

So to bring the point home once more, what will the organization of the world’s information mean for our society?  As search gets better at connecting us to the content that we are looking for, what are the ripple effects for us?  Will our children’s and grandchildren’s brains be wired in a different way than ours are?  Will they assimilate information differently? Will they research differently? Will they structure their logic in a different way?

Creation of Ideological Communities

The Web has redefined our idea of community.  It used to be the communities were defined along geographic lines.  You need a physical proximity to people in order to create a community because physical proximity was a prerequisite for communication.  Communities could exist if there was two way communication.  That’s the reason why community and communication are extensions of the same root word and concept. 

Perhaps the most powerful change introduced by the Internet has been the enabling of real, two way communication between people where physical proximity was not required.  Consider the chain of events that typifies online interaction.  You become aware of someone who shares an ideological interest, usually through stumbling upon them somewhere online.  You initiate communication.  Depending on the scope of your shared interest, you may create the core of the community by inviting others into it.  The Internet gives us the platform that allows for the creation of ideological communities.  We see this happen all the time on properties such as YouTube or MySpace.  Ideological communities are created on the fly, flourish for awhile, and then fade away as interest in the idea that engendered them also fades away.  The Internet, at any given point in time, is a snapshot of thousands, or perhaps millions, of these ad hoc ideological communities.  They form, they flourish and then they disappear.

But in our real world there was physicality to the concept of community.  The way our world is built, our political boundaries, come from physical considerations.  There are distinct geographic boundaries like mountain ranges, oceans and rivers that, in the past, prevented the flow of people across them.  Because of the restricted ability to move, people spent long enough together to share ideals and create communities.  As time moved on these communities became larger and larger.  Transportation allowed us to share common ideals over a greater expanse and nations became possible.  The more efficient the transportation, the larger the nation became.  But throughout this entire process, the concept of geography defined communities and defined nations.  Our entire existing political structure was built around this geographic foundation.

With the Internet, geography ceases to have meaning.  It’s now a virtual world, and I can feel closer to someone in China with whom I share one particularly strong mutually held belief then I might with my next-door neighbor.  More fundamentally, I can belong to several different communities at the same time.  Again, the restraint of the physical world usually restricted the number of interests we had that we could share with those immediately around us.  Our sphere of interest as an individual was somewhat dictated by the critical mass each of those interest areas had within the community in which we lived.  If we thought particularly strongly about one interest we could physically move to a community where there were more people who shared that interest.  So we tended to move to communities that felt “right” ideologically as well as physically.  But with the Internet, does that need for ideological “sameness” where we live eventually disappear?  Does our physical need for community decrease as our ideological need for community is fulfilled through the Internet?

And, if this physical definition of community begins to erode, what does that do for the concept of nationhood and all the things that come along with it? Increasingly, communication and commerce travel along lines not defined by geography.  The idea of a nation, as we currently understand it, is inextricably bound to the realities of geography.  Politics, trade, laws and defense are all concepts that are rooted in thinking developed over the past several centuries.  In the past 30 years we’ve seen the erosion of the concept of nationhood through the creation of common markets and free trade areas.  The very breakdown of the Soviet Union comes from the inability to isolate the population from the concepts which flourished in the free world.  And that was before the Internet ever became a factor.  What happens when we take this movement, already afoot, and add the tremendous catalyst that is the Internet?

It’s in these macro trends that the true power of the Internet can be seen.  It’s not about an individual technology or even the cumulative power of all the technology.  It’s about how the sum of all that affects us as individuals, how we interact with the world around us and how we connect with other individuals.  The seeds have been planted, we can’t turn back, and we can’t foresee what will be.  The world is evolving and truly becoming a global community.  We are entering a time when change will accelerate faster than our society may be able to keep up.  There will be costs, certainly, but my hope and belief is that the rewards will far outweigh the costs.

Personalized Search Brouhaha

Predictably, Google’s announcement late last week about pushing more users to personalized search results has created a lot of buzz in the blogosphere. There’s a lot of “what the hell does this mean” questioning going on out there. This will continue for the forseeable future as more engines move down the personalization road.

Normally, I’d be right in there swinging, but I have been on vacation this week, so I’m somewhat looking from afar. However, I do think that we can debate personalized search all we want in the SEM/SEO circles, but Google is going to do what Google is going to do. So, to that end, I’m reaching out to the two people who really have a say in this. Matt Cutts and I have been chatting about this for some time, but Matt wanted to defer an official interview until later this month (due, no doubt, to the timing of Google’s recent accouncement). I’m just confirming a time with Matt now. More details on this soon.

The other person I need to speak to is Marissa Mayer, on what this means for the Google search experience. Again, the wheels are in motion and I’m hoping to jump on this as soon as I get back (next week, reluctantly–I mean reluctantly returning to work, not reluctantly interviewing Marissa, which is always a delight!).

Which leads me to a lot of the buzz that’s currently happening. There’s a lot of talk about user experience. Honestly, most of the opposition I’ve heard to personalized search results are coming from SEO’s, and I have to question whether their motives are pure as they take up the UI banner here. Graywolf has been one of the most prolific critics, including posts on my blog. Here was one:

Let’s take personalized SERP’s a bit farther, let’s imagine we have something like digital books that can rewrite themselves based on user preferences. Instead of Hermione Granger being a brown haired slightly bookish student at Hogwarts, she’s a buxom blonde in a mini-skirt because I’ve demonstrated a preference for that in the past. For someone else she’s a raven haired gothic princess, for another she’s more of a debutante prom queen.

Sure the example is bit over the top but that’s not that far in concept to what they are doing. The top 10 listings in a SERP are pretty similar in concept to the main characters of a book, making them different for everyone is like having a different book for everyone.

Not sure I get the analogy here. It’s a stretch to try to compare SERP’s with a book. It doesn’t work on a number of levels. The average person spends a few seconds on a SERP, several hours with a book. And the goal is to spend as little time as possible on the results page.

Also, the nature of engagement is totally different. I’m looking for one link, the best one, on a SERP, not delving into the nuances of a character, whatever her appearance.

I do agree that Google is making it more difficult to know if you’re signed in, which is not ideal, and the current level of personalization is pretty watered down, but ultimately if personalization increases relevance to me, that’s a good thing.

Here are the challenges for Google in the personalization path they’re going down. Right now, the introduction of a few organic listings doesn’t really make a significant difference for the user. To significantly change the user experience, someone has to be bolder with personalization. And that means you have to be pretty confident that you’ve disambiguated intent. Google currently uses sites you’ve visited in the past as the indicator. As Danny said in his post, the net effect of this is your own sites, which you visit regularly, will enjoy a boost but other than that, I don’t really count this as personalization, at least not to the level I want.

What if you use the immediately preceding clickstream, as in behavioral targeting? What if you start identifying themes in the clickstream data and become bolder in grouping related search suggestions. What if you do, as Marissa Mayer suggested in her interview with me, and start mixing in contextual relevance based on your current task, as determined by Google desktop search or another Google plug in. And what if you use all this to drop the user into a much richer experience?

Let me give you an example. I’m currently on my way to Kauai, Hawaii. I’ve been doing a lot of searching for things to do, especially in the area around our hotel in Lihue. We’ve been looking for family beaches, places to go snorkeling, places to rent a bike, local events in the time frame we’re there, etc. This could have all been captured in my search history. Now, let me go to Google and launch a search for Kauai Restaurants. What would be cool is if Google presented me with restaurants close to my hotel, preferably with maps. Also, it could suggest other geographically targeted results or suggested searches. That’s personalization.

I do believe Google needs to allow users to toggle any type of customized results, with clear controls. One of the current user issues I have with Google is their transparent geo-targeting of results in Canada. When I search using a non-geographically specific query, as in “search engine marketing”, I see different results in Canada than I would in the US, favoring sites based in Canada. But 99.9999% of users in Canada would never know this, as there is nothing on the results page to indicate this. I only know it because we need to see results as they appear on both sides of the border and so use US based proxies a lot to fool Google into thinking we’re searching from the US.

For a lot of searches from Canada, it probably makes sense to push Canadian based sites higher in the result set, but for others, it doesn’t. Whatever the search, Google needs to be clearer when they filter results based on a criteria the user might not be aware of, such as personalization or geographic location.

For the search user experience, it comes down to two significant issues, and whoever can do this best will win:

Relevance Aligned to Intent: I’ve always said that search is the connector between intent and content. The more successful you can make that connection, the better. Take my intent and by whatever means necessary, personalization, demographic targeting, behavioral targeting, social targeting, give me links to the content I’m looking for. Be the best at doing that and you’ll win. And you simply can’t do that with universal search results. Personalization is inevitable.

User Control: If I have a quibble with what Google is doing, it’s in the taking control from the hands of the user. What we don’t want here is the “Google knows best” attitude that the company has been guilty of in the past. Always leave clear options for the user to navigate and tailor the results to their preferences. If you go to personalized results as a default, indicate how the user can toggle the option on and off.

We can debate whether personalization is a good or bad thing. Honestly, I think it’s a moot point. The next generation of search is impossible without personalization, in one form or another. In three interviews with usability people at Google, Yahoo and Microsoft, when I asked them about the biggest challenge to overcome, they all pointed to getting away from the current paradigm of a query box and a standard set of results. Everyone acknowledges that search is in it’s infancy. By saying that we shouldn’t go down the personalization path, it’s like saying we always want our baby to remain 9 months old. Sure, they’re easier to control at that age, but it makes it a little difficult for them to realize their potential as a human being.

The Personalized Results are Coming, The Personalized Results are Coming!

Okay, sometimes the temptation to say I told you so is overwhelming. Danny has a nice long post in Searchengineland about Google’s changes to Personalized Search, making it more of a default and less of an option for millions of users. Danny details it more than I intend to, so please check it out.

As Danny says, he’s been talking about personalization for years, but up to now, it never materialized. After interviews with head user experience people at all three engines, I felt the time was right for personalized search to roll out (check The Future of SEO in a Personalized Search Interface and The SEO Debate Continues). And it appears my sense of timing was bang on. Much as I’d like to claim to be prescient, it’s really just common sense. You could see all the engines inching towards it. Now, Google has just upped the ante a little.

There are two major implications to this: what it means for search marketers, especially organic optimizers, and what it means to users. I’ll deal with each in turn.

What it Means for Search Marketers

The “Is SEO Dead? Rocket Science? A Scam?” Debate has been winding it’s weary way through several blogs in the past few weeks. My take was that SEO is, and will continue to be, vitally important as long as organic search results continue to be important to the user. Based on what I’m seeing, that continues to be very much the case. But, organic optimization now has a completely new rule set, which will irritate the hell out of many organic optimizers. The disgruntlement is already beginning to show. Michael Gray, better known as Graywolf, was the first to post a comment on Danny’s story:

Just because I ordered my coke with extra ice last time doesn’t mean I want it that way this time. I hate personalized SERP’s, I despise it even more that they don’t tell me they are personalized, and I loathe not being able to turn it off. I also have extreme antipathy for not being able to keep my search history on and not be part of personalized search.

Let me have it the way I want, not the way you think I do. I don’t want SERP’s that work like Microsoft programs that try to anticipate what I want to do, because more often than not it’s wrong. Bring back truth, purity, and clarity to the SERP’s.

Graywolf is complaining as a user, but I can’t help thinking that the more significant pain he’s feeling is as an organic optimizer who’s world suddenly just became a lot more complicated. “Truth, purity and clarity to the SERP’s”? In whose eyes? Come on. Personalization is being implemented because it enhances the user experience. It doesn’t take a “Rocket Scientist” (sorry, couldn’t resist) to see that one set of search results is not the best way to serve millions of users.

As Danny said, there’s now an explosion of new fronts for the organic optimizer to consider. Right now, Google is only injecting a few personalized results into the search page, but expect that threshold to gradually creep up as Google gains confidence in the targeting of the results to the person. The days of the universal results page are numbered. Which means that the days of the reverse engineering approach to SEO are equally numbered. I’m sure people will try to figure out ways to spam personalized search, but as I’ve said before, reverse engineering requires a fixed constant to test against. Up to now, the results page and the other sites that appeared on it represented that fixed constant. That’s gone now.

So where does that leave SEO? Well, it’s certainly not dead, but it has dramatically changed. You can’t optimize against a results set, but you can optimize against a user. Let’s use an analogy that’s often been used before to describe SEO. Think of it as Public Relations on the Web. If you launch a PR campaign, you don’t target a particular position on the front page of the NY Times, you target a type of audience. You plan your release distribution and messaging accordingly. And you give reporters what you think will catch their attention. Most of all, you have to wrap your campaign around something that’s genuinely interesting. Then, you hope for the best.

Now, SEO becomes the same thing. You don’t target the first page of results on Google for a particular term. You target an end user. You wrap your site messaging in terms that resonate with that user. You write in their language, you give them a reason to seek you out, and you sure as hell don’t disappoint them when they click through to your site. You do all this, and you remove all the technical barriers between your content and the indexes you need to be in. Then, you hope for the best.

The problem with SEO has always been that it’s been treated like some magical voodoo that can be applied after the fact, like some “secret sauce”. And yes, that was what the infamous Dave Pasternack has been trying to say. He just went several steps too far. The fact is, with universal search results, you could actually do this. Thousands of affiliates have made millions of dollars doing it. Link spamming, cloaking, doorway sites..the fact is, up to now, this bag of tricks has worked. It’s gotten harder, but it’s worked. Site owners looked to SEO to help them hi jack traffic that wasn’t rightfully theirs. They hadn’t done the heavy lifting to create a site that justified a place in the top rankings, and they tried to take an easy short cut.

But now, organic optimization means that you have to do the heavy lifting. It has to be integrated into the entire online presence. What Marshall Simmonds has done with About.com and the NY Times is a perfect example of the new definition of SEO. Get to the front lines, to the people who are churning out the content, and teach them about what search engines are looking for. Make sure SEO best practices are baked right into the overall process flow. Work with the IT team to create a platform that entices the spider to crawl deeper. Work with the marketing team to crawl inside the head of your target audience and figure out the who, the when and the why. Don’t worry so much about the where, because you can’t really control that any more. It’s a tough paradigm to break. We’ve been struggling with our clients for the past year or so. They’re still fixated on “being number one” for a particular term. We’ve been trying to ease them into the new reality, but it’s not easy.

I guarantee this will create an identity crisis for the SEO industry. As recently as a few months ago I was moderating a panel that was talking about analytics, and in the Q&A someone asked the panel, who had a few very well known SEM’s on it, about what they used for ranking reporting. The names of various options were thrown out and people started scribbling them down. I saw this and thought I had to comment.

“You know, the whole concept of ranking is quickly becoming irrelevant”

Nobody lifted their head, they were still busy writing down tool names. Maybe they hadn’t heard.

“As search engines move to personalized results, there will be no such thing as ranking. It will all be relative to the user.”

That should get their attention. Nope, nothing.

One of the search marketers said, “Yes, but knowing how they rank is still important to people.”

Huh? Am I speaking a different language here? I shook my head and gave up.

So, does this mean SEO is dead? Absolutely not. It becomes more vital than ever. Here are a few things that remain to be true. Preliminary results from the new SEMPO survey say SEO continues to be the number one tactic in search marketing. Yes, people want to bring it in house, but they recognize it’s importance.

Why do they think it’s important? Because it kicks ass in ROI. Here are the results from another recent study by Ad:Tech and MarketingSherpa, asking advertisers about the return they get from various marketing channels.

080569

The biggest jump from year to year? SEO. Now, let’s look at where marketers plan to spend more money in the next year.

080570

SEO, from flatlined last year to looking to spend 25% more this year. So SEO definitely isn’t dead. But it is moving to a new home. Here’s some early results from the SEMPO State of the Market Study (by the way, final results should be available next week. Look for them):

SEMPO2a

It’s true that most companies would far rather bring SEO in house, if they could. And when we consider the new definition of SEO, it probably makes sense for SEO to be integrated into the internal work flow. But the problem is that there’s not a lot of SEO expertise out there. If SEO was so easy, why don’t more companies do it, or do it well? Contrary to Pasternack’s argument, it’s not a “set and forget” type of tactic. It requires a champion, buy in and diligence.

I think the future is bright for SEO as a skill set, but we’re talking a modified set of skills. I talked about this in a recent SearchInsider column and a follow up online debate with Andrew Goodman. My view of the future for the really good SEO’s out there fall into three categories:

Get a (Really Good) Job

As companies bring this in house, there will be a firestorm of demand for skilled SEO Directors, but ideally as employees, not consultants.

Broaden Your View

Become an expert in how consumers navigate online and help your customers with the big picture, including the new reality of SEO.

Adapt and Survive

Find a new online niche where your search honed skills give you an advantage.

User’s View

Okay, this is already a much longer post than I intended, so I should probably talk about personalized search from the user’s perspective now.

Personalized search is a big win for the user. Don’t judge by the first few tentative steps Google is taking. Personalization is a much bigger deal than that. Google is easing us in so the experience isn’t too jarring. By the end of 2007, all 3 of the major engine’s results pages will look significantly different than they do today. Personalization will be like a breached dam. Right now we’re seeing the first few trickles, but there will be a wave of much deeper personalization options over the next several months. Search will become your personalized assistant, tailored to your tastes. As you search more, your results will draw more and more away from the universal default and closer and closer to your unique intent. Immediately after your query, you’ll be dropped into a much richer search experience. Disambiguation will become much more accurate, and you’ll find that you will pretty much always find just what you’re looking for right at the top of your page, without having to dig deeper. Here’s how I see it playing out at each of the big three:

Google

Google has a religious devotion to relevance, and as they gain confidence with personalized search and their ability to disambiguate, this will manifest itself with a laser focus on relevance above the fold. They will continue to maintain a good balance of organic results, but these results will not just be the current web search results. They could be local, image, news or a mix of each. And ads. Yes, you won’t escape ads, but Google will be the most judicious in what they show. Expect more stringent quality scoring, down to the landing page level and a high degree of relevancy in the ads that do show. Google will be the most concerned of the three in disambiguating intent.

Yahoo

Yahoo will put their own spin on personalization by wrapping in Social Search. They will continue to leverage their community, as they currently do in Yahoo! Answers so when you’re logged into Yahoo, you’ll be plugged into their community and that will impact the search results you see. Relevancy will be determined more by what the community finds interesting than what you find interesting, although it will be a mix between the two. Yahoo will target two types of searches, serendipitous search, where you’re looking to discover new sites, and what I call “frustrated” search, where your own efforts to unearth the data online have come up empty and you want the help of the community. When it comes to monetization, Yahoo will be the most aggressive, pushing more ads above the fold into Golden Triangle real estate. These ads will trail Google’s in terms of relevance

Microsoft

Microsoft will use their targeting capabilities and probably tie in some behavioral targeting to personalize their search results. Also expect personalization in the Microsoft product to be integrated at a deeper, more ubiquitous level, into apps and OS. This probably won’t happen in 07, but it will be a long term goal. When it comes to ad presentation, Microsoft will fall somewhere between Google and Yahoo in both the number and relevance of the ads being presented. The heaviest investment will be in building out the platform to manage and model the ad program, rather than in policing the quality of the ads themselves.

It promises to be a very interesting year in the Search Marketing biz!