This is the full transcript of an interview with Professor Jim Jansen at Penn State University. Excerpts from the Interview are running in two parts (Part One ran a few weeks ago) on Search Engine Land. I wrote a column that provided a little background on Dr. Jansen on Search Engine Land.
Gord:
Jim, we’ll start by laying out some of the research you’ve been doing over the past year and a half and then we’ll dig deeper into each of those as we find interesting things. Just give me the quick 30- or 60-second summary of what you’ve been working on in the last little while.
Jim:
I have several research projects going on. One that I really find interesting is analyzing a five calendar year search engine marketing campaign from a major online retailer and brick-and-mortar retailer. It’s about 7 million interactions over that time, multi-million dollar accounts and sales and stuff. A fascinating temporal analysis of a search engine marketing effort.
I’ve been looking at that at several different levels – the buying funnel being one, aspect of branding being another, and then the aspect of some type of personalization, specifically along gender issues. And so that’s been very, very exciting and interesting and (has offered) some great insights.
Gord:
I’m familiar with the buying funnel one because you were kind enough to share that with me and ask for my feedback, so let’s start there. I know you went in to prove out some assumptions, for example, is there a correlation between the nature of the query and where people would be in the buying funnel? Is there identifiable search behaviours that map to where they might be in their purchase process? What did you find?
Jim:
I looked at it at several different levels. One goal was to verify whether the buying funnel was really a workable model for online e-commerce searching or was it just a paradigm for advertisers to, you know, get their handle around this chaos. And if it’s an effective model, what can it tell us in terms of how advertisers should respond?
In terms of the first question, we had mixed results. At the individual query level you can classify individual queries into different levels of this buying funnel model. There are unique characteristics that correspond very nicely to each of those levels. So in that respect, I think the model is valid.
Where it may not be valid is specifying this process that online consumers go through. We found that, no, it didn’t happen quite like we assume. There was a lot of drop-out and they would do a very broad query and that might be all.
So we looked at the academic literature – you know, what theoretically could deal with that or explain that? – and the idea of sufficing seemed to fit. If it is a low cost, they won’t spend a lot of time, they will just purchase it and buy it.
In terms of classifying queries in terms of what advertisers’ payoff is, I think the most interesting finding was that the purchase queries – the last stage of the buying funnel – were the most expensive and had no higher payoff than the awareness or the very broad, relatively cheaper queries. From talking to practitioners, that is a phenomena that they have noted also … which is why a lot of people bid still on very broad terms, to snatch these potential customers at an early stage.
Gord:
Based on what you’ve seen, there are a couple of really interesting things. You and I have talked a little bit about this, but we similarly have found that you can’t assume a search funnel is happening because people use search at different stages and they’ll come in and then they’ll drop out of the process, and they may come in later or they may not, they may pursue other channels. But the other thing we found is sometimes there’s a remarkable consistency in the query used all the way through the process and that quite often can be navigational behaviour. It can be people who say, “Okay, the last time I did this, I searched on Google for so-and-so and I remember the site I found was the third or fourth level down,” and they just use the same route to navigate the online space over and over again. If you’re looking at it from a pure query level, it’s a bit of a head-scratcher because you’re going, “Well, why did they use the same query over and over?” but again, it’s one of those nuances of online behaviour. Did that seem to be one of the possible factors of some of the anomalies in the data?
Jim:
Well, that trend or something similar to it has been appearing in a lot of different domains and researchers are attributing it to “When I do a query, I expect a certain result.” So, you know, a query that may be very informational, what we’re finding is that searchers expect a Wikipedia entry. So in other words, a very navigational intent behind that very informational query. And I think the phenomena you’re describing is very similar. We have a transactional-type query and users are expecting a certain web page, a navigational aspect, and that “Okay, I have an anchor point here that I’m going to go to.” And then off search engine, maybe they do more searching and actually do some type of buying funnel process. But at the search engine, yes, we’re seeing a lot of that navigational aspect. I just looked at a query log from a major search engine and an unbelievable amount of queries were just navigational in nature.
Gord:
We’ve certainly seen that. A lot of our recent research has been in the B2B space, so it’s a little bit different but certainly it follows those same lines. When we looked at queries that people would use, a large percentage of them were either very specific or navigational in nature.
You know, the idea of satisficing, of taking a heuristic shortcut with their level of research is also interesting. It seems like if the risk is fairly low, the online paths are shorter. Is that what you were finding?
Jim:
Yes, and the principle of least effort is how it’s also presented. We see it in web searching itself generally in how people interact with search engines and how they interact with sites on the web. They may not get an optimal solution, but if it’s something that’s reasonable and if it’s good enough, they’ll go for it. That seems to be occurring in the e-commerce area also: “I want to buy something relatively cheap. This particular vendor may not have the best price, but it’s close to what I’m thinking it should be. Just go and get it done, get it over with, buy it.”
Gord:
I would suspect that that would also be true in product categories where you have mentally a good idea of what an acceptable price range would be, right?
Jim:
Yes.
Gord:
So if it’s a question of making a trade-off for $2 but saving yourself a half hour of time, as long as you’re aware of what those price ranges would be, you’re more apt to make that shortcut call, correct?
Jim:
Yes. It does assume some knowledge and risk mitigation –if it’s a small purchase and that varies a little bit for each of us, but you’re willing to cut your costs of searching and trying to find the best deal just to get it done.
Gord:
I suspect part of this would also be your level of personal engagement with the product category you’re shopping in. So I’ll spend way too much time researching a purchase of a new gadget or something that I’m interested in just because I have that level of engagement. But if it’s a purchase that’s on my to-do list, if it’s just one task I have to get done and then move on to the next thing, I suspect that that’s where that satisficing behaviour would be more common.
Jim:
Yes. Now you bring up a really good point. If it becomes entertainment – like a gadget that you enjoy researching – it’s no longer work, it’s no longer something you get done. The process of doing it makes it enjoyable so you don’t mind spending a lot of time. In those kind of cases, the goal really is not the purchase, the goal is the looking.
Gord:
We found that alters the behaviour on the search page as well. So if it’s a task-type purchase where I just have to go and get there, you see that satisficing play out on the search page too. Typically when we look at engagement with the search page, you see people scan the top four, three or four listings. If it’s that satisficing type of intent where they’re saying, “I just want to buy this thing,” you’ll see people scan those first three or four and pick what they feel is the path of least effort. They go down and say, “Okay. It’s a book. Amazon’s there. I know Amazon’s price. I’m just going to click through and order this,” but if it’s entertainment, then suddenly they start treating the search page more like a catalogue where they’re paying more attention to the brands and they’re using that as a navigational hub to branch off to three or four different sites. Again, it can really impact the nature of engagement with the web… or with the search page.
Jim:
Absolutely, and I really like your analogy of a catalogue. You know, there are some people that love just looking at a catalog – flipping through it, looking at the dresses and shirts or gadgets or sporting gear or whatever. And so that’s a much different engagement than flipping through the classified ads trying to find some practical thing you need. The whole level of engagement is at totally opposite ends of the spectrum, really.
Gord:
As an extreme example of that, we did some eye-tracking with Chinese search engines and we found that with Baidu in particular, people were using it to look for MP3 files to download. So when we first saw the heat maps – and of course it was all in Chinese, so I could understand what the content on the page was without having it translated – I saw these heat maps going way deeper and much longer than we ever saw in typical North American behaviour. We saw a level of engagement unlike anything we had ever seen before. And it was exactly it. It was a free task - They were looking for MP3 files to download and they were treating the search page like a catalogue of MP3 files. They were reading everything on the page. I think that’s just one extreme example of this catalogue browsing behaviour that we were talking about.
Let’s go to one of the other findings on the buying funnel: that quite often the more general, broader categories from an ROI perspective can perform just as well as what traditional wisdom tells us is your higher return terms. Those closer to the end of the funnel – the ones that are more specific, longer, more transactionally oriented. What’s behind that?
Jim:
Like a lot of these questions there’s no simple answer because there are plenty of exceptions to the rule you just described. There are some very broad terms that are very cheap, others that are very expensive. On the purchase side, there are some key phrases that are very cheap because they’re so focussed and others are expensive. But in this particular analysis – and again, this was 7 million transactions over 33 months, from mid-2005 to mid-2008 – the awareness terms were cheaper than the purchase terms and they generated just as much revenue.
I think a lot of it is that perhaps the items this particular retailer was selling fell into that sufficing behaviour: gifts, fairly low-cost items - there was just no need to progress all the way to that particular purchase phase.
To me it was really very unexpected. I really expected those purchase terms to actually be cheaper because they were more narrowly focussed and to generate more revenue, but it didn’t turn out that way.
Gord:
That brings up an interesting point we’ve seen with client behavior, especially given the current economic condition. We found is a lot of clients are tending to optimize down the funnel – they are tending to look at their keyword lists they’re bidding on and move further and further down to more and more specific phrases, because the theory is – and generally they do have analytics to back this up – that there’s greater ROI on that because these are usually people that are searching for a specific model or something which is a pretty good indicator that they’re close to purchase. But I think one of the by-products of that is as people optimize their campaigns, those long tail phrases are getting more and more expensive because there’s more and more competition around them, and as people move some of their keyword baskets away from those awareness terms, maybe the prices on that, it all being based on an auction model, are starting to drop. Do you think that could be one of the factors happening here?
Jim:
That very well could be. The whole online auction is designed around (the concept that) as competition increases, cost-per-clicks will increase also. It also may be that those particular customers don’t mind clicking on a few links to do some comparison-shopping and may end up going somewhere else. They may have a higher aspect of intent to purchase, but the competition among where they’re going to buy is more intense.
You know, compare that to this sufficing shopper: you just have to get that person’s attention first with a reasonably priced product and you will make the sale. That is the one issue with analytics in terms of transaction log analysis – we can analyze behaviours and we can make some conjectures about what happened, but you need lab studies and surveys to pan all data, to get the why part.
Gord:
That’s a great comment and obviously something that people have heard from me over and over again, because we do tend to focus more on the quantitative approach. I think this goes back to what we were talking about originally –online information gathering is a natural extension of where we are in our actual lives so it’s not like a distinct, contained activity. It’s not like we set aside an hour each day to go through all our online research. More and more, we always have an outlet to the internet close by and as we’re talking or as we’re thinking about something, it’s a natural reaction just to go and use a search engine to find out more information. And I think because it’s such a natural extension of what’s happening in our day-to-day lives, that the idea of this one linear progression through an online research session isn’t the way people act. I think it’s just an extension of whatever’s happening in our real world. So we may do a search, we may find something, it may be an awareness search, and then we may pursue other paths to the eventual purchase. It’s not like we keep going back and forth between a search engine with this nicely refined search funnel. It’s not that neat and simple, just like our lives aren’t that neat and simple.
Jim:
Yes, all models get rid of all the noise that reflect reality. So the neater they are, the less accurate they are, and the buying funnel is obviously very neat and so I think it’s reasonable that it represents a very small number of searches that actually progress exactly like that. We’re very nonlinear in things we do and so I assume our purchase behaviors are too.
Gord:
I want to move on to the question of branding a little bit, because you mentioned that that was one of the areas you were looking at. And at Enquiro, we’ve done our own lab-based studies on branding, so I’d be fascinated to hear what came out as far as the impact of branded search.
Jim:
This year, I’ve really got into this whole idea of branding in terms of information seeking. That’s really my background, web searching and how people find and assemble information. One of my first studies was to look at the comparison of what a search engine brand would do to how searchers interpreted the results. So I ran a little experiment where I switched the labels from Google, Yahoo, and MSN, and the results were the same. Certainly the search engine brand has a major lift to it.
In this particular study using the search engine marketing data, we did multiple comparisons of brand or product name and the keyword in the title, in the snippet, in the URL to see if there was a correlation with higher sales. And without a doubt the correlation between a query with a brand term and an advertisement with a brand term is extremely, extremely positive. That particular tightness seems to resonate with online consumers.
Gord:
So just to repeat, so if somebody’s using a branded query and they see that brand appear in the advertising, there’s obviously a statistical correlation between the success of that, right?
Jim:
Yes. In that particular case, one, that the click will happen, and two, that the click will result in a sale was yes, very positive. It really relates to the whole idea of dynamic keyword insertion in advertisements…
Gord:
So to follow that thread a little bit further, obviously if people have a brand in mind and they see that brand appear, then that’s an immediate reinforcement of relevancy. But what happens if the query is generic in nature, it’s for a product category, but a brand appears that people recognize as being a recognized and trusted brand within that product category? Did you do any analysis on that side of things?
Jim:
Not specifically. No, I did not. That’s a real good question though, but no, I did not do that type of correlation.
Gord:
The last thing I want to ask you about today, Jim, is this idea of personalization by gender. I believe from our initial discussions that you’re just in the process of looking at the data from this portion. Is that right?
Jim:
Well, we finished the analysis. Now we’re just writing it up.
Gord:
So is there anything that you can share with us at this point?
Jim:
Again, the results to me were counterintuitive from what I expected. Usually, the idea of personalization is that the more personalized you get, the higher the payoff, the efficiency and effectiveness is. We took queries from this particular search engine marketing campaign and classified them based on gender probability using Microsoft’s demographic tool, which will classify a query by it’s probability of being male or female. We looked at it this way: now whether the searcher was male or female but did the particular query fit a gender stereotype – did it have a kind of a male, for example, feel to it or stereotype implications.
Gord:
So more women would search for “Oprah,” and more men would search for “NASCAR”?
Jim:
Exactly.
Gord:
What did you find?
Jim:
In terms of sales, far and away the most profitable were the set of queries that were totally gender-neutral. We took the queries and divided them into seven categories: “very strongly male,” “generally male,” “slightly male,” “gender neutral,” “slightly female,” “strongly female,” “very female.” By two orders of magnitude, the most profitable were the ones that were totally gender-neutral.
Gord:
Fascinating.
Jim:
Yes, as a researcher who does personalization research, my guess would be “Ah, the more targeted they are, the more profitable.” But no, the means were two orders of magnitude different.
Gord:
So give us an example of a gender-neutral query.
Jim:
We defined gender-neutral to be were queries that the Microsoft tool classified somewhere between- exactly gender-neutral is zero - up to like 59% either side. So we had a fairly big spread here. And there was a trend that was somewhat expected - that the queries that were more female-targeted generated higher sales than the corresponding male counterparts.
So here’s some examples of queries based off the Microsoft tool: “Electronic chess,” 100%. You know, the Microsoft tool classified that 100% male. For a gender-neutral query, I’ll just randomly pick up a couple here: “Atomic desk clock.” “Water purifier.”
Gord:
I know you’re just writing this up now, but any ideas as to why that might be?
Jim:
One thing that is coming out in the personalization research is that at a certain level, we have totally unique differences. You can personalize to a general category and to a certain level, but beyond that, it’s either not doing much good or may actually get in the way. And that may be something that is happening here - that these particular, very targeted gender keyword phrases are just not attracting the audience that the more gender-neutral queries and keywords are.
Again, it’s a “why” thing. We spend a lot of time in web search trying to personalize to the individual level and really haven’t got very far. But now people are trying to do things like personalize to the task rather than the individual person, and there’s some interesting things happening there. Spell checks and query reformulations and things like that are very task-oriented rather than individual searcher oriented.
Gord:
I remember Marissa Mayer from Google saying that when Google was looking at personalization, they found by far the best signal to look at was what’s the string, what immediately preceded the search or a series of search iterations. They found that a much better signal to follow than trying to do any person-level personalization, which is what you’re saying. If you can look at the context of the tasks they’re engaged in and get some kind of idea of what they’re doing or trying to accomplish in that task, that’s probably a better application of personalization than trying to get to know me as an individual and to try to anticipate what I might say or query for any given objective.
Jim:
Yes, It’s just so hard to do. You know, Gord is different than Jim, and Gord today is different than Gord was five years ago. Personalizing at the individual level is just very difficult and may not even be a fruitful area to pursue.
Gord:
I remember when Google first came out with talking about personalization there was this flurry around personalization in search. That was probably two, two and a half years ago and it really seems to have died down. You just don’t hear about it as much. And at the time I remember saying that personalization is a great thing to think of in ideal terms – you know, it certainly would make the search experience better if you could get it right or even half-ways right, but the challenge is doing just that. It’s a tough problem to tackle.
Jim:
Yes, and as you mentioned earlier, we’re nonlinear creatures, we’re changing all the time. I can’t even keep up with all my changes and I can’t imagine some technology trying to do it. It just seems an unbelievably challenging, hard task to do.
Gord:
I think the other thing is – and certainly in my writings and readings this becomes clearer and clearer – that we don’t even know what we’re doing most of the time. We think we have one intent but there’s much that’s hidden below the rational surface that’s actually driving us. And for an algorithm to try to read something that we can’t even read ourselves is a task of large magnitude to take on.
Jim:
That’s a really good way of looking at it. I’ve commented on that before in terms of recommending a movie or book to me. I don’t even know what books and movies I like until I see them. Sometimes I pick up a book and say, “Oh, I’m going to really love this,” only to get a chapter into it and realize “Okay, this is horrible.” And I think you see that in the NetFlix challenge - So many organizations have laboured for a decade now, and finally it looks like perhaps this year someone may win by combing 30 different approaches simultaneously to the very simple problem of “Recommend a movie. It’s just amazing the computational variations that are going on.
Gord:
Amazon has obviously been trying to do this. They were one of the first to look at collaborative filtering and personalization engines, and they probably do it about as well as anyone. But even then, when I log on to Amazon, it’s not that they’re that far off base in their recommendations to me, but given what I buy on Amazon, it’s like they’re dealing with this weird fragmented personality because one time I’m ordering a psychology textbook because it has to do with the research I’m doing for something and the next time I’m turning around and ordering a DVD box set of The Office or even worse, the British version of The Office which really throws it for a loop.
Jim: [laughs]
Gord:
Then I’m ordering a book for my daughter like Twilight. Amazon is going, “I don’t know who this Gord Hotchkiss is, but he’s one strange individual.”
Jim:
From my interaction with Amazon, the recommendations I have found most effective are “You bought this book. Other people that bought this book bought these books” which I view as a very task-oriented personalization. And the other is a very broad, contextual one, “Here’s what other people in your area are buying,” which fascinates me. It’s almost like a Twitter, Facebook, social networking thing: “Oh, wow. I like that book,” you know? These task-oriented context personalizations, at least in my interactions, have been the most effective.
Gord:
You obviously bring up that intersection between social and search, which is getting a lot of buzz with the explosion of Twitter and the fact that there’s now real-time search that allows you to identify patterns within the complexity of the real-time searches. We’ve known in the past in other areas that generally those patterns as they emerge can be pretty accurate, so that opens up a whole new area for improving the relevancy of search.
Jim, one last question while we’re talking about personalization. This is something I wrote about in an article a little while ago and I’d love to get your take on it as the last word of this interview. We were talking about personalization and getting it right more often, and the fact is the way we search now, engines can be somewhat lax in getting it right. There’s a lot of real estate there, we scroll up and down. The average search page has something between 18 and 20 links on it when you include the sponsored ones. It’s more like a buffet: “We’re hoping one of these things might prove interesting to you or whet your appetite.” But when we move to a mobile device, the real estate becomes a lot more restrictive and it becomes incumbent on the engines to get it right more often. We can’t afford a buffet anymore, we just need that waiter who knows what it is we like and can recommend it. What happens with personalization as the searches we’re launching are coming from a mobile device?
Jim:
That’s a great question. I think it’s one of those areas that have got a lot of talk – everybody is saying (again) “This is the year mobile searching’s going take off.” It’s been going on for four or five years now, and really, I mean at least here in the US, it hasn’t really happened yet. But what I think is going to make it hit the mainstream is this combination of localized search.
When you have a mobile device, the technology has so much more information about you: it’s got your location to within a couple feet, the context that you’re in can really start entering the picture and information gets pushed to you –I’m thinking tagged buildings and restaurants and cultural events and on and on. And so with my mobile device, where I can talk into it, I don’t even have to type anything. I want “what’s going on in the area?” and it automatically knows my location and the time and perhaps something about me and the things that I’ve searched on before. “Oh, you like coffee shops where there’s some music playing. Guess what? Boom. There’s five right near, in your area that have live entertainment right then.” So I think in that respect it’ll be a little more narrowed search, but the technology will have so much more information about us that in a way it makes the job easier. The problem’s going to be the interface and the presentation of the results.
Gord:
We’re talking about, you know, subvocalization commands and heads-up display. You start looking at that and say, “Wow, that would be pretty cool,” but…
Jim:
Yes. Imagine being able to walk through a town … I live in Charlottesville, Virginia. Tons of history here from 400 years ago when Europeans first settled here, Thomas Jefferson, James Madison, etc., etc. Being able just to walk down Main Street and have tagged buildings interface with my mobile device… I’m a big history buff and so getting that particular information, one, pushed to me or at least available to push when I ask for it is a wonderful, wonderful area of personalization. This idea of localized search and mobile devices and mobile search may be the thing that brings it all together and makes mobile search happen.
Gord:
It’s fascinating to contemplate. And I know I promised that was going to be my last question, but I’m going to cheat and squeak one more in, and it’s really a continuation. You remember the old days of Longhorn with Microsoft, when they were working what eventually became Vista. They were talking about building search more integrally into everything they did and they had this whole idea of Implicit Query – which really excited me because if anyone knows what you’re working on at any given time, it should be Microsoft, at least on the desktop. They control your e-mail, they control your word processing, they control your calendar. If you could combine all this… all those as signals – the document you’re writing and the next appointment you’ve got coming up and the trip you’re taking tomorrow – imagine how that could intersect with search and really turn into a powerful, powerful thing. I remember saying…this was years ago… “That could kill Google. If Microsoft can pull this off, that could be the Google killer.” Of course we know now that that never happened. But if we take all that integration and all that knowledge about what you’re doing and what you’re doing next and where you are and move that to a mobile device, that’s really interesting. In looking at where Google is going, introducing more and more things that compete directly against Microsoft… is that where Google’s heading, to become our big brother that sits in our pocket and continually tells us what we might be interested in?
Jim:
You know, the “Big Brother” idea label has certain negative connotations, so I don’t want to say that they are Big Brother-ish in that regard. But certainly I think with their movement into free voice and free directory assistance, they will soon have a voice data archive that will allow them to do some amazing things with voice search, which would be an awesome feature for mobile devices. Being able to talk into a mobile device, have it recognize you nearly 100% of the time and execute the search.
Google of course is the one that knows what they’re doing, but certainly I think it would be naive not to be exploring that particular area. And I think the contrast from what you said about Microsoft and the desktop, the desktop is just so busy. You’re getting so many different signals in terms of business, personal things, my kids use my computer sometimes. And so the context is so large on the desktop, but the mobile device, it’s narrower. You know, you have some telephone calls, you can do some GPS things, so the context is narrower but very, very rich in that very narrow domain. I think it’s a really hot area of search.