Author Archives: ghotchkiss

Being in the Room Where It Happens

I spent the past weekend attending a conference that I had helped to plan. As is now often the case, this was a hybrid conference; you could choose to attend in person or online via Zoom. Although it involved a long plane ride, I choose to attend in person. It could be because – as a planner – I wanted to see how the event played out. Also, it’s been a long time since I attended a conference away from my home. Or – maybe – it was just FOMO.

Whatever the reason, I’m glad I was there, in the room.

This was a very small conference planned on a shoestring budget. We didn’t have money for extensive IT support or AV equipment. We were dependent solely on a laptop and whatever sound equipment our host was able to supply. We knew going into the conference that this would make for a less-than-ideal experience for those attending virtually. But – even accounting for that – I found there was a huge gap in the quality of that experience between those that were there and those that were attending online. And, over the duration of the 3-day conference, I observed why that might be so.

This conference was a 50/50 mix of those that already knew each other and those that were meeting each other for the first time. Even those who were familiar with each other tended to connect more often via a virtual meeting platform than in a physical meeting space. I know that despite the convenience and efficiency of being able to meet online, something is lost in the process. After the past two days, carefully observing what was happening in the room we were all in, I have a better understanding of what that loss might be – it was the vague and inexact art of creating a real bond with another person.

In that room, the bonding didn’t happen at the speaking podium and very seldom happened during the sessions we so carefully planned. It seeped in on the sidelines, over warmed-over coffee from conference centre urns, overripe bananas and the detritus of the picked over pastry tray. The bonding came from all of us sharing and digesting a common experience. You could feel a palpable energy in the room. You could pick up the emotion, read the body language and tune in to the full bandwidth of communication that goes far beyond what could be transmitted between an onboard microphone and a webcam.

But it wasn’t just the sharing of the experience that created the bonds. It was the digesting of those experiences after the fact. We humans are herding animals, and that extends to how we come to consensus about things we go through together. We do so through communication with others – not just with words and gesture, but also through the full bandwidth of our evolved mechanisms for coming to a collective understanding. It wasn’t just that a camera and microphone couldn’t transmit that effectively, it was that it happened where there was no camera or mic.

As researchers have discovered, there is a lived reality and a remembered reality and often, they don’t look very much alike. The difference between the effectiveness of an in-person experience and one accessed through an online platform shouldn’t come as a surprise to us. This is due to how our evolved sense-making mechanisms operate. We make sense of reality both internally, through a comparison with our existing cognitive models and externally, through interacting with others around us who have shared that same reality. This communal give-and-take colors what we take with us, in the form of both memories and an updated model of what we know and believe. When it comes to how humans are built, collective sense making is a feature, not a bug.

I came away from that conference with much more than the content that was shared at the speaker dais. I also came away with a handful of new relationships, built on sharing an experience and, through that, laying down the first foundations of trust and familiarity. I would not hesitate to reach out to any of these new friends if I had a question about something or a project I felt they could collaborate on.

I think that’s true largely because I was in the room where it happened.

Saying Goodbye to our Icons

It’s been a tough couple of months for those of us who grew up in the 60s and 70s. Last month, we had to say goodbye to Robert Redford, and then, just over a week ago, we bid farewell to Diane Keaton.

It’s always sobering to lose those cultural touchstones of our youth. It brings us to forcibly reckon with our own mortality. Our brains play that maudlin math, “I remember them being young when I was young, so they can’t be that much older than me.”  We tend to conflate the age difference between us and those we watch when we’re young, so when they’re gone, we naturally wonder how much time we have left.

This makes it hard to lose any of the icons of our youth, but these two – for me – felt different: sadder, more personal. It was like I had lost people I knew.

I know there are many who swooned for Bobby Redford. But I know first-hand that an entire generation of male (and possibly female) adolescents had a crush on Diane Keaton’s Annie Hall. Her breakout role was one of those characters that carved a permanent place in our psyche. “Annie Hall-esque” became a descriptor we could all immediately understand – quirky, cute, with insecurities that were rendered as charming. We all wanted to be her port in a storm.

Diane Keaton and Robert Redford seemed like people we could know, given the chance. If circumstances ever caused our paths to cross, we felt we could have a real conversation with them. We could talk about meaningful things and perhaps connect on a personal level. There was depth below the celebrity and the heart of a real person beating there. We may have just known them through a screen – but they used those platforms to build a connection that felt real and human.

I wondered what it was about these two – in particular – that made the connection real. It was something that went beyond their talent, although their talent was undeniable. One only has to watch an example of Keaton’s master acting with Al Pacino in The Godfather: Part Two. After a visit with her estranged children, she is being pushed out the door before ex-husband Michael Corleone comes home, but he walks in while she’s still standing in the doorway. No words are said between the two for almost a minute. Everything is conveyed just by their expressions. It’s a scene that still rips my heart out.

It was also not about celebrity. In fact, Redford and Keaton both eschewed the life of a celebrity. Robert Redford found his life away from Hollywood in the ranch lands of Montana and Diane Keaton – well – in typical Keaton fashion, she just kind of ignored being a celebrity. In an interview with Vanity Fair in 1985, she said, ““I think I like to deny it (being famous). It suits me to deny it. It’s more comfortable for me to deny it, but I suppose that’s another one of my problems. Look, I don’t think it’s such a big deal. I don’t think I’m that big a thing.”

So, if it wasn’t their talent or their celebrity status, what was it about Keaton and Redford that forged such a strong bond with many of us? I think it may have been three things.

First, it was about consistency. They were judicious about what they shared with us but what they did choose to share was rock solid and reliable. Whatever was at the core of who they were – it shone through their performances. There was a foundation to each Redford and Keaton performance that was both essential and relatable. You couldn’t imagine anyone else playing these roles. 

The authenticity of their humanness was another factor. Robert Redford’s acting style was restrained and typically underplayed, but his charismatic good looks sometimes got in the way of the depth and vulnerability he tried to bring to his performances. He famously tried out for the title role in 1967’s The Graduate (which went to Dustin Hoffman) but was turned down by director Mike Nichols because he couldn’t see Redford as a believable “loser.” “Let’s put it this way,” Nichols reportedly said, “Have you ever struck out with a girl?” “What do you mean?” Redford replied.

Keaton was a little different. She embodied vulnerability in every role she played. She wasn’t perfect, and that was the point. We loved her imperfections. The characters Diane Keaton played were neither aspirational nor cautionary, they were revelatory. We connected with them, because we could see ourselves in them.

Finally, we knew there was depth to both Diane Keaton and Robert Redford. They believed passionately in things and weren’t afraid to speak out on behalf of those beliefs. I would have loved to have a conversation with either of them about serious things, because I feel I would have walked away with a perspective worth discovering.

It’s sadly ironic that for two icons who shared so much screen time with us, they never shared it with each other. They were tentatively scheduled to appear in a 2012 Holiday comedy but it never made it to the screen.

I will miss having both Robert Redford and Diane Keaton in my world. They made it better.

Lilith Fair: A Quarter Century and A Different World Ago

Lilith Fair: Building a Mystery, a new documentary released on Hulu (CBC Gem in Canada), is much more than a chronicle of a music festival. It’s a very timely statement on the both the strength and fragility of community.

Lilith Fair was the festival launched in 1997 by Canadian singer/songwriter Sarah McLachlan. It was conceived as a feminine finger in the eye of a determinedly misogynistic music industry. At the end of the 90’s, despite a boom in talented female singer songwriters (Tracy Chapman, Jewel, Paula Cole, Sheryl Crow, Natalie Merchant, Shawn Colvin, Lisa Loeb, Suzanne Vega and others too numerous to mention), radio stations wouldn’t run two songs by women back-to-back. They also wouldn’t book two women on the same concert ticket. The feeling, based on nothing other than male intuition, was that it would be too much “femininity” for the audience to handle.

McLachlan, in her charmingly polite Canadian way, said “Fudge you!” and launched her own festival. The first one, in 1997, played almost 40 concerts over 51 days across North America. The line-up was exclusively female – 70 singers in all playing on three stages. Almost every concert sold out. Apparently, there was an audience for female talent. Lilith Fair would be repeated in 1998 and 1999, with both tours being smashing successes.

The World needed Lilith Fair in the late 90s. It wasn’t only the music industry that was misogynistic and homophobic. It was our society. The women who played Lilith Fair found a community of support unlike anything they had ever experienced in their careers. Performers who had been feeling isolated for years suddenly found support and – more than anything – understanding.

It was women who made the rules and ran the Lilith Fair show. It was okay to perform when you were 8 months pregnant. It was okay to hold your baby onstage as you performed the group encore. It was okay to bring the whole family on tour and let the kids play backstage while you did your set. These were things that were – up until then – totally foreign in the music industry. It was the very definition of community – diverse people having something in common and joining together to deal from a position of strength.

But it didn’t happen overnight. It took a while – and a lot of bumping into each other backstage – for the community to gel. It also needed a catalyst, which turned out to be Amy Ray and Emily Saliers – officially known as the Indigo Girls. It was their out-going friendliness that initially broke the ice “because we were so gay and so puppy dog-like.”

This sense of community extended beyond the stage to the thousands who attended: men and women, old and young, straight and gay. It didn’t matter – Lilith Fair was a place where you would be accepted and understood. As documentary producer Dan Levy (of Schitt’s Creek fame) – who was 12 years old when he attended and was yet to come out – said, “Being there was one of the earliest memories I’ve had of safety.”

The unity and inclusiveness of Lilith Fair stood in stark contrast to another festival of the same era – Woodstock 99. There, toxic masculinity from acts like Limp Bizkit singer Fred Durst and Kid Rock, swung the vibe of the event heavily towards anarchy and chaos rather than community.

But while Lilith Fair showed the importance of community, it also showed how fragile it could be. The festival became the butt of jokes on late night television (including one particularly cringe-worthy one by Jay Leno about Paula Cole’s body hair) and those that sought to diminish its accomplishments and importance. Finally, at the end of the 1999 tour, McLachlan had had enough. The last concert was played in the rain at Edmonton, Alberta on August 31st.

McLachlan did try to revive Lilith Fair in 2010, but it was a complete failure. Whatever lightening in a bottle she had captured the first time was gone. The world had passed it by. The documentary didn’t dwell on this other than offering a few reasons why this might be. Perhaps Lilith Fair wasn’t needed anymore. Maybe it had done its job. After all, women had mounted some of the top tours of that time, including Taylor Swift, Madonna, Pink and Lady Gaga.

Or maybe it had nothing to do with the industry. Maybe it had everything to do with us, the audience.

The world of 1999 was very different place than the world of 2010. Community was in the midst of being redefined from those sharing a common physical location to those sharing a common ideology in online forums. And that type of community didn’t require a coming together. If anything, those types of communities kept us apart, staring at a screen – alone in our little siloes.

According to the American Time Use Survey, the time spent in-person socializing has been on a steady decline since 2000.  This is especially true for those under the age of 25, the prime market for musical festivals. When we did venture forth to see a concert, we are looking for spectacle, not community. This world was moving too fast for the coalescing of the slow, sweet magic that made Lilith Fair so special.

At the end of the documentary, Sarah McLachlan made it clear that she’ll never attempt to bring Lilith Fair back to life. It was a phenomenon of that time. And that is sad – sad indeed.

When Did the Future Become So Scary?

The TWA hotel at JFK airport in New York gives one an acute case of temporal dissonance. It’s a step backwards in time to the “Golden Age of Travel” – the 1960s. But even though you’re transported back 60 years, it seems like you’re looking into the future. The original space – the TWA Flight Center – was designed in 1962 by Eero Saarinen. This was a time when America was in love with the idea of the future. Science and technology were going to be our saving grace. The future was going to be a utopian place filled with flying jet cars, benign robots and gleaming, sexy white curves everywhere.  The TWA Flight Center was dedicated to that future.

It was part of our love affair with science and technology during the 60s. Corporate America was falling over itself to bring the space-age fueled future to life as soon as possible. Disney first envisioned the community of tomorrow that would become Epcot. Global Expos had pavilions dedicated to what the future would bring. There were four World Fairs over 12 years, from 1958 to 1970, each celebrating a bright, shiny white future. There wouldn’t be another for 22 years.

This fascination with the future was mirrored in our entertainment. Star Trek (pilot in 1964, series start in 1966) invited all of us to boldly go where no man had gone before, namely a future set roughly three centuries from then.   For those of us of a younger age, the Jetsons (original series from 1963 to 64) indoctrinated an entire generation into this religion of future worship. Yes, tomorrow would be wonderful – just you wait and see!

That was then – this is now. And now is a helluva lot different.

Almost no one – especially in the entertainment industry – is envisioning the future as anything else than an apocalyptic hell hole. We’ve done an about face and are grasping desperately for the past. The future went from being utopian to dystopian, seemingly in the blink of an eye. What happened?

It’s hard to nail down exactly when we went from eagerly awaiting the future to dreading it, but it appears to be sometime during the last two decades of the 20th Century. By the time the clock ticked over to the next millennium, our love affair was over. As Chuck Palahniuk, author of the 1999 novel Invisible Monsters, quipped, “When did the future go from being a promise to a threat?”

Our dread about the future might just be a fear of change. As the future we imagined in the 1960’s started playing out in real time, perhaps we realized our vision was a little too simplistic. The future came with unintended consequences, including massive societal shifts. It’s like we collectively told ourselves, “Once burned, twice shy.” Maybe it was the uncertainty of the future that scared the bejeezus out of us.

But it could also be how we got our information about the impact of science and technology on our lives. I don’t think it’s a coincidence that our fear of the future coincided with the decline of journalism. Sensationalism and endless punditry replaced real reporting just about the time we started this about face. When negative things happened, they were amplified. Fear was the natural result. We felt out of control and we keep telling ourselves that things never used to be this way.  

The sum total of all this was the spread of a recognized psychological affliction called Anticipatory Anxiety – the certainty that the future is going to bring bad things down upon us. This went from being a localized phenomenon (“my job interview tomorrow is not going to go well”) to a widespread angst (“the world is going to hell in a handbasket”). Call it Existential Anticipatory Anxiety.

Futurists are – by nature – optimists. They believe things well be better tomorrow than they are today. In the Sixties, we all leaned into the future. The opposite of this is something called Rosy Retrospection, and it often comes bundled with Anticipatory Anxiety. It is a known cognitive bias that comes with a selective memory of the past, tossing out the bad and keeping only the good parts of yesterday. It makes us yearn to return to the past, when everything was better.

That’s where we are today. It explains the worldwide swing to the right. MAGA is really a 4-letter encapsulation of Rosy Retrospection – Make America Great Again! Whether you believe that or not, it’s a message that is very much in sync with our current feelings about the future and the past.

As writer and right-leaning political commentator William F. Buckley said, “A conservative is someone who stands athwart history, yelling Stop!”

It’s Tough to Consume Conscientiously

It’s getting harder to be both a good person and a wise consumer.

My parents never had this problem when I was a kid. My dad was a Ford man. Although he hasn’t driven for 10 years, he still is. If you grew up in the country, your choices were simple – you needed a pickup truck. And in the 1960s and 70s, there were only three choices: Ford, GMC or Dodge. For dad, the choice was Ford – always.

Back then, brand relationships were pretty simple. We benefited from the bliss of ignorance. Did the Ford Motor Company do horrible things during that time? Absolutely. As just one example, they made a cost-benefit calculation and decided to keep the Pinto on the road even though they knew it tended to blow up when hit from the rear. There is a corporate memo saying – in black and white – that it would be cheaper to settle the legal claims of those that died than to fix the problem. The company was charged for negligent homicide. It doesn’t get less ethical than that.

But that didn’t matter to Dad. He either didn’t know or didn’t care. The Pinto Problem, along with the rest of the shady stuff done by the Ford Motor Company, including bribes, kickbacks and improper use of corporate funds by Henry Ford II, was not part of Dad’s consumer decision process. He still bought Ford. And he still considered himself a good person. The two things had little to do with each other.

Things are harder now for consumers. We definitely have more choice, and those choices are harder, because we know more.  Even buying eggs becomes an ethical struggle. Do we save a few bucks, or do we make some chicken’s life a little less horrible?

Let me give you the latest example from my life. Next year, we are planning to take our grandchildren to a Disney theme park. If our family has a beloved brand, it would be Disney. The company has been part of my kids’ lives in one form or another since they were born and we all want it to be part of their kid’s lives as well.

Without getting into the whole debate, I personally have some moral conflicts with some of Disney’s recent corporate decisions. I’m not alone. A Facebook group for those planning a visit to this particular park has recently seen posts from those agonizing over the same issue. Does taking the family to the park make us complicit in Disney’s actions that we may not agree with? Do we care enough to pull the plug on a long-planned park visit?

This gets to the crux of the issue facing consumers now – how do we balance our beliefs about what is wrong and right with our desire to consume? Which do we care more about? The answer, as it turns out, seems to almost always be to click the buy button as we hold our noses.

One way to make that easier is to tell ourselves that one less visit to a Disney mark will make virtually no impact on the corporate bottom line. Depriving ourselves of a long-planned family experience will make no difference. And – individually – this is true. But it’s exactly this type of consumer apathy which, when aggregated, allows corporations to get away with being bad moral characters.

Even if we want to be more ethically deliberate in our consumer decisions, it’s hard to know where to draw the line. Where are we getting our information about corporate behavior from? Can it be trusted? Is this a case of one regrettable action, or is there a pattern of unethical conduct? These decisions are always complex, and coming to any decision that involves complexity is always tricky.

To go back to a simpler time, my grandmother had a saying that she applied liberally to any given situation, “What does all this have to do with the price of tea in China?” Maybe she knew what was coming.

There Are No Short Cuts to Being Human

The Velvet Sundown fooled a lot of people, including millions of fans on Spotify and the writers and editors at Rolling Stone. It was a band that suddenly showed up on Spotify several months ago, with full albums of vintage Americana styled rock. Millions started streaming the band’s songs – except there was no band. The songs, the album art, the band’s photos – it was all generated by AI.

When you know this and relisten to the songs, you swear you would have never been fooled. Those who are now in the know say the music is formulaic, derivative and uninspired. Yet we were fooled, or, at least, millions of us were – taken in by an AI hoax, or what is now euphemistically labelled on Spotify as “a synthetic music project guided by human creative direction and composed, voiced and visualized with the support of artificial intelligence.”

Formulaic. Derivative. Synthetic. We mean these as criticisms. But they are accurate descriptions of exactly how AI works. It is synthesis by formulas (or algorithms) that parse billions or trillions of data points, identify patterns and derive the finished product from it. That is AI’s greatest strength…and its biggest downfall.

The human brain, on the other hand, works quite differently. Our biggest constraint is the limit of our working memory. When we analyze disparate data points, the available slots in our temporary memory bank can be as low as in the single digits. To cognitively function beyond this limit, we have to do two things: “chunk” them together into mental building blocks and code them with emotional tags. That is the human brain’s greatest strength… and again, it’s biggest downfall. What the human brain is best at is what AI is unable to do. And vice versa.

A few posts back when talking about one less-than-impressive experience with an AI tool, I ended by musing what role humans might play as AI evolves and becomes more capable. One possible answer is something labelled “HITL” or “Humans in the Loop.” It plugs the “humanness” that sits in our brains into the equation, allowing AI to do what it’s best at and humans to provide the spark of intuition or the “gut checks” that currently cannot come from an algorithm.

As an example, let me return to the subject of that previous post, building a website. There is a lot that AI could do to build out a website. What it can’t do very well is anticipate how a human might interact with the website. These “use cases” should come from a human, perhaps one like me.

Let me tell you why I believe I’m qualified for the job. For many years, I studied online user behavior quite obsessively and published several white papers that are still cited in the academic world. I was a researcher for hire, with contracts with all the major online players. I say this not to pump my own ego (okay, maybe a little bit – I am human after all) but to set up the process of how I acquired this particular brand of expertise.

It was accumulated over time, as I learned how to analyze online interactions, code eye-tracking sessions, talked to users about goals and intentions. All the while, I was continually plugging new data into my few available working memory slots and “chunking” them into the building blocks of my expertise, to the point where I could quickly look at a website or search results page and provide a pretty accurate “gut call” prediction of how a user would interact with it. This is – without exception – how humans become experts at anything. Malcolm Gladwell called it the “10,000-hour rule.” For humans to add any value “in the loop” they must put in the time. There are no short cuts.

Or – at least – there never used to be. There is now, and that brings up a problem.

Humans now do something called “cognitive off-loading.” If something looks like it’s going to be a drudge to do, we now get Chat-GPT to do it. This is the slogging mental work that our brains are not particularly well suited to. That’s probably why we hate doing it – the brain is trying to shirk the work by tagging it with a negative emotion (brains are sneaky that way). Why not get AI, who can instantly sort through billions of data points and synthesize it into a one-page summary, to do our dirty work for us?

But by off-loading, we short circuit the very process required to build that uniquely human expertise. Writer, researcher and educational change advocate Eva Keiffenheim outlines the potential danger for humans who “off-load” to a digital brain; we may lose the sole advantage we can offer in an artificially intelligent world, “If you can’t recall it without a device, you haven’t truly learned it. You’ve rented the information. We get stuck at ‘knowing about’ a topic, never reaching the automaticity of ‘knowing how.’”

For generations, we’ve treasured the concept of “know how.” Perhaps, in all that time, we forgot how much hard mental work was required to gain it. That could be why we are quick to trade it away now that we can.

The Credibility Crisis

We in the western world are getting used to playing fast and loose with the truth. There is so much that is false around us – in our politics, in our media, in our day-to-day conversations – that it’s just too exhausting to hold everything to a burden of truth. Even the skeptical amongst us no longer have the cognitive bandwidth to keep searching for credible proof.

This is by design. Somewhere in the past four decades, politicians and society’s power brokers have discovered that by pandering to beliefs rather than trading in facts, you can bend to the truth to your will. Those that seek power and influence have struck paydirt in falsehoods.

In a cover story last summer in the Atlantic, journalist Anne Applebaum explains the method in the madness: “This tactic—the so-called fire hose of falsehoods—ultimately produces not outrage but nihilism. Given so many explanations, how can you know what actually happened? What if you just can’t know? If you don’t know what happened, you’re not likely to join a great movement for democracy, or to listen when anyone speaks about positive political change. Instead, you are not going to participate in any politics at all.”

As Applebaum points out, we have become a society of nihilists. We are too tired to look for evidence of meaning. There is simply too much garbage to shovel through to find it. We are pummeled by wave after wave of misinformation, struggling to keep our heads above the rising waters by clinging to the life preserver of our own beliefs. In the process, we run the risk of those beliefs becoming further and further disconnected from reality, whatever that might be. The cogs of our sensemaking machinery have become clogged with crap.

This reverses a consistent societal trend towards the truth that has been happening for the past several centuries. Since the Enlightenment of the 18th century, we have held reason and science as the compass points of our True Norh. These twin ideals were buttressed by our institutions, including our media outlets. Their goal was to spread knowledge. It is no coincidence that journalism flourished during the Enlightenment. Freedom of the press was constitutionally enshrined to ensure they had the both the right and the obligation to speak the truth.

That was then. This is now. In the U.S. institutions, including media, universities and even museums, are being overtly threatened if they don’t participate in the wilful obfuscation of objectivity that is coming from the White House. NPR and PBS, two of the most reliable news sources according to the Ad Fontes media bias chart, have been defunded by the federal government. Social media feeds are awash with AI slop. In a sea of misinformation, the truth becomes impossible to find. And – for our own sanity – we have had to learn to stop caring about that.

But here’s the thing about the truth. It gives us an unarguable common ground. It is consistent and independent from individual belief and perspective. As longtime senator Daniel Patrick Moynihan famously said, “Everyone is entitled to his own opinion, but not to his own facts.” 

When you trade in falsehoods, the ground is consistently shifting below your feet. The story is constantly changing to match the current situation and the desired outcome. There are no bearings to navigate by. Everyone had their own compass, and they’re all pointing in different directions.

The path the world is currently going down is troubling in a number of ways, but perhaps the most troubling is that it simply isn’t sustainable. Sooner or later in this sea of deliberate chaos, credibility is going to be required to convince enough people to do something they may not want to do. And if you have consistently traded away your credibility by battling the truth, good luck getting anyone to believe you.

Exclusive Interview: Larry Cornett, Yahoo Director Of User Experience Design

Note: This is the 2nd of the Just Behave series from Search Engine Land. This one ran back in February, 2007. At the time of this interview, Yahoo Search was retooling and trying to gain some marketshare lost to Google. They had just launched Panama, their ad management platform. In our eyetracking study, we found Yahoo – more than any of the other engines – loaded the top of the SERP with sponsored ads. We felt this would not bode well for Yahoo’s user experience and we talked about that in this interview. Yahoo pushed back, saying in many cases, a sponsored ad was what the user was looking for. I remember thinking at the time that this didn’t pass our own usability “smell test.” Three years later, Yahoo Organic search would be shuttered and replaced with results from Microsoft’s Bing.

This week I caught up with Larry Cornett, the relatively new Director of User Experience Design at Yahoo and Kathryn Kelly, Director of PR for Yahoo! Search. Again, to set the stage for the interview, here are some high level findings from our eye tracking study that I’ll be discussing in more detail with Larry and Kathryn.

Emphasis on Top Sponsored Results

In the study we found Yahoo emphasized the top sponsored results more than either Microsoft or Google. They showed top sponsored results for more searches and devoted more real estate to them. This had the effect of giving Yahoo! the highest percentage of click throughs on top sponsored, but on first visits only. On subsequent visits the click through rate on these top sponsored ads dropped to a rate lower than what was found on Google or Microsoft.

Better Targeted Vertical Results

Yahoo’s vertical results, or Shortcuts, seem to be better targeted to the queries used by the participants in the survey. Especially for commercial searches, Yahoo did a good job disambiguating intent from the query and providing a researcher with relevant vertical results in the product search category.

How Searching from a Portal Impacted the Search Experience

When we look at how the search experience translated from a portal page where the query is launched to the results page, we found that Yahoo had a greater spread of entry points on the actual search results page. This brings up the question of how launching a search from a portal page rather than a simple search page can impact the user experience and their interaction with the results they see.

I had the chance to ask Larry and Catherine about the Yahoo search experience and how their own internal usability testing has led to the design and the experience we see today. Further I asked them about their plans for the future and what their strategy is for differentiating themselves from the competition, namely Microsoft and Google. One difference you’ll notice from Marissa’s interview last week is the continual reference to Yahoo’s advertisers as key stakeholders in the experience. At Yahoo, whenever user experience is mentioned, it’s always balanced with the need for monetization.

Here’s the interview:


Gord: First let’s maybe just talk in broad terms about Yahoo’s approach to the user experience and how it affects your search interface. How are decisions made? What kind of research is done? Why does Yahoo’s search page looks like it does?

Larry: I can give you a little bit. I have been here about 7 months, so I’m still fairly new to Yahoo. But I can tell you a little bit about how we move forward with our decision making process and the approach we are taking with Yahoo.. We try to strike a balance between user experience and the needs of business as well as our advertising population. We want to provide the best user experience for the users who are trying to find information and give them most efficient experience they can have and then also provide a good ecosystem for our advertisers. And we do a tremendous amount of research here: we do a lot of usability testing, we do surveys. We also do eye tracking studies. All those help inform us when thing are working really well for users and one we need to work on improving things.

Kathryn: And we do bucket testing on a lot of different features, on different properties, to get feed back from our users before ever implementing any new features.

Gord: Since we conducted the study we’ve noticed some changes on the search interface. Do changes tend to be more evolutionary or do you lump them together into a major revision and roll them out together?

Larry: I’d say we do both. We have essentially two parallel tracks. One is continuous improvement, so we’re always looking to improve the experience. They would be considered the evolutionary changes based on a lot of data we are looking at. Then we also have larger things that we’re definitely interested on a more strategic direction, that we look at in a longer term window for larger changes.

Gord: One of the things that we did spend a fair amount of time on in this study is this whole idea of perceived relevancy. If we set a side the whole question of how relevant are the actual results based on the content of those results and what shows, and look more how quickly scent is picked up by the user and how the results that they see are perceived to be relevant. Does that notion coincide with your findings from the internal research and how is that idea of the appearance of relevancy rather the actual relevancy play into the results you present?

Larry: Yes, that absolutely is similar to the types of research findings that we’ve had, specifically with some of the eye tracking studies. We also continue to make efforts on actual relevance so our Yahoo search team is constantly doing improvements to everything to have real relevance improve. But you right, that perceived relevance is actually the most important thing because, at the end of the day, that’s what the users are looking at and that’s what they walk away with. In terms of: Was my search relevant? Did I find what I was looking for?

I do like the concept that you have with the information scent, the semantic mapping. I think it definitely ties into the mental model that a user has when they approach search and they are doing a query. They looking for things that come back to match what they have on their mind, what they are looking for in the results, so the more they actually see those search terms and things they are having in their mind, in terms of what they’re expecting to see, the more relevant the search is going to be for them.

Gord: It comes down to the efficiency of the user experience too, how quickly they find what they are looking for, how quickly they think they find what they are looking for and how successful that click through is. Did the promise match up with what was actually delivered on the other end?

Larry: Yes absolutely.

Gord: One of the biggest differences we saw between Yahoo and the other engines was the treatment of the top sponsored ads. I think it’s fair to say that in both the percentages of the searches that ads were presented for and the number presented, you were more aggressive than MSN (now Live Search) and Google. Obviously I understand the monetization reasoning behind that, but maybe you can speak a little bit as far as the user experience.

Larry: Sure, I mean in many cases those results are, and even in your report you showed this, those results are exactly what the users are looking for. Very often what they see in that sponsored section actually is a good fit for the type of query they are doing, especially if you look at a commercial query. So it’s always finding that balance between monetization and showing organic results. We’re just trying to get the best results for the user based on what they are looking for.

Gord: I certainly agree with you with the fact that in a lot of cases the sponsored results were what they were looking for, but we couldn’t help but notice that there was always a little bit of suspicion or skepticism on the part of the user, both in how they scan the results, and even when they do click through to a result there seems to be a hesitancy to stop there. We found a tendency to want to check out at least the top organic listing as well. One thing with Yahoo is that, with the more aggressive presentation of top sponsored, their choices on the organic tend to get pushed closer and closer to the fold. Have you done any testing on that?

Larry: Yes, those are definitely things that we are exploring as we’re trying to improve the user experience. And we’ve done our own eye tracking as well. A lot of it does come down to a big difference between what’s above the fold and what’s below the fold. So we’re always being very careful when we’re exploring that, thinking about the dominant monitor resolution, settings that we’re looking at as people start to have more advanced systems and larger monitors and really trying to understand what they seeing when they given that first load of the search page.

Gord: We did notice that of all the three engines, Yahoo has the highest percentage of click throughs on top sponsored and first visits. A little more then 30 % of the clicks happened there. But we noticed that it dropped substantially on repeat visits, much more then it did on the other engines. Combine that with the fact that we saw more pogo sticking on Yahoo then we did on the other engines; someone would click through a top sponsored ad then click back to the search results. So, my question is; does the more aggressive presentation of top sponsored ads even out when you factor in the repeat visits and does those lower repeat click through rates negatively impact the monetization opportunities on those repeat visits?

Larry: I can’t get into too many details about that, because it starts to get into some of the business logic and business rules that we have. Especially looking at CTR (click through rates) and repeat visits, so yes, that’s probably a little more information that I actually have access to, myself.

Gord: OK, we’ll put that one off limits for now. Let’s shift gears a little bit. One of the other things that we noticed, that actually works very well on Yahoo, was Yahoo Shortcuts on the vertical results. It seems like you are doing a great job at disambiguating intent based on query and really giving searchers varied options in that vertical real state. Maybe you can talk about that.

Larry: You are absolutely right that those are very effective. And you’ll probably notice over time that we’re continuing to refine our Shortcuts, try to find even more appropriate shortcuts for different types of queries. A lot of that is based on the best end result, what the user is trying to find, and the more we can give them that information the better

Kathryn: And it’s faster too, right?

Larry: Exactly. So a lot of time people just want a quick answer; they don’t want to have to dig through a lot of web pages. They just want a very simple quick answer, so if we can provide that, then that is a great experience for them.
Kathryn: And we found that certain queries like movies, entertainment, weather, sports, travel, blend very nicely with those types of Shortcuts.

Gord: In talking to Google about this, they have fairly strictly monitored click throughs thresholds in both their top sponsored ads and on their vertical results; and if results aren’t getting clicked they don’t tend to show. It automatically gets turned off. What’s Yahoo’s approach to that? Are you monitoring CTRs and determining whether or not vertical results and top sponsored ads will appear for certain types of queries.

Larry: We definitely monitor that as well. We’re interested in tracking usage and so looking at the CTR, because we don’t want to be showing things that are not actually getting usage, so we do continually monitor the CTR and the Shortcuts.

Kathryn: Are you just referring to the Shortcuts or to all of the ads?

Gord: Both; top sponsored and vertical results or Shortcuts.

Kathryn: It’s the same for both.

Larry: Obviously we track CTR in both of those areas and look at that trend over time.

Gord: When we look at the visibility difference or the delta between those top sponsored ads and those side sponsored ads, when you factor in conversion rates, click through rates and everything else, is the difference as significant as it appears to be from an eye tracking study? How do you work with your advertisers to maximize their placement and to help them understand how people are interacting with that search real estate?

Larry: That is actually a separate team that works with those folks.

Gord: Is there overlap between the two departments? You would be on top of how the user is interfacing or interacting with the search results page. Do you share that information with that team and keep them up to date with how that real estate is been navigated?

Larry: Absolutely, it’s a very collaborate relationship, We are in communication constantly, they are giving us performance, we are giving them performance. So it’s always a very collaborative kind of relationship with that team. We definitely can give them recommendations and vice versa. We each have our own worlds that we own.

Gord: Maybe you can speak a little bit from the user’s perspective how Yahoo is when you position it against Microsoft Live Search and Google. What’s unique, why should a user be using Yahoo rather than the other two?

Larry: I’d say one of the key differentiators that you’ve seen released last year, and Terry Semel actually talks about this, is that we’re starting to introduce social search. If you look at Yahoo Answers, it’s one of the key examples of that. It’s a very exciting site that’s performing very well. There’s a lot of great press around it, and we starting to integrate that within the search experience itself, so you can do certain types of queries within search and at the bottom of the page you’ll see relevant best answers that are brought from Yahoo’s Answers. In many cases people look at that and say that it actually adds value. That’s one of the key differentiators here; there is definitely a social aspect to Yahoo Search.

Gord: As Yahoo Answers and that whole social aspect gains traction , is that something that either be moved up as far as visibility on the real state page, moved up into that Golden Triangle real estate or would it be rolled in almost transparently in what the results being shown are?

Larry: Anything is possible, but it’s something that we’re evaluating, so we’re constantly looking at data that comes from user studies and the live site performance, and so we’ll be making a decision about that as the year goes on.

Gord: One other thing that I think somewhat distinguishes Yahoo, especially from Google, is where the searches are launched from. When you look at user’s experience, obviously you are taking into consideration where those Yahoo searches are being launched from; a tool bar versus a portal versus the search page. How does that factor into the user experience?

Larry: You’re right. We definitely look at that type of data and really try to understand how those users might be different and their expectations might be different. So we’re constantly looking at that whole ecosystem because Yahoo is a very large network, with a lot of wonderful properties, so you have to understand how we all play together and what the relationship is between the properties back and forth.

Kathryn: Another thing is knowing where to put a search box on what property and what is going to work with the right mix of users for that property, because not every property is conducive to having a search box prominently displayed, and that’s something that we look at very closely.

Gord: Which brings up another question. One of the things that we speculated on the study is; does the intent of the user get colored based on the context that shows around that search box? If it gets launched from a very clean minimalist search page, there is little influence on intent, but if it gets launched from a portal, where there is a lot of content surrounding it, the intent can then be altered in between the click on the search box and ending up on the search page . Do you have any insight on that? Have you done your own studies on that impact?

Larry: We are definitely doing research within that area to understand the affect of the context and I don’t really have anything I can share at this point but I would say that there’s probably a lot of very interesting information to be derived from looking at that.

Gord: Ok. I’ll go out on the limb once more and say: Obviously if you can take the contextual messaging or what is surrounding that search box, and if it obviously correlates to the search, then I suppose that will help you in potentially targeting the advertising messaging that can go with that, right?

Larry: I think that’s fair to say.

Gord: Ok. I’ll leave it there. One question I have to ask comes down to the user interface. It seems as changes are made the differences between the three engines are getting fewer and fewer and it does seem like everyone is moving more to the standards that Google has defined. Is Google’s interface as it sits the de facto standard for a search results page now and if so, then in what areas does Yahoo differentiate itself? I’m talking more about the design of the page, white space, font usage, where the query bolding is…that type of thing.

Larry: There are a lot of really smart people in each of those companies. They also do their own user studies and they look at their metrics, so I think everyone is realizing over time that they’ve refined their search experience, what is working and what is not. So it’s not surprising to see some convergence in terms of the design and what seems to be most effective. I can’t really speak to whether Google is the de facto standard, but definitely they have a lot of eyeballs, so I think people do get use to seeing things in a certain way. I know from the Yahoo perspective that we want to do what’s best for our users. And I think we do have a user population that has certain expectations of us. I think a big part of that is the social search component, because people do think of Yahoo as a distinct company with its own brand, so there’s a lot that we want to do on that page that is completely independent of what other people might be doing because we want to do what’s best for our users.

Kathryn: And our users tend to be different than Google users. There is obviously overlap but we also have a distinct type of user than Google. We have to take that into consideration.

Gord: Can we go a little bit further down that road? Can we paint a picture of the Yahoo user and then explain how your interface is catering to their specific needs?

Larry: I can’t speak too much about it but one difference is that Yahoo is a lot of different things. We’re not just a search company, not just a mail company, not just a portal company; we serve a lot of needs. And we have a lot of tremendously popular, very effective properties that people use. Millions and millions come to the Yahoo network every day for a whole variety of reasons, so I think that’s one thing that’s different about the Yahoo user. They’re not coming to Yahoo for one purpose. There is often many, many purposes, so that’s something we definitely we have to take into consideration. And I think that’s one reason of many that we’re looking at social search. We know that our users are doing a lot of things in our network and it’s really effective if we’re aware of that.

Gord: So, rather then the task oriented approach with Google where their whole job is to get people in and out as quickly as possible, Yahoo Search supports that community approach where search is just one aspect of several things that people might be doing when they are engaged with the various properties?

Larry: We want to support whatever the user’s task is; and I think search is actually a very simple term and it encompasses a lot. People use search for a whole variety of reasons, millions and millions of reasons, so you have to be aware of what their intent is and, you talk a lot about that in your report, you support that. If they want to get in and out, that’s one task flow. If they want to have a place where they have access to data and information that is coming from their community, all the social information that they think is valuable, that’s another task flow. So, I think just being aware of the fact that search is multi-faceted, it’s not just a simple single type of task flow.

Kathryn: And another thing we talk about a lot is that Google is really about getting people off of their networks as fast as possible; we tend to want to keep people in our network and introduce them to other properties and experiences. So I think that’s also something that we take a look at.

Gord: So, what’s the challenge for Yahoo for search in the future, if you were looking at your whiteboard of the things that you’re tackling in 2007? We talked a little bit about social search, but as far as the user’s experience, what is the biggest challenge that has to be cracked over the next year or two?

Larry: We’ve been touching on that and I think the biggest challenge is really disambiguating intent. Really trying to understand what does the user want when they enter a few words into the search box. It’s not a lot to work with, obviously. So the biggest challenge is understanding the intent and giving them what they’re looking for, and doing that in the most effective way we can. Yes, probably not anything new but I’d say that is the biggest challenge.

Gord: And in dealing with that challenge, I would suspect that moving beyond the current paradigm is imperative in doing that. We’re used to interacting with search in a certain way, but to do what you’re saying we have to move quickly beyond the idea of a query and getting results back on a fairly static page.

Larry: There are certain expectations that users have, because search is search, and it’s been that way for many years, but I think you can see with our strategy with social search and what we’ve been doing with the integration of Yahoo Answers that it is a shift. And it’s showing that we believe for certain types of queries and for certain information that it’s very useful to bring it up, not just purely algorithmic results.

Gord: I’m just going to wrap up by asking one question, and I guess…somewhat of a self serving question, but with our eye tracking report, are there parts where we align with what you have found?

Larry: No…I found the report fascinating. I think you guys have done a wonderful job. It’s a very interesting read. There is a lot of great information there. And I think there is a lot that is in sync with some of our findings as well. So I think you definitely found some themes that make a lot of sense.

Gord: Thanks very much.

Next week, I talk with Justin Osmer, Senior Product Manager at Microsoft about the new Windows Live Search experience, how MSN Search fared in the eye tracking study, and how MSN Search evolved into the Live Search experience.

Bots and Agents – The Present and Future of A.I.

This past weekend I got started on a website I told a friend I’d help him build. I’ve been building websites for over 30 years now, but for this one, I decided to use a platform that was new to me. Knowing there would be a significant learning curve, my plan was to use the weekend to learn the basics of the platform. As is now true everywhere, I had just logged into the dashboard when a window popped up asking if I wanted to use their new AI co-pilot to help me plan and build the website.

“What the hell?” I thought, “Let’s take it for a spin!” Even if it could lessen the learning curve a little bit, it could still save me dozens of hours. The promise given me was intriguing – the AI co-pilot would ask me a few questions and then give me back the basic bones of a fully functional website. Or, at least, that’s what I thought.

I jumped on the chatbot and started typing. With each question, my expectations rose. It started with the basics: what were we selling, what were our product categories, where was our market? Soon, though, it started asking me what tone of voice I wanted, what was our color scheme, what search functionality was required, were there any competitor’s sites that we liked or disliked, and if so, what specifically did we like or dislike?  As I plugged my answers, I wondered what exactly I would get back.

The answer, as it turned out, was not much. As I was reassured that I had provided a strong enough brief for an excellent plan, I clicked the “finalize” button and waited. And waited. And waited. The ellipse below my last input just kept fading in and out. Finally, I asked, “Are you finished yet?” I was encouraged to just wait a few more minutes as it prepared a plan guaranteed to amaze.

Finally – ta da! – I got the “detailed web plan.” As far as I can tell, it had simply sucked in my input and belched it out again, formatted as a bullet list. I was profoundly underwhelmed.

Going into this, I had little experience with AI. I have used it sparingly for tasks that tend to have a well-defined scope. I have to say, I have been impressed more often than I have been disappointed, but I haven’t really kicked the tires of AI.

Every week, when I sit down to write this post, Microsoft Co-Pilot urges me to let it show what it can do. I have resisted, because when I do ask AI to write something for me, it reads like a machine did it. It’s worded correctly and usually gets the facts right, but there is no humanness in the process. One thing I think I have is an ability to connect the dots – to bring together seemingly unconnected examples or thoughts and hopefully join them together to create a unique perspective. For me, AI is a workhorse that can go out and gather the information in a utilitarian manner, but somewhere in the mix, a human is required to add the spark of intuition or inspiration. For now, anyway.

Meet Agentic AI

With my recent AI debacle still fresh in my mind, I happened across a blog post from Bill Gates. It seems I thought I was talking to an AI “Agent” when, in fact, I was chatting with a “Bot.” It’s agentic AI that will probably deliver the usefulness I’ve been looking for for the last decade and a half.

As it turns out, Gates was at least a decade and a half ahead of me in that search. He first talked about intelligent agents in his 1995 book The Road Ahead. But it’s only now that they’ve become possible, thanks to advances in AI. In his post, Gate’s describes the difference between Bots and Agents: “Agents are smarter. They’re proactive—capable of making suggestions before you ask for them. They accomplish tasks across applications. They improve over time because they remember your activities and recognize intent and patterns in your behavior. Based on this information, they offer to provide what they think you need, although you will always make the final decisions.”

This is exactly the “app-ssistant” I first described in 2010 and have returned to a few times since, even down to using the same example Bill Gates did – planning a trip. This is what I was expecting when I took the web-design co-pilot for a test flight. I was hoping that – even if it couldn’t take me all the way from A to Z – it could at least get me to M. As it turned out, it couldn’t even get past A. I ended up exactly where I started.

But the day will come. And, when it does, I have to wonder if there will still be room on the flight for we human passengers?

Just Behave Archive: Q&A With Marissa Mayer, Google VP, Search Products & User Experience

This blog is the most complete collection of my various posts across the web – with one exception. For 4 years, from 2007 to 2011, I wrote a column for Search Engine Land called “Just Behave” (Danny Sullivan’s choice of title, not mine – but it grew on me). At the time, I didn’t cross-post because Danny wanted the posts to be exclusive. Now, with almost 2 decades past, I think it’s safe to bring these lost posts back home to the nest, here at “Out of My Gord”. You might find them interesting from a historical perspective, and also because it gave me the chance to interview some of the brightest minds in search at that time. So, here’s my first, with Google’s then VP of Search Products and User Experience – Marissa Mayer. It ran in January, 2007 :

Marissa Mayer has been the driving force behind Google’s Spartan look and feel from the very earliest days. In this wide-ranging interview, I talked with Marissa about everything from interface design to user behavior to the biggest challenge still to be solved with search as we currently know it.

I had asked for the interview because of some notable findings in our most recent eye tracking study. I won’t go into the findings in any great depth here, because Chris Sherman will be doing a deep dive soon. But for the purpose of setting the background for Marissa’s interview, here are some very quick highlights:


MSN and Yahoo Users had a better User Experience on Google

In the original study, the vast majority of participants were Google users, and their interactions were restricted to Google. With the second study, we actually recruited participants that indicated their engine of preference was Yahoo! or MSN (now Live Search), as the majority of their interactions would be with those two engines. We did take one task at random, however, and asked them to use Google to complete the task. By almost every metric we looked at, including time to complete the task (choose a link), the success of the link chosen, the percentage of the page scanned before choosing a link and others, these users had a more successful experience on Google than on their engine of choice.

Google Seemed to Have a Higher Degree of Perceived Relevancy

In looking at the results, we didn’t believe that it was the actual quality of the results that lead to a more successful user experience as much as it was how those results were presented to the user. Something about Google’s presentation made it easier to determine which results were relevant. We referred to it in the study as information scent, using the term common in the information foraging theory.

Google Has an Almost Obsessive Dedication to Relevancy at the Top of the Results Page

The top of the results, especially the top left corner, is the most heavily scanned part of the results page. Google seemed to be the most dedicated of all the three engines in ensuring the results that fall in this real estate are highly relevant to the query. For example, Google served up top sponsored ads in far fewer sessions in the study than did either Yahoo or MSN.

Google Offers the “Cleanest” Search Experience

Google is famous for its Spartan home page. It continues this minimalist approach to search with the cleanest results page. When searching, we all have a concept in mind and that concept can be influenced by what else we see on the page. Because a number of searches on Yahoo! and MSN were launched from their portal page, we wondered how that impacted the search experience.

Google Had Less Engagement than Yahoo with their Vertical Results

The one area where Google appeared to fall behind in these head to head tests was with the relevance of the OneBox, or their vertical results. Yahoo! in particular seemed to score more consistently with users with their vertical offerings, Yahoo! Shortcuts.

It was in these areas in particular that I wanted to get the thinking of Marissa and her team at Google. Whatever they’re doing, it seems to be working. In fact, I have said in the past that Google has set the de facto standard for what we expect from a search engine, at least for now.

Here’s the interview:

Gord: What, at the highest level, is Google’s goal for the user?

Marissa: Our goal is to make sure that people can find what they’re looking for and get off the page as quickly as possible

If we look at this idea of perceived versus real relevancy, some things seemed to make a big difference in how relevant people perceived the results to be on a search engine: things like how much white space there was around individual listings, separating organic results from the right rail, the query actually being bolded in the title and the description and very subtle nuances like a hair line around the sponsored ads as opposed to a screened box. What we found when we delved into it was there seemed to be a tremendous attention to that detail on Google. It became clear that this stuff had been fairly extensively tested out.

I think all of your observations are correct. I can walk you through any one of the single examples you just named and I can talk you through the background and exactly what our philosophy was when we designed it and the numbers we saw in our tests as we had tested them, but you’re right in that it’s not an accident. For example, putting a line along the side of the ad as opposed to boxing it allows it to integrate more into the page and lets it fall more into what people read.

One thing that I think about a lot are people that are new to the internet. A lot of times they subconsciously map the internet to physical idioms. For example, when you look at how you parse a webpage, chances are that there are some differences if there are links in the structure and so forth, but a lot of times it looks just like a page in a book or a page on a magazine, and when you put a box around something, it looks like a sidebar. The way people handle reading a page that has a sidebar on it is that they read the whole main page and then, at the end, if it’s not too interesting, they stop and read the sidebar on that page.

For us, given that we think our ads in some cases are as good an answer as our search results and we want them to be integral to the user experience, we don’t want that kind of segmentation and pausing. We tried not to design it so it looked like a side bar, even though we have two distinct columns. You know, There are a lot of philosophies like that that go into the results page and of course, testing both of those formats to see if that matches our hypothesis.

That brings up something else that was really interesting. If we separate the top sponsored from the right rail, the majority of the interaction happens on the page in that upper left real estate. One thing that became very apparent was that Google seemed to be the most aware of relevancy at that top of page, that Golden Triangle real estate. In all our scenarios, you showed top sponsored the least number of times and generally you showed fewer top sponsored results. We saw a natural tendency to break off the top 3 or 4 listings on a page and scan them as a set and then make your choice from those top 3 or 4. In Google, those top 3 or 4 almost always include 1 or 2 organic results, sometimes all organic results.

That’s absolutely the case. Yes, we’re always looking at how can we do better targeting with ads. But we believe part of the targeting for those ads is “how well do those ads match your query?” And then the other part is how well does this format and that prominence convey to you how relevant it is. That’s baked into the relevance.

Our ad team has worked very very hard. One of the most celebrated teams at Google is our Smart Ads team. In fact, you may have heard of the Google Founder’s Awards, where small teams of people get grants of stock of up to $10,000,000 in worth, split across a small number of individuals. One of the very first teams at Google to receive that award was the Smart Ads team. And they were looking, interestingly enough, at how you target things. But they were also looking at what’s the probability that someone will click on a result. And shouldn’t that probability impact our idea of relevance, and also the way we choose to display it.

So we do tend to be very selective and keep the threshold on what appears on the top of the page very high. We only show things on the top when we’re very very confident that the click through rate on that ad will be very high. And the same thing is true for our OneBox results that occasionally appear above the top (organic) results. Larry and Sergey, when I started doing user interface work, said we’re thinking of making your salary proportional to the number of pixels above the first result, on average. We’ve mandated that we always want to have at least one result above the fold. We don’t let people put too much stuff up there. Think about the amount of vertical space on top of the page as being an absolute premium and design it and program it as if your salary depended on it.

There are a couple of other points that I want to touch on. When we looked at how the screen real estate divided up on the search results page, based on a standard resolution, there seemed to be a mathematical precision to the Google proportions that wasn’t apparent on MSN and on Yahoo. The ratio seemed pretty set. We always seemed to come up with a 33% ratio dedicated to top organic, even on a fully loaded results page, so obviously that’s not by accident. That compared to, on a fully loaded page, less than 14% on Yahoo.

That’s interesting, because we never reviewed on a percentage basis that you’re mentioning. We’ve had a lot of controversy amongst the team, should it be in linear inches along the left hand margin, should it actually be square pixelage computed on a percentage basis? Because of the way that the search is laid out linear inches or vertical space may be more accurate. As I said, the metric that I try to hold the team to is always getting at least one organic result above the fold on 800 by 600, with the browser held at that size.

The standard resolution we set for the study was 1024 by 768.

Yes, we are still seeing as many as 30% plus of our users at 800 by 600. My view is, we can view 1024 by 768 as ideal. The design has to look good on that resolution. It has to at least work and appear professional on 800 by 600. So all of us with our laptops, we’re working with 1024 by 768 as our resolution, so we try to make sure the designs look really good on that. It’s obvious that some of our engineers have bigger monitors and bigger resolutions than that, but we always are very conscious of 800 by 600. It’s pretty funny, most of our designers, myself included, have a piece of wall paper that actually has rectangles in the back where if you line up the browser in the upper left hand corner and then align the edge of the browser with the box you can simulate all different sizes so we can make sure it works in the smaller browsers.

One of the members of our staff has a background in physics and design and he was the one that noticed that if you take the Golden Ratio it lined up very well with how the Google results page is designed. The proportions of the page lined up pretty closely with how that Ratio is proportioned.

I’m a huge fan of the Golden Ratio. We talk about it a lot in our design reviews, both implicitly and explicitly, even when it comes down to icons. We prefer that icons not be square, we prefer that they be more of the 1.7:1.

I wanted to talk about Google OneBox for a minute. Of all the elements on the Google page, frankly, that was the one that didn’t seem to work that well. It almost seemed to be in flux somewhat while we were doing the data collection. Relevancy seemed to be a little off on a number of the searches. Is that something that is being tested.

Can you give me an example?

The search was for digital cameras and we got news results back in OneBox. Nikon had a recall on a bunch of digital cameras at the time and we went, as far as disambiguating the user intent from the query, it would seem that news results for the query digital cameras is probably not the best match.

It’s true. The answer is that we do a fairly good job, I believe, in targeting our OneBox results. We hold them to a very high click through rate expectation and if they don’t meet that click through rate, the OneBox gets turned off on that particular query. We have an automated system that looks at click through rates per OneBox presentation per query. So it might be that news is performing really well on Bush today but it’s not performing very well on another term, it ultimately gets turned off due to lack of click through rates. We are authorizing it in a way that’s scalable and does a pretty good job enforcing relevance. We do have a few niggles in the system where we have an ongoing debate and one of them is around news versus product search

One school of thought is what you’re saying, which is that it should be the case that if I’m typing digital cameras, I’m much more likely to want to have product results returned. But here’s another example. We are very sensitive to the fact that if you type in children’s flannel pajamas and there’s a recall due to lack of flame retardation on flannel pajamas, as a parent you’re going to want to know that. And so it’s a very hard decision to make.

You might say, well, the difference there is that it’s a specific model. Is it a Nikon D970 or is it digital cameras, which is just a category? So it’s very hard on the query end to disambiguate. You might say if there’s a model number then it’s very specific and if only the model number matches in the news return the news and if not, return the products. But it’s more nuanced than that. With things like Gap flannel pajamas for children, it’s very hard to programmatically tell if that’s a category or a specific product. So we have a couple of sticking points.

So that would be one of the reasons why, for a lot of searches, we weren’t seeing product results coming back, and in a lot of local cases, we weren’t seeing local results coming back?. That would be that click through monitoring mechanism where it didn’t meet the threshold and it got turned off?

That’s right.

Here’s another area we explored in the study. Obviously a lot of searches from Yahoo or MSN Live Search get launched from a portal and the user experience if you launch from the Google home page is different. What does it mean as far as interaction with search results when you’re launching the search from what’s basically a neutral palette versus something that’s launched from a portal that colors the intent of the user as it passes them through to the search results?

We want the user to not be distracted, to just type in what they want and not be very influenced by what they see on the page, which is one reason why the minimalist home page works well. It’s approachable, it’s simple, it’s straightforward and it gives the user a sense of empowerment. This engine is going to do what they want it to do, as opposed to the engine telling them what they should be doing, which is what a portal does. We think that to really aid and facilitate research and learning, the clean slate is best.

I think there’s a couple of interesting problems in the portal versus simple home page piece. You might say it’s easier to disambiguate from a portal what a person might be intending. They look at the home page and there’s a big ad running for Castaway and if they search Castaway, they mean the movie that they just saw the ad for. That might be the case but the other thing that I think is more confusing than anything is the fact that most people who launch the search from the portal home page are actually ignoring and tuning out most of the content on a page. If anything you’re more inclined to mistake intent, to think, “Oh, of course when they typed this they meant that,” but they actually didn’t, because they didn’t even see this other thing. One thing that we’re consistently noticing, which your Golden Triangle finding validated, is that users have a laser focus on their task.

The Google home page is very simple and when we put a link underneath the Google search box on the home page to advertise one of our products, we say, “Hey, try Google video, it’s new, or download the new Picassa.” Basically it’s the only other thing on the page, and while it does get a fair amount of click through, it’s nothing compared to the search, because most users don’t even see it. Most users on our search results page don’t see the logo on the top of the page, they don’t see OneBox, they don’t even see spelling corrections, even though it’s there in bright red letters. There’s a single-mindedness of I’m going to put in my search, not let anything on the home page get in the way, and I’m going to go for the first blue left aligned link on the results page and everything above it basically gets ignored. And we’ve seen that trend again and again. My guess is that if anything, that same thing is happening at the portals but because there is so much context around it on the home page, their user experience and search relevance teams may be led astray, thinking that that context has more relevance than it has.

One thing eye tracking allowed us to pull this apart a little bit is that when we gave people two different scenarios, one aimed more towards getting them to look at the organic results and one that would have them more likely to look at sponsored results, and then look down to organic results, we saw the physical interaction with the page didn’t vary as much as we thought, but the cognitive interaction with the page, when it came to what they remembered seeing and what they clicked on, was dramatically different. So it’s almost like they took the same path through, but the engagement factor flicked on at different points.

My guess is that people who come to the portal are much more likely to look at ads. I like to think of them as users with ADHD. They’re on the home page and they enjoy a home page that pulls their attention in a lot of different directions. They’re willing to process a lot of information on the way to typing in their search, and as a result, that same mind that likes that, it may not even be a per user thing, it may be an of-the-moment thing, but a person that’s in the mindset of enjoying that, on the home page, is also going to be much more likely to look around on the search results page. Their attention is going to be much more likely to be pulled in the direction of an ad, even if it’s not particularly relevant, banner, brand, things like that.

I want to wrap up by asking you, what in your mind is the biggest challenge still to be solved with the search interface as we currently know it?

I think there’s a ton of challenges, because in my view, search is in its infancy, and we’re just getting started. I think the most pressing, immediate need as far as the search interface is to break paradigm of the expectation of “You give us a keyword, and we give you 10 URL’s”. I think we need to get into richer, more diverse ways you’re able to express their query, be it though natural language, or voice, or even contextually. I’m always intrigued by what the Google desktop sidebar is doing, by looking at your context, or what Gmail does, where by looking at your context, it actually produces relevant webpages, ads and things like that. So essentially, a context based search.

So, challenge one is how the searches get expressed, I think we really need to branch out there, but I also think we need to look at results pages that aren’t just 10 standard URLS that are laid out in a very linear format. Sometimes the best answer is a video, sometimes the best answer will be a photo, and sometime the best answer will be a set of extracted facts. If I type in general demographic statistics about China, it’d be great if I got “A” as a result. A set of facts that had been parsed off of and even aggregated and cross validated across a result set.

And sometimes the best result would be an ad. Out of interest, when we tracked through to the end of the scenario to see which links provided the greatest degree of success, the top sponsored results actually delivered the highest success rates across all the links that were clicked on in the study.

Really? Even more so than the natural search results?

Yes. Even the organic search results. Now mind you, the scenarios given were commercial in nature.

Right… that makes much more sense. I do think that for the 40 or so percent of page views that we serve ads on that those ads are incredibly relevant and usually do beat the search results, but for the other 60% of the time the search results are really the only reasonable answer.

Thanks, Marissa.

In my next column, I talk with Larry Cornett, Senior Director of Search & Social Media in Yahoo’s User Experience & Design group about their user experience. Look for it next Friday, February 2.