Category Archives: Behavior & Psychology

Evolved Search Behaviors: Take Aways for Marketers

In the last two columns, I first looked at the origins of the original Golden Triangle, and then looked at how search behaviors have evolved in the last 9 years, according to a new eye tracking study from Mediative. In today’s column, I’ll try to pick out a few “so whats” for search marketers.

It’s not about Location, It’s About Intent

In 2005, search marketing as all about location. It was about grabbing a part of the Golden Triangle, and the higher, the better. The delta between scanning and clicks from the first organic result to the second was dramatic – by a factor of 2 to 1! Similar differences were seen in the top paid results. It’s as if, given the number of options available on the page (usually between 12 and 18, depending on the number of ads showing) searchers used position as a quick and dirty way to filter results, reasoning that the higher the result, the better match it would be to their intent.

In 2014, however, it’s a very different story. Because the first scan is now to find the most appropriate chunk, the importance of being high on the page is significantly lessened. Also, once the second step of scanning has begun, within a results chunk, there seems to be more vertical scanning within the chunk and less lateral scanning. Mediative found that in some instances, it was the third or fourth listing in a chunk that attracted the most attention, depending on content, format and user intent. For example, in the heat map shown below, the third organic result actually got as many clicks as the first, capturing 26% of all the clicks on the page and 15% of the time spent on page. The reason could be because it was the only listing that had the Google Ratings Rich Snippet because of the proper use of structured data mark up. In this case, the information scent that promised user reviews was a strong match with user intent, but you would only know this if you knew what that intent was.

Google-Ford-Fiesta

This change in user search scanning strategies makes it more important than ever to understand the most common user intents that would make them turn to a search engine. What will be the decision steps they go through and at which of those steps might they turn to a search engine? Would it be to discover a solution to an identified need, to find out more about a known solution, to help build a consideration set for direct comparisons, to look for one specific piece of information (ie a price) or to navigate to one particular destination, perhaps to order online? If you know why your prospects might use search, you’ll have a much better idea of what you need to do with your content to ensure you’re in the right place at the right time with the right content.  Nothing shows this clearer than the following comparison of heat maps. The one on the left was the heat map produced when searchers were given a scenario that required them to gather information. The one on the right resulted from a scenario where searchers had to find a site to navigate to. You can see the dramatic difference in scanning behaviors.

Intent-compared-2

If search used to be about location, location, location, it’s now about intent, intent, intent.

Organic Optimization Matters More than Ever!

Search marketers have been saying that organic optimization has been dying for at least two decades now, ever since I got into this industry. Guess what? Not only is organic optimization not dead, it’s now more important than ever! In Enquiro’s original 2005 study, the top two sponsored ads captured 14.1% of all clicks. In Mediative’s 2014 follow up, the number really didn’t change that much, edging up to 14.5% What did change was the relevance of the rest of the listings on the page. In 2005, all the organic results combined captured 56.7% of the clicks. That left about 29% of the users either going to the second page of results, launching a new search or clicking on one of the side sponsored ads (this only accounted for small fraction of the clicks). In 2014, the organic results, including all the different category “chunks,” captured 74.6% of the remaining clicks. This leaves only 11% either clicking on the side ads (again, a tiny percentage) or either going to the second page or launching a new search. That means that Google has upped their first page success rate to an impressive 90%.

First of all, that means you really need to break onto the first page of results to gain any visibility at all. If you can’t do it organically, make sure you pay for presence. But secondly, it means that of all the clicks on the page, some type of organic result is capturing 84% of them. The trick is to know which type of organic result will capture the click – and to do that you need to know the user’s intent (see above). But you also need to optimize across your entire content portfolio. With my own blog, two of the biggest traffic referrers happen to be image searches.

Left Gets to Lead

The Left side of the results page has always been important but the evolution of scanning behaviors now makes it vital. The heat map below shows just how important it is to seed the left hand of results with information scent.

Googlelefthand

Last week, I talked about how the categorization of results had caused us to adopt a two stage scanning strategy, the first to determine which “chunks” of result categories are the best match to intent, and the second to evaluated the listings in the most relevant chunks. The vertical scan down the left hand of the page is where we decide which “chunks” of results are the most promising. And, in the second scan, because of the improved relevancy, we often make the decision to click without a lot of horizontal scanning to qualify our choice. Remember, we’re only spending a little over a second scanning the result before we click. This is just enough to pick up the barest whiffs of information scent, and almost all of the scent comes from the left side of the listing. Look at the three choices above that captured the majority of scanning and clicks. The search was for “home decor store toronto.” The first popular result was a local result for the well known brand Crate and Barrel. This reinforces how important brands can be if they show up on the left side of the result set. The second popular result was a website listing for another well known brand – The Pottery Barn. The third was a link to Yelp – a directory site that offered a choice of options. In all cases, the scent found in the far left of the result was enough to capture a click. There was almost no lateral scanning to the right. When crafting titles, snippets and metadata, make sure you stack information scent to the left.

In the end, there are no magic bullets from this latest glimpse into search behaviors. It still comes down to the five foundational planks that have always underpinned good search marketing:

  1. Understand your user’s intent
  2. Provide a rich portfolio of content and functionality aligned with those intents
  3. Ensure your content appears at or near the top of search results, either through organic optimization or well run search campaigns
  4. Provide relevant information scent to capture clicks
  5. Make sure you deliver on what you promise post-click

Sure, the game is a little more complex than it was 9 years ago, but the rules haven’t changed.

Google’s Golden Triangle – Nine Years Later

Last week, I reviewed why the Golden Triangle existed in the first place. This week, we’ll look at how the scanning patterns of Google user’s has evolved in the past 9 years.

The reason I wanted to talk about Information Foraging last week is that it really sets the stage for understanding how the patterns have changed with the present Google layout. In particular, one thing was true for Google in 2005 that is no longer true in 2014 – back then, all results sets looked pretty much the same.

Consistency and Conditioning

If humans do the same thing over and over again and usually achieve the same outcome, we stop thinking about what we’re doing and we simply do it by habit. It’s called conditioning. But habitual conditioning requires consistency.

In 2005, The Google results page was a remarkably consistent environment. There was always 10 blue organic links and usually there were up to three sponsored results at the top of the page. There may also have been a few sponsored results along the right side of the page. Also, Google would put what it determined to be the most relevant results, both sponsored and organic, at the top of the page. This meant that for any given search, no matter the user intent, the top 4 results should presumably include the most relevant one or two organic results and a few hopefully relevant sponsored options for the user. If Google did it’s job well, there should be no reason to go beyond these 4 top results, at least in terms of a first click. And our original study showed that Google generally did do a pretty good job – over 80% of first clicks came from the top 4 results.

In 2014, however, we have a much different story. The 2005 Google was a one-size-fits-all solution. All results were links to a website. Now, not only do we have a variety of results, but even the results page layout varies from search to search. Google has become better at anticipating user intent and dynamically changes the layout on each search to be a better match for intent.

google 2014 big

What this means, however, is that we need to think a little more whenever we interact with a search page. Because the Google results page is no longer the same for every single search we do, we have exchanged consistency for relevancy. This means that conditioning isn’t as important a factor as it was in 2005. Now, we must adopt a two stage foraging strategy. This is shown in the heat map above. Our first foraging step is to determine what categories – or “chunks” of results – Google has decided to show on this particular results page. This is done with a vertical scan down the left side of the results set. In this scan, we’re looking for cues on what each chunk offers – typically in category headings or other quickly scanned labels. This first step is to determine which chunks are most promising in terms of information scent. Then, in the second step, we go back to the most relevant chunks and start scanning in a more deliberate fashions. Here, scanning behaviors revert to the “F” shaped scan we saw in 2005, creating a series of smaller “Golden Triangles.”

What is interesting about this is that although Google’s “chunking” of the results page forces us to scan in two separate steps, it’s actually more efficient for us. The time spent scanning each result is half of what it was in 2005, 1.2 seconds vs. 2.5 seconds. Once we find the right “chunk” of results, the results shown tend to be more relevant, increasing our confidence in choosing them.  You’ll see that the “mini” Golden Triangles have less lateral scanning than the original. We’re picking up enough scent on the left side of each result to push our “click confidence” over the required threshold.

A Richer Visual Environment

Google also offers a much more visually appealing results page than they did 9 years ago. Then, the entire results set was text based. There were no images shown. Now, depending on the search, the page can include several images, as the example below (a search for “New Orleans art galleries”) shows.

Googleimageshot

The presence of images has a dramatic impact on our foraging strategies. First of all, images can be parsed much quicker than text. We can determine the content of an image in fractions of a second, where text requires a much slower and deliberate type of mental processing. This means that our eyes are naturally drawn to images. You’ll notice that the above heat map has a light green haze over all the images shown. This is typical of the quick scan we do immediately upon page entry to determine what the images are about. Heat in an eye tracking heat map is produced by duration of foveal focus. This can be misleading when we’re dealing with images for two reasons. First, the fovea centralis is, predictably, in the center of our eye where our focus is the sharpest. We use this extensively when reading but it’s not as important when we’re glancing at an image. We can make a coarse judgement about what a picture is without focusing on it. We don’t need our fovea to know it’s a picture of a building, or a person, or a map. It’s only when we need to determine the details of a picture that we’ll recruit the fine-grained resolution of our fovea.

Our ability to quickly parse images makes it likely that they will play an important role in our initial orientation scan of the results page. We’ll quickly scan the available images looking for information scent. It the image does offer scent, it will also act as a natural entry point for further scanning. Typically, when we see a relevant image, we look in the immediate vicinity to find more reinforcing scent. We often see scanning hot spots on titles or other text adjacent to relevant images.

We Cover More Territory – But We’re Also More Efficient

So, to sum up, it appears that with our new two step foraging strategy, we’re covering more of the page, at least on our first scan, but Google is offering richer information scent, allowing us to zero in on the most promising “chunks” of information on the page. Once we find them, we are quicker to click on a promising result.

Next week, I’ll look at the implications of this new behavior on organic optimization strategies.

The Evolution of Google’s Golden Triangle

In search marketing circles, most everyone has heard of Google’s Golden Triangle. It even has it’s own Wikipedia entry (which is more than I can say). The “Triangle” is rapidly coming up to its 10th birthday (it was March of 2005 when Did It and Enquiro – now Mediative – first released the study). This year, Mediative conducted a new study to see if what we found a decade ago still continues to be true. Another study from the Institute of Communication and Media Research in Cologne, Germany also looked at the evolution of search user behaviors. I’ll run through the findings of both studies to see if the Golden Triangle still exists. But before we dive in, let’s look back at the original study.

Why We Had a Golden Triangle in the First Place

To understand why the Golden Triangle appeared in the first place, you have to understand about how humans look for relevant information. For this, I’m borrowing heavily from Peter Pirolli and Stuart Card at PARC and their Information Foraging Theory (by the way, absolutely every online marketer, web designer and usability consultant should be intimately familiar with this theory).

Foraging for Information

Humans “forage” for information. In doing so, they are very judicious about the amount of effort they go to find the available information. This is largely a subconscious activity, with our eyes rapidly scanning for cues of relevancy. Pirolli and Card refer to this as “information scent.” Picture a field mouse scrambling across a table looking for morsels to eat and you’ll have an appropriate mental context in which to understand the concept of information foraging. In most online contexts, our initial evaluation of the amount of scent on a page takes no more than a second or two. In that time, we also find the areas that promise the greatest scent and go directly to them. To use our mouse analogy, the first thing she does is to scurry quickly across the table and see where the scent of possible food is the greatest.

The Area of Greatest Promise

Now, Imagine that same mouse comes back day after day to the same table and every time she returns, she finds the greatest amount of food is always in the same corner. After a week or so, she learns that she doesn’t have to scurry across the entire table. All she has to do is go directly to that corner and start there. If, by some fluke, there is no food there, then the mouse can again check out the rest of the table to see if there are better offerings elsewhere. The mouse has been conditioned to go directly to the “Area of Greatest Promise” first.

Golden Triangle original

F Shaped Scanning

This was exactly the case when we did the first eye tracking study in 2005. Google had set a table of available information, but they always put the best information in the upper right corner. We became conditioned to go directly to the area of greatest promise. The triangle shape came about because of the conventions of how we read in the western world. We read top to bottom, left to right. So, to pick up information scent, we would first scan down the beginning of each of the top 4 or 5 listings. If we saw something that seemed to be a good match, we would scan across the title of the listing. If it was still a good match, we would quickly scan the description and the URL. If Google was doing it’s job right, there would be more of this lateral scanning on the top listing than there would be on the subsequent listings. This F shaped scanning strategy would naturally produce the Golden Triangle scanning pattern we saw.

Working Memory and Chunking

There was another behavior we saw that helped explain the heat maps that emerged. Our ability to actively compare options requires us to hold in our mind information about each of the options. This means that the number of options we can compare at any one time is restricted by the limits of our working memory. George Miller, in a famous paper in 1956, determined this to be 7 pieces of information, plus or minus two. The actual number depends on the type of information to be retained and the dimension of variability. In search foraging, the dimension is relevancy and the inputs to the calculation will be quick judgments of information scent based on a split second scan of the listing. This is a fairly complex assessment, so we found that the number of options to be compared at once by the user tends to max out about 3 or 4 listings. This means that the user “chunks” the page into groupings of 3 or 4 listings and determines if one of the listings is worthy of a click. If not, the user moves on to the next chunk. We also see this in the heat map shown. Scanning activity drops dramatically after the first 4 listings. In our original study, we found that over 80% of first clicks on all the results pages tested came from the top 4 listings. This is also likely why Google restricted the paid ads shown above organic to 3 at the most.

So, that’s a quick summary of our findings from the 2005 study. Next week, we’ll look how search scanning has changed in the past 9 years.

Note: Mediative and SEMPO will be hosting a Google+ Hang Out talking about their research on October 14th. Full details can be found here.

The Metaphysical Corporation and The Death of Capitalism?

Something strange is happening to companies. More and more, their business is being conducted in non-physical markets. Businesses used to produce stuff. Now, they produce ideas. A recent op-ed piece from Wharton speculated that companies are working their way up Maslow’s Hierarchy. The traditional business produced things that met the needs of the lowest levels of the pyramid – shelter, food, warmth, security. As consumerism spread, companies worked their way up to next levels: entertainment, attainment and enjoyment.  Now, the things that companies sell sit at the top of the pyramid – fulfillment, creativity, self-actualization.

ComponentsSP500_2010The post also talks about another significant shift that’s happening on the balance sheets of Corporate America. Not only are the things that corporations sell changing, but the things that make up the value of the company itself are also changing.  According to research by Ocean Tomo, a merchant bank that specializes in intellectual property, the asset mix of companies has shifted dramatically in the past 40 years. In 1975, tangible assets (buildings, land, equipment, inventory) made up 83% of the market value of the S&P 500 companies. By 2010, that had flipped – Intangible assets (patents, trademarks, goodwill and brand) made up 80% of the market value of the S & P 500.

Chains vs Networks and the Removal of Friction

Barry Libert, Jerry Wind and Megan Beck Finley, the authors of the Wharton piece, focus mainly on the financial aspects of this shift. They point out that general accounting principles (GAAP) are quickly falling behind this corporate evolution. For example, employees are still classified as an expense, rather than an asset. I’m personally more interested in what this shift means for the very structure of a corporation.

If you built stuff, you needed a supply chain. Vertical integration was the way to remove physical transactional friction from the manufacturing process. Vertical integration bred hierarchal management styles. Over time, technology would remove some of the friction and some parts of the chain may evolve into open markets. The automotive industry is a good example. Many of the components of your 2015 Fusion are supplied to Ford by independent vendors. Despite this, makers of “stuff” still want to control the entire chain through centralized management.

But if you sell ideas, you need to have a network. Intangible products don’t have any physical friction, so supply chains are not required. And if you try to control a network with a centralized hierarchy, branches of your network soon wither and die.

The New Real Thing

coca-cola-freestyle-machineCoke has not been a maker of stuff for quite some time now. Sure, they make beverages, so technically they’re quenching our thirst, but the true value of Coke lies in its brand and our connection to that brand. The “Real Thing” is, ironically and quite literally, a figment of our imagination. If you were to place Coke on Maslow’s Hierarchy – it wouldn’t sit on the bottom level (physiological) but on the third (Love/Belonging) or even the fourth (Esteem).

Coke is very aware of its personal connection with it’s customers and the intangibles that come with it. That’s why the Coca-Cola Freestyle Vending Machine comes with the marketing tag line: “So many options. Thirst isn’t one of them.” You can customize your own formulation from over 100 choices, and if you have the Freestyle app, you can reorder your brand at any Coke Freestyle machine in the world. Of course, Coke is quietly gathering all this customer data that’s generated, including consumption patterns and regional preferences. Again, this intimate customer insight is just one of the intangibles that is becoming increasing valuable.

Coke is not only changing how it distributes its product. It’s also grappling with changing its very structure. In a recent conversation I had with CMO Joe Tripodi, he talked a lot about Coke’s move towards becoming a networked corporation. Essentially, Coke wants to make sure that worldwide innovation isn’t choked off by commands coming from Atlanta.

The Turning Point of Capitalism

As corporate America moves away from the making of physical stuff and towards the creation of concepts that it shares with customers, what does that mean for capital markets? If you believe Jeremy Rifkin, in his new book The Zero Marginal Cost Society, he contends that capitalism is dying a slow death. Eventually, it will be replaced by a new collaborative common market made possible by the increasing shrinkage of marginal costs. As we move from the physical to the metaphysical, the cost of producing consumable services or digital concept-based products (books, music, video, software) drops dramatically. Capital was required to overcome physical transactional friction. If that friction disappears, so does the need for capital.   Rifkin doesn’t believe the death of capitalism will be any time soon, but he does see an inevitable trend towards a new type of market he calls the Collaborative Commons.

Get Intimate

My last takeaway is this – if future business depends on connecting with customers and their conceptual needs, it becomes essential to know those customers on a deeply intimate level.  Throw away any preconceptions from the days of mass marketing and start thinking about how to connect with the “Market of One.”

A Prospect Ignored isn’t Really a Prospect

asleep at work / schoolI’ve ranted about this before and – oh yes – I shall rant again!

But first – the back-story.

I needed some work done at a property I own. I found three contractors online and reached out to each of them to get a quote.

Cue crickets.

No response. Nothing! So a few days later I politely followed up with each to prod the process along. Again, nothing. Finally, after 4 weeks of repeated e-nagging, one finally coughed up a quote. Most of the details were wrong, but at least someone at the other end was responding with minimal signs of consciousness.

Fast-forward 2 months. The work is still not done. At this point, I’m still trying to convey the specifics of the job and to get an estimated timeline. If I had an option, I’d take it. But the sad fact is, as spotty as the communication is with my contractor of choice, it’s still better than his competitors. One never did respond, even after a number of emails and voicemails. One finally sent a quote, but it was obvious he didn’t want the work. Fair enough. If the laws of supply and demand are imbalanced this much in their favor, who am I to fight it?

But here’s the thing. Market balances can change on a dime. Someday I’ll be in the driver’s seat and they’ll be scrambling to line up work to stay in business. And when they reach out to their contact list, a lot of those contacts will respond with an incredulous WTF. If you didn’t want my business when I needed you, why would you think I would give you it when you need me? A prospect spurned has a long memory for the specifics of said spurning. So, Mr. (or Ms.) Contractor, you can go take a flying leap.

If you’re going to use online channels to build your business, don’t treat it like a tap you can turn on and off at your discretion. Your online prospects have to be nurtured. If you can’t take any new business on, that’s fine. But at least have enough respect for them to send a polite response explaining the reason you can’t do the work. As long as we prospects are treated with respect, you’d be amazed at how reasonable we can be. Perhaps we can schedule the job for when you do have time. At the very least, we won’t walk away from the interaction with a bitter taste that will linger for years to come.

In 2005, Benchmark Portal did a study to compare response rates for email requests. The results were discouraging. Over 50% of SMB’s never responded at all. Only a small fraction actually managed to respond within 24 hours of the request.

I would encourage you to do a little surreptitious checking on your own response rates. Prospects contacting you need your help, and none of us like to hear our pleas for help go unanswered. 24 hours may seem like a reasonable time frame to you, but if you’re on the other end, it’s more than enough time to see your enthusiasm cool dramatically. Make it someone’s job to field online requests and set a 4-hour response time limit. I’m not talking about an auto-generated generic email here. I’m talking about a personalized response that makes it clear that someone has taken the time to read your request and is working on it. Also give a clear indication of how long it will take to follow up with the required information.

Why are these initial responses so critical? It’s not just to keep your field of potential prospects green and growing. It’s also because we prospects are using something called “signaling” to judge future interactions with a business. When we reach out to a new business we find online, we have no idea what it will be like to be their customer. We don’t have access to that information. So, we use things we do know as a proxy for that information. These things provide “signals” to help us fill in the blanks in our available information. An example would be hiring new employees. We don’t know how the person we’re interviewing will perform as an employee, so we look for certain things in a resume or an interview to act as signals that would indicate that the candidate will perform well on the job if hired.

If I’m a prospect looking for a business – especially one providing a service that will require an extended relationship between the business and myself – I need signals to show me how reliable the business will be if I chose them. Will they get the work done in a timely manner? Will the quality of the work be acceptable? Will they be responsive and accommodating to my requirements? If problems arise, will they be willing to work through those problems? Those are all questions I don’t have the answer to. All I have are indications based on my current interactions with the business. And if those interactions have required my constant nagging and clarification to avoid incorrect responses, guess what my level of confidence might be with said business?

The Apple Watch – More Than Just a Pretty Face

wpid-iwatch-goldI just caught Tim Cook’s live streaming introduction of the Apple Watch (I guess they’ve given up the long running “i” naming theme). What struck me most is how arduously Apple has stuck with traditional touch points in introducing a totally new product category (well, new for Apple anyway).

If you glanced quickly across the room at someone wearing Apple’s new wonder, you probably wouldn’t even know they’re wearing technology. The Apple Watch looks a lot like an analog watch. There is even a Mickey Mouse face you can choose. The interchangeable bracelets smack of tradition. Jon Ive verified this point in the video that ran at the introduction, saying they borrowed heavily from the “watchmaker’s vocabulary” in the design process. They even consulted “horological experts from around the world” to provide a time keeping experience rooted in cultural nuance. The primary interface to the watch is a modified version of the very old fashioned watch-winding crown.

Now, appearances can be deceiving. As Cook, Ive and Kevin Lynch put the watch through its paces, it was clear that this is an impressive little piece of technology. Particular attention has been paid to making this an intimate device, with new advances in touch technology, biometric and motion sensors and the ability to personalize interfaces and hardware to make it uniquely yours. Watching, I couldn’t help but compare this to Google’s introduction of Google Glass. In many ways, Glass is the more revolutionary device. But the Apple Watch will have a much faster adoption path.

Google impresses first with sheer brute-force technological effort. Design is an afterthought. Google uses UI testing and design to try to corral a Pandora’s box full of raw innovation into a usable package. Apple takes a much different approach. They look first at the user experience and then they pick and choose the technologies required to deliver the intended experience. They lavish ridiculous amounts of time on seemingly miniscule design details but the end result is typically nothing less than breathtaking. We’re impressed with the technology, sure, but the overriding emotion is one of lust. We just have to have what ever the hell it is that is being introduced on the main stage of the Flint Center.

larrygiseleDespite the many who have said otherwise, including the late Steve Jobs, Apple has never really made a revolutionary device. Others have always been there first. What they have done, however, is taken raw innovation and packaged it in a way that resonates with its audience at a deep and hormonal level. Apple products are stylish and sexy – the Gisele Bündchen of technology – yet attainable to mere mortals. They take the “next big thing” and push them past the tipping point by kindling lust in the hearts and wallets of the market. Google products, despite their geeky technical prowess, have a nasty habit of getting stuck on the wrong side of the adoption curve. They are the – well, let’s face it – they are the Larry Page of technology – smart, but considerably less sexy.

Apple times entrance to the adoption curve to near perfection. They have a knack of positioning just ahead of the masses. Google’s target is much further down the road. They release betas well ahead of any market demand. That’s why most of us can’t wait to wear an Apple Watch, but wouldn’t be caught dead in a pair of Google Glass.

One last thought on this week’s introduction of the Apple Watch. Wearable technology is following an interesting path. Your smartphone now acts as a connected main base for more intimate pieces of tech like the Apple Watch or Google Glass. Increasingly, the actual user interfaces will be on these types of devices, but the heavy lifting will happen on a smart phone tucked into a pocket, purse or backpack. Expect specific purpose devices to proliferate, all connected to increasingly powerful MPUs (Mobile Processing Units) that will orchestrate the symphony of tech that you’re wearing.

Learning about Big Data from Big Brother

icreach-search-illo-feature-hero-bYou may not have heard of ICREACH, but it has probably heard of you. ICREACH is the NSA’s own Google-like search engine.  And if Google’s mission is to organize the world’s information, ICREACH’s mission is to snoop on the world.  After super whistle blower Edward Snowden tipped the press off to the existence of ICREACH, the NSA fessed up last month. The amount of data we’re talking about is massive. According to The Intercept website, the tool can handle two to five billion new records every day, including data on the US’s emails, phone calls, faxes, Internet chats and text messages. It’s Big Brother meets Big Data.

I’ll leave aside for the moment the ethical aspect of this story.  What I’ll focus on is how the NSA deals with this mass of Big Data and what it might mean for companies who are struggling to deal with their own Big Data dilemmas.

Perhaps no one deals with more big data than the Intelligence Community. And Big Data is not new for them. They’ve been digging into data trying to find meaningful signals amongst the noise for decades. Finally, the stakes of successful data analysis are astronomically high here. Not only is it a matter of life and death – a failure to successfully connect the dots can lead to the kinds of nightmares that will haunt us for the rest of our lives. When the pressure is on to this extent, you can be sure that they’ve learned a thing or two. How the Intelligence community handles data is something I’ve been looking at recently. There are a few lessons to be learned here.

Owned Data vs Environmental Data

The first lesson is that you need different approaches for different types of data. The Intelligence Community has their own files, which include analyst’s reports, suspect files and other internally generated documentation. Then you have what I would call “Environmental” data. This includes raw data gathered from emails, phone calls, social media postings and cellphone locations. Raw data needs to be successfully crunched, screened for signals vs. noise and then interpreted in a way that’s relevant to the objectives of the organization. That’s where…

You Need to Make Sense of the Data – at Scale

Probably the biggest change in the Intelligence community has been to adopt an approach called “Sense making.”  Sense making really mimics how we, as humans, make sense of our environment. But while we may crunch a few hundred or thousand sensory inputs at any one time, the NSA needs to crunch several billion signals.

Human intuition expert Gary Klein has done much work in the area of sense making. His view of sense making relies on the existence of a “frame” that represents what we believe to be true about the world around us at any given time.  We constantly update that frame based on new environmental inputs.  Sometimes they confirm the frame. Sometimes they contradict the frame. If the contradiction is big enough, it may cause us to discard the frame and build a new one. But it’s this frame that allows us to not only connect the dots, but also to determine what counts as a dot. And to do this…

You Have to Be Constantly Experimenting

Crunching of the data may give you the dots, but there will be multiple ways to connect them. A number of hypothetical “frames” will emerge from the raw data. You need to test the validity of these hypotheses. In some cases, they can be tested against your own internally controlled data. Sometimes they will lie beyond the limits of that data. This means adopting a rigorous and objective testing methodology.  Objective is the key word here, because…

You Need to Remove Human Limitations from the Equation

When you look at the historic failures of Intelligence gathering, the fault usually doesn’t lie in the “gathering.” The signals are often there. Frequently, they’re even put together into a workable hypothesis by an analyst. The catastrophic failures in intelligence generally arise because some one, somewhere, made an intuitive call to ignore the information because they didn’t agree with the hypothesis. Internal politics in the Intelligence Community has probably been the single biggest point of failure. Finally…

Data Needs to Be Shared

The ICREACH project came about as a way to allow broader access to the information required to identify warning signals and test out hunches. ICREACH opens up this data pool to nearly two-dozen U.S. Government agencies.

Big Data shouldn’t replace intuition. It should embrace it. Humans are incredibly proficient at recognizing patterns. In fact, we’re too good at it. False positives are a common occurrence. But, if we build an objective way to validate our hypotheses and remove our irrational adherence to our own pet theories, more is almost always better when it comes to generating testable scenarios.

Twitch – Another Example of a Frictionless Market

twitch_logo3Twitch just sold for $1 billion dollars. That’s not really news. We’ve become inured to the never-ending stream of tech acquisitions that instantly transforms entrepreneurial techies into some of the richest people on the planet. No, what’s interesting about Twitch is if we slow down long enough to think about how this particular start up managed to create $1 billion in value.

A billion dollars is a lot of money. If we looked back just 50 years, a billion dollars in assets would make a company number 40 on the Fortune 500. If Twitch were somehow teleported back to 1964, it would rank just eight slots under Procter and Gamble (assets worth $1.15 billion) and three slots above Sunoco (assets of $0.88 billion). Coca-Cola would be left in the dust with a mere $485 million in assets. Today a half billion dollars is chump change in Silicon Valley terms.

This becomes more amazing when you consider that Twitch is only 3 years old. And it really started as an accident.

justin_kanRemember EDtv? Probably not. It was a pretty forgettable 1999 movie (based on a 1994 Quebec film called Louis 19, King of the Airwaves) starring Matthew McConaughey. The idea was that Ed would be followed by cameras 24 hours a day, 7 days a week, making his life a reality TV show. 1998’s The Truman Show had a similar theme (albeit with better ratings). Anyway, the point made in both movies was that an average life, if televised, could be entertaining enough to make people watch. In 2006, Emmett Shear and Justin Kan decided to test the premise. They launched Justin.tv. Soon they invited others to simulcast their lives as well.

What Kan and Shear did, although they probably weren’t intending to at the time, was create a platform that allowed anyone to be a real-time broadcaster with zero transactional costs. They created a perfect market for live TV. Last week I talked about AirBnB, TripAdvisor and VRBO.com creating a more perfect market for tourism. The key characteristic of a perfect market is that barriers to entry are reduced to zero, turning the market into an emergent sandbox from which new things tend to pop up. And that’s exactly what happened with Twitch.

Shear and Kan found that one group in particular embraced the idea of livecasting – gamers. They could communicate with other gamers, but they could also show off their mad gaming skills. Using the Justin.tv platform, Twitch was launched for the gaming industry in 2011. And thanks to Twitch, gaming has become a spectator sport – at a massive scale.

Twitch’s “stars” – like 30-year-old Tessa Brooks, who goes by “Tessachka” and broadcasts an average of 42 hours of programming a week – post their schedules so that their audiences can tune in. Twitch has about 55 million viewers per month who consume over 16 billion minutes of video programming. According to SocialBlade.com, this month, “Riotgames” is the top ranked Twitch broadcaster, with almost a million followers and over 18 million channel views.

Again, those are big numbers. A network show that pulls in 18 million viewers would be number 5 in the Nielsen ratings. And while Netflix’s House of Cards or Orange is the New Black may have made waves at the Emmies, The Atlantic estimates that only 2 – 3 million people watch a newly posted episode in the first week. On a good week, Riotgames could blow that away without twitching a trigger finger.

Twitch not only created a platform that generates audiences, it also generated a marketplace. Where there are eyeballs, there’s revenue potential. Twitch cuts its gamers in for a cut of the advertising revenue. I couldn’t find numbers on how lucrative this could be, but I suspect Justin may be able to quit his day job.

Like I said, the Twitch story is interesting, but what is vastly more interesting is the market dynamics that it has unleashed. Amazon’s $1 billion bid is not for the technology. It’s for the community and the market that comes with that community. When it comes to leveraging the potential of zero transactional cost markets, Amazon knows a thing or two. And one of the things it knows is that in frictionless markets, if you can navigate the turbulence, tremendous value can be created in an amazing short time. Say, for instance, $1 billion in just 3 years. It took Procter and Gamble 127 years to be worth that much.

Technology is Moving Us Closer to a Perfect Market

I have two very different travel profiles. When I travel on business, I usually stick with the big chains, like Hilton or Starwood. The experience is less important to me than predictability. I’m not there for pleasure; I’m there to sleep. And, because I travel on business a lot (or used to), I have status with them. If something goes wrong, I can wave my Platinum or Diamond guest card around and act like a jerk until it gets fixed.

But, if I’m traveling for pleasure, I almost never stay in a chain hotel. In fact, more and more, I stay in a vacation rental house or apartment. It’s a little less predictable than your average Sheraton or Hampton Inn, but it’s almost always a better value. For example, if I were planning a last minute get away to San Francisco for Labor Day weekend, I’d be shelling out just under $400 for a fairly average hotel room at the Hilton by Union Square. But for about the same price, I could get an entire 4 bedroom house that sleeps 8 just two blocks from Golden Gate park. And that was with just a quick search on AirBnB.com. I could probably find a better deal with the investment of a few minutes of my time.

perfect_market_1Travel is just one of the markets that technology has made more perfect. And when I say “perfect” I use the term in its economic sense. A perfect market has perfect competition, which means that the barriers of entry have been lowered and most of the transactional costs have been eliminated. The increased competition lowers prices to a sustainable minimum. At that point, the market enters a state called the Pareto Optimal, which means that nothing can be changed without it negatively impacting some market participants and positively impacting others.

Whether a perfect market is a good thing or not depends on your perspective. If you’re a long-term participant in the market and your goal is to make the biggest profit possible, a perfect market is the last thing you want. If you’re a new entrant to the market, it’s a much rosier story – any shifts that take the market closer to a Pareto Optimal will probably be to your benefit. And if you’re a customer, you’re in the best position of all. Perfect markets lead inevitably to better value.

Since the advent of VRBO.com and, more recently, AirBnB.com, the travel marketplace has moved noticeably closer to being perfect. Sites like these, along with travel review aggregators like TripAdvisor.com, have significantly reduced the transaction costs of the travel industry. The first wave was the reduction of search costs. Property owners were able to publish listings in a directory that made it easy to search and filter options. Then, the publishing of reviews gave us the confidence we needed to stray beyond the predictably safe territory of the big chains.

But, more recently, a second wave has further reduced transaction costs independent vacation property owners. I was recently talking to a cousin who rents his flat in Dublin through AirBnB, which takes all the headaches of vacation property management away in return for a cut of the action. He was up and running almost immediately and has had no problem renting his flat during the weeks he makes it available. He found the barriers to entry to be essentially zero. A cottage industry of property managers and key exchange services has sprung up around the AirBnB model.

What technology has done to the travel industry is essentially turned it into a Long Tail business model. As Chris Anderson pointed out in his book, Long Tail markets need scale free networks. Scale free networks only work when transaction costs are eliminated and entry into the market is free of friction. When this happen, the Power Law distribution still stays in place but the tail becomes longer . The Long Tail of Tourism now includes millions of individually owned vacation properties. For example, AirBnB has almost 800 rentals available in Dublin alone. According to Booking.com, that’s about 7 times the total number of hotels in the city.

Another thing that happens is, over time, the Tail becomes fatter. More business moves from the head to the tail. The Pareto Principle states that in Power Law distributions, 20 % of the businesses get 80% of the business. Online, the ratio is closer to 72/28.

These shifts in the market are more than just interesting discussion topics for economists. They mark a fundamental change in the rules of the game. Markets that are moving towards perfection remove the advantages of size and incumbency and reward nimbleness and adaptability. They also, at least in this instance, make life more interesting for customers.

Why Cognitive Computing is a Big Deal When it comes to Big Data

IBM-Watson

Watson beating it’s human opponents at Jeopardy

When IBM’s Watson won against humans playing Jeopardy, most of the world considered it just another man against machine novelty act – going back to Deep Blue’s defeat of chess champion Garry Kasporov in 1997. But it’s much more than that. As Josh Dreller reminded us a few Search Insider Summits ago, when Watson trounced Ken Jennings and Brad Rutter in 2011, it ushered in the era of cognitive computing. Unlike chess, where solutions can be determined solely with massive amounts of number crunching, winning Jeopardy requires a very nuanced understanding of the English language as well as an encyclopedic span of knowledge. Computers are naturally suited to chess. They’re also very good at storing knowledge. In both cases, it’s not surprising that they would eventually best humans. But parsing language is another matter. For a machine to best a man here requires something quite extraordinary. It requires a machine that can learn.

The most remarkable thing about Watson is that no human programmer wrote the program that made it a Jeopardy champion. Watson learned as it went. It evolved the winning strategy. And this marks a watershed development in the history of artificial intelligence. Now, computers have mastered some of the key rudiments of human cognition. Cognition is the ability to gather information, judge it, make decisions and problem solve. These are all things that Watson can do.

 

Peter Pirolli - PARC

Peter Pirolli – PARC

Peter Pirolli, one of the senior researchers at Xerox’s PARC campus in Palo Alto, has been doing a lot of work in this area. One of the things that has been difficult for machines has been to “make sense” of situations and adapt accordingly. Remember, a few columns ago where I talked about narratives and Big Data, this is where Monitor360 uses a combination of humans and computers – computers to do the data crunching and humans to make sense of the results. But as Watson showed us, computers do have to potential to make sense as well. True, computers have not yet matched humans in the ability to sense make in an unlimited variety of environmental contexts. We humans excel at quick and dirty sense making no matter what the situation. We’re not always correct in our conclusions but we’re far more flexible than machines. But computers are constantly narrowing the gap and as Watson showed, when a computer can grasp a cognitive context, it will usually outperform a human.

Part of the problem machines face when making sense of a new context is that the contextual information needs to be in a format that can be parsed by the computer. Again, this is an area where humans have a natural advantage. We’ve evolved to be very flexible in parsing environmental information to act as inputs for our sense making. But this flexibility has required a trade-off. We humans can go broad with our environmental parsing, but we can’t go very deep. We do a surface scan of our environment to pick up cues and then quickly pattern match against past experiences to make sense of our options. We don’t have the bandwidth to either gather more information or to compute this information. This is Herbert Simon’s Bounded Rationality.

But this is where Big Data comes in. Data is already native to computers, so parsing is not an issue. That handles the breadth issue. But the nature of data is also changing. The Internet of Things will generate a mind-numbing amount of environmental data. This “ambient” data has no schema or context to aid in sense making, especially when several different data sources are combined. It requires an evolutionary cognitive approach to separate potential signal from noise. Given the sheer volume of data involved, humans won’t be a match for this task. We can’t go deep into the data. And traditional computing lacks the flexibility required. But cognitive computing may be able to both handle the volume of environmental Big Data and make sense of it.

If artificial intelligence can crack the code on going both broad and deep into the coming storm of data, amazing things will certainly result from it.