Posts by Mike Moran
While IBM Watson’s correct Jeopardy answers are remarkable in many ways, we can learn even more about the state of text analytics when we examine the wrong answers. Specifically, if you use a computer to analyze text and inform business decisions (e.g., social listening) you need to ensure that you have people in the process who review the computer’s output before you use it to make decisions. And, if you expect to achieve significant business outcomes through social media, then you need to make sure those people are supported by appropriate training and continuous process improvement.
One of the great ideas of Watson’s presence on Jeopardy is that the show exposes Watson’s “thoughts”. In addition to showing us Watson’s correct answers, we also see the runners-up. Often, the incorrect answers are “howlers” — answers so dumb to you and me that we can’t understand how a computer could even consider them.
In fact, many of Watson’s incorrect answers are so bad that you know they are wrong when they are given, even if you don’t know the right answer. For example, when Watson was given a category of ‘decades’, he didn’t always guess an actual decade. While you might not have known the right answer, you knew Watson’s answer was wrong.
Watson is a state-of-the-art pile of computing that is tuned to solve a very narrow (but challenging) problem: winning on Jeopardy. Bigger text analytics problems, such as listening to social media, have even more howlers, because:
- the problems we intend to solve through social intelligence are more open-ended than Jeopardy questions
- social media text varies a lot more than Jeopardy questions, and
- the answers we seek from social media vary significantly more than Jeopardy answers.
It’s understandable if buyers of social media monitoring services are a bit confused by all the numbers they hear in the market. I spent most of my career in text analytics, and I’ve been surprised at what I hear from monitoring companies about the accuracy of their text analytics.
It reminds me of Nigel Tufnel of the fictional rock group Spinal Tap, who “proved” that his heavy metal band was louder than others by pointing to the dials on their speakers, asserting, “The numbers all go to eleven” (which certainly beats all those other speakers that stop at 10). If you think that the accuracy claims of social media vendors sound a lot like “Ours go to eleven,” you may be right.
Let’s start by looking at what the problem is — or, rather, what the two problems are. After all, when you see the results in a social media dashboard, you need to evaluate two factors at once:
First, you are deciding whether this particular conversation is relevant — is it a conversation that actually talks about the issue that you are monitoring? Second, you want to know whether it is correctly identified as positive or negative. Your dashboard will truly be correct only if the vendor is right on both counts, and both of these problems can be very tricky in text analytics.
For relevance, you might be lucky. If you work for T-Mobile, it’s likely that every mention of “T-Mobile” is actually relevant, so algorithm-based text analytics software can do a good job at that. But if you have the same job at Sprint, you aren’t so lucky because many occurrences of the word “sprint” have nothing to do with phones. For example, such discussions might pertain to a high school track meet.
You might think that the algorithmic software could just look for a capital “S” in “Sprint” to find the right ones, but that doesn’t work very well, for lots of reasons. For example, people often skip proper capitalization when writing in social media, especially from mobile devices.
As usual, Google drops a new feature, and the blogosphere becomes overwrought with suggestions that SEO is dead. Honestly, if Google Instant kills your success in SEO, you’ve been doing it wrong.
We all need to take a deep breath and remember that search marketing is more about marketing than about search. If you get your knickers in a twist every time Google changes its ranking algorithm or its user interface, then you aren’t focusing on the marketing part, just the search part.
Yes, it’s true that Google Instant will change what people search for and what they click on, just as a change to Google’s ranking algorithm changes what sites get shown and clicked. So, if you’ve been happily sitting around in the #1 slot for a popular keyword, Google Instant might change how many people see and click on your site, because it might talk them into completing a different keyword than what they set out to enter.
Live with it.
If you’re focused on rankings or even traffic, you are focused on the wrong thing. It’s all about the conversions.
If you’ve been focusing on conversions, you haven’t been sitting around expecting that everyone will type in your favorite popular #1 keyword. You’ve been working on optimizing for many variations, including deep (long tail) keywords that few people look for.
Some people say that Google Instant will cause fewer people to search for those less popular keywords, because they’ll just follow the suggestions, but Google Suggest has been around a long time, so we’ve already seen this movie, and I wonder if something else will happen.
We already know that fewer and fewer people, with each passing year, go to page 2 of the search results, preferring to enter a second query and spin again. With Google Instant, might people scan the results as they type and enter longer keyword phrases until they see what they want coming up?
We should also remember that many people never use Google’s site, preferring toolbars from Google or from their browser. While those toolbars might suggest keywords, they certainly do not show search results on the fly.
And a certain Bing search engine seems to power almost one-quarter of all U.S. queries. So, whatever havoc Google Instant wreaks on search marketers will affect a lot of searches, but it doesn’t affect them all.
The bottom line with Google Instant is that, like most search engine changes, you win some and you lose some (and some are rained out). But if you’ve been focusing on everything that your searchers need, you’ll find that you weather changes better than if you calculate everything you do to please the way Google works right now.
If you are into Google-pleasing, then, when Google sneezes, you catch a cold.