Beneath the Surface of Search

Itâ??s 2007 already. If you are like most merchants, youâ??ve followed the advice of your NSO firm and completed some basic site optimization projects. You routinely spot check your Google indexation, and your rankings on 100 or so â??trophyâ?? keywords to show your executive team. And a look at your web analytics shows your natural search channel sales growing. So whatâ??s wrong with this picture?

The problem is you are staring at the tip of the iceberg, oblivious to the iceberg below. Something gigantic lies beneath. And you need more sophisticated tools and methods to understand and diagnose the opportunity in order to take appropriate action. In fact, chances are high that your natural search channel is seriously underperforming as a result.

As NSO best practices become commonplace amongst your competition, taking this evolutionary step is required for merchants to gain advantage. Fortunately, by approaching the natural search channel with an understanding of long tail dynamics, new key performance indicators (KPIs), and yield management techniques, savvy merchants can move beyond the traditional project-based mentality of NSO, to a more sophisticated, yet DM-friendly approach â?? one that leverages website size, brand power, and direct-marketing principles to capture the enormous potential that lies beneath the surface.

The Long Tail of Natural Search

To understand the KPIs and management techniques, letâ??s examine what lies beneath the surface. In a word, it is not an iceberg, but the â??long tailâ?? – a colloquial term referring to a common statistical distribution featuring a â??tail-shapedâ?? curve. Thanks to Chris Andersenâ??s book, the long tail concept has received much press recently, and carries many significant Internet business-based implications. But there is also a long tail distribution that contains the promise of natural search for large, dynamic e-commerce merchants.

Source Netconcepts, 2006

In fact, to appreciate the enormity of this potential, contrasting it against a known quantity is helpful. According to research published recently by Netconcepts (â??Chasing the Long Tail of Natural Searchâ??, 2006) for every search that occurs for the average merchantâ??s brand name, nearly 40 relevant searches occur for more generic, brand-neutral keywords. For instance, if there are 100,000 searches for â??LL Beanâ?? this month, there are an estimated 4,000,000 searches for hundreds of thousands of â??unbrandedâ?? keyword markets LL Bean pages could compete within, ranging from â??furniture slipcoversâ?? to â??womenâ??s flannel pajamasâ?? to â??menâ??s reindeer sweater,â?? and more.

For these coveted and targeted keywords, LL Bean is likely to have matching category or product pages somewhere on the website, that ideally could position the LL Bean brand high enough in the search results to win these millions of click decisions (and sales) on the cheap. And even if these pages do not convert 100% of these searches into clicks, if well ranked, these pages still serve the purpose of building brand by association with these millions of corresponding â??unbrandedâ?? searches, and by extension, the searchers that used them.

This is the long tail of natural search â?? the universe of diverse, unbranded keyword markets that, while perhaps not as frequently searched on an individual basis, cumulatively, add up to orders-of-magnitude greater (40x, in our research) search potential than the brand search traffic that most merchants naturally receive. This is the prize every savvy merchant seeks.

Since brand popularity is relative, a more objective way to think of this long tail potential is as a function of your website size. In particular, our research suggests that for every unique page that exists on a merchantâ??s website, nearly 100 unbranded searches are conducted in an average month. If you have a 20,000 page website, your long tail potential would be calculated as 2,000,000 unbranded searches, and so on. Tracking your unique pages and their yield therefore is a critical measurement to understanding the dynamics of your natural search channel performance.

Capturing the Elusive Tail

The long tail of unbranded keywords is a new entity for most marketers to grapple with. What drives it? How do you quantify it? What is good performance? What is bad? How do you capture it?

We set out to answer these questions in the research alluded to earlier. To this end, we developed a theory we refer to as â??Page Yield Theory.â?? Page Yield Theory aims to breakdown, quantify, and model the components of a merchantâ??s long tail into manageable units, and was informed via natural search data gathered from a few dozen top online merchant (clients utilizing our GravityStreamâ?¢ natural search optimization proxy technology). In the process, we developed a set of metrics to understand the dynamics of the long tail. We have since applied these same ratios as key performance indicators (KPIs) explained below, for growing natural search channel programs under management.

KPI #1 â?? Brand to Non-Brand Mix

What percentage of your natural search comes from brand keywords versus non-brand keywords? What does it mean if your keyword curve is dominated by brand terms? If you discover that most of your traffic is coming from searches for your brand, we view this as symptomatic of a larger problem lying just beneath the surface – that very few of your pages are actually yielding traffic.

Many retailers find 95% of their search traffic comes for brand terms with, not coincidentally, a very small percentage of their site powering that brand traffic. Whereas what you may find, when you have done some search optimization, is a very different distribution curve. You may find that 40% of your pages are yielding traffic, but that 60% of that traffic is unbranded keyword traffic. This in fact, is the core hypothesis of Page Yield Theory â?? unbranded keyword traffic volume grows as the number of pages yielding natural search traffic grows.

KPI #2 â?? Unique Pages

Just how big is your website? How many pages are there? This is a critical metric for establishing a yield management foundation. You may be tempted to approximate this number using a product count. This will most likely significantly understate your actual pages. Or you might want to use a search engineâ??s reported index. This too will return mixed results, as each engine indexes dramatically different amounts of pages for each website.

However, search engines are in the business of discovering pages. So we recommend using the number of pages that bots (such as googlebot) are able to crawl over a 30 â?? 60 day period as the most useful proxy for how many unique pages are available on your site. (This assumes your URLs are in a condition that permits bots to crawl deeply. If they are not, you may need to resort to an approximation.)

As a comparison, the average merchant in our research had roughly 73,000 uniquely crawled pages.

Note: Most analytical packages do not track page crawl metrics because they rely on Javascript to track user actions. Since search engine bots do not have Javascript enabled, their crawl activity goes unregistered â?? leaving marketers blind to the dynamics of their “search conversion funnel.

KPI #3 â?? Pages Yielding Traffic

What percentage of those pages yield search traffic? Is it 10%? 90%? This ratio essentially dictates the length (the X axis) of your unbranded keyword long tail, and suggests remaining potential.

With this KPI in hand, you understand your missed opportunity cost, and monitor your progress throughout your NSO engagement.

If you do not know your amount of unique pages or yielding pages, the above matrix can be consulted for an estimate. It was developed using data from the average merchant in our research and illustrates the inverse relationship between Page Yield and Brand / Non-Brand Traffic. Simply seek your current level of brand and non-brand traffic to estimate how many pages are accounting for that traffic.

The average merchant in our study had 14% of pages yielding traffic.

KPI #4 â?? Keywords per Page Yield

Now that you have an idea of your Page Yield rate, how many keywords does each of those producing pages yield over the course of a month â?? 2 keywords? 10? This is the fourth KPI: keyword yield. This KPI is responsible for creating scale. That is, the more keywords each yielding page attracts or targets, the longer your tail.

The average merchant in our study found 2.4 keywords produced per yielding page.

KPI #5 â?? Visitor per Keyword Yield

This fifth KPI â?? Visitor per Keyword Yield â?? is basically determined by how highly a page ranks for a keyword, and tells you how much traffic each keyword drives. This metric determine the height or thickness of your long tail.

The average merchant in our study experienced 1.9 visitors per keyword.

KPI #6 â?? Index-to-Crawl Ratios

The first step in the search conversion funnel is converting a crawled page to an indexed page. While we do not put significant stock in the reported engine indexes, this KPI can illustrate engine differences or potential trouble-spots.

For instance, the average merchant in our study saw a 3:1 index-to-crawl ratio in Google. That is, for every page that was crawled, there were 3 in the index, as strange as it sounds. If a merchant finds the index shrinking to say, only 50% of the crawled pages, that may be a signal of crawl pattern changes, or perhaps pages are being shifted into the supplemental index.

KPI #7 â?? Engine Yield

A final KPI to monitor is engine yield. Each engine has different audience sizes. How do you fairly compare the referral traffic you get from each? Simply calculate how much referral traffic each sends for every page it crawls or consumes. Compare engine by engine. What we have found is that MSN and Yahoo! tend to crawl a lot more pages, but the yield per crawled page from Google is typically significantly higher.

Reconstructing the Tail

Once known, these KPIs can be reassembled to create a vivid picture of your performance and empower smarter decisions. For instance, brand and non-brand traffic composition (KPI #1) provide a baseline on the effectiveness of your channel yield. This can be contrasted against your total available pages (KPI #2) and actual yielding pages (KPI #3) to understand the gap between current and potential performance. The rate at which these pages yield keywords (KPI #4), times the visitor driven by each of those keywords (KPI #5) form the length, and height, of your tail, and can each be optimized individually. Meanwhile your index-to-crawl ratio (KPI #6) and engine yield (KPI #7) help quantify your siteâ??s effectiveness at converting web pages into units of â??search produce.â??

For instance, if the number of pages yielding traffic is low (under 10%), it may be time for some basic, scalable, URL rewriting to maximize the flow of link juice through your internal link text. On the other hand, if 80% of pages are yielding traffic at a rate of one keyword per page and one visitor per keyword, it may be time to conduct some auto-generated title tag tests to scalably lift rankings of thousands of pages, by incorporating more relevant keywords per page. And perhaps a pinch of organic link building via RSS may be in order to power-up the link juice that flows through the inbound link text. Each situation is unique and requires a custom approach to optimization.

The secret to capturing the long tail of natural search is this: Fully maximize the page yield of your website. That is, develop a metrics-driven optimization process that empowers each page to be viewed as the authority on the subject â?? be it for â??furniture slipcovers,â?? â??menâ??s reindeer sweater,â?? â??18 volt Makita power drillâ?? or other. This is not without challenge; empowering thousands of pages to yield natural search traffic may seem daunting, but it can be achieved by focusing on these new KPIs, managing your NSO channel effort against these yield metrics, and committing to an iterative NSO testing program.

Increasingly, the natural search game revolves around getting your brand associated with unbranded terms. While these new KPIs do not change the basics of NSO, they can provide the context and management discipline that enable marketers to make more informed decisions how to best spend time and budget optimizing your long tail.

Comments

[…] Beneath the Surface of Search […]

Reply

Excellent article. KPIs are so important for SEO. Rankings are great, but what else are we doing? How do we truly measure success? This is a great article outlining quite a variety of things that we need to realize are playing a part of SEO. Thanks again!

Reply

Good article but there are a number of other issues which should also be taken into considerations for covering the complete range of KPIs.

Also I’ve doubt over the KPI #1 â?? Brand to Non-Brand Mix.

What do you term as non-branded? And why would you want to have non-branded content on your website first of all?

Do you mean not-main-keywords are non-branded?

Also the KPIs listed here are more or less oriented towards the sales/marketing guys and doesn’t leave much scope for SEO gurus.

For instance I can make a website appear on the top for given keywords which not necessarily always generates more revenue and which will be completely ignored by marketing people.

any way good food for brain 🙂

Reply

Great insights. I always ignore more than 6 word keywords though as you said I find them too daunting to optimize for.

It could be nice to implement a script which will automatically search the site for a long tail keyword of 5 or so words and return the page accordingly for product based pages instead of random landing.

You can even analyze your logs for long tail keywords (those getting more traffic) and create special landing pages for them and direct the visitors to those pages without breaking the real page which has the long tail.

Reply

I think its very important to note this to any client or any business that has many terms they can but are not vying for. Is the KW discovery portion of this also built in to your product services? if there are tens of thousands of keywords who generates these, I have a set of tools and logic to do this but it takes a human processor at the end of the day to look at what are possibly high yield terms. Can you explain any further this notion of bringing out the value of longtail?

Reply

Great Great tips, the understanding of KPI’s in NSO is the beginning of a successful SEO campaign.I emphasis alot on Keyword/conversion ratio which is always the backbone to deliver a true representation of one’s effort. Thanks for the post.

Reply