When will major search engines start indexing RSS feeds properly?

I find it a bit unbelievable that the major search engines — Google, Yahoo!, MSN Search, and Ask Jeeves — still don’t offer RSS feed searching combined with RSS search results feeds as part of their Web search. Specialized RSS feed search engines like Feedster, PubSub and Technorati have risen to the occasion, filling the void left by the major engines’ inaction. Bloglines, the AskJeeves-owned company, has announced a blog/RSS search engine service that’ll compete with Feedster, PubSub, and Technorati, but still that’s a far cry from embedding RSS search right into the Web search box.

Here’s how each of the majors handles RSS feeds:

Google:

• has URLs of valid RSS feeds in its index (due to links that point to those feeds)
• doesn’t recognize the XML file format of RSS feeds (as you can read on the excerpted screenshots above)
• only rarely indexes the feed (I base that not just on the fact that nearly all RSS feeds are shown in Google results with no title or snippet as in the first screenshot above, but also because, out of 64,000 RSS feed files hosted by feeds.feedburner.com, only 19 are shown to contain the word cheese, the last 2 of which show up in the results only because cheese appears in links pointing to the feed; yet the same search on Yahoo! shows over 400. So clearly a lot of files that should have matched are missing from the Google search results.)
• only rarely caches the XML (see example) with most caches being blank (like this)
• associates words in links pointing to the page (as demonstrated with this search)
• doesn’t allow refining of your query with the operators ?Ä® filetype:rss, filetype:xml, or filetype:rdf

Yahoo:

• has URLs of valid RSS feeds in its index
• indexes the feed (Evidenced by above screenshot, which was a match for a search on text contained within the feed. Also, ResearchBuzz found this to be the case too.)
• caches the XML (see example)
• doesn’t display the “Add to My Yahoo!” link for RSS feed listings (this is a disappointing omission, as Yahoo! displays this link on listings for HTML pages that have an associated RSS feed but not for the listing of the RSS feed itself)
• associates words in links pointing to the page
• doesn’t allow refining of your query with the operators ?Ä® filetype:rss, filetype:xml, or filetype:rdf

MSN Search:

• doesn’t have URLs of valid RSS feeds in its index (Evidence of this: not a single feed out of 64,000 feeds at feeds.feedburner.com is displayed, even though there are links that point to those feeds. Note that the couple feeds that are displayed are not valid feeds but error pages outputted in HTML.)
• doesn?Ä´t recognize the XML file format of RSS feeds (file type is displayed in the search listing after Cached link when it’s a recognized non-HTML file type)
• doesn’t index the feed
• doesn’t cache the XML
• doesn’t allow refining of your query with the operators ?Ä® filetype:rss, filetype:xml, or filetype:rdf

Teoma (Ask Jeeves):

• has URLs of valid RSS feeds in its index
• indexes the feed
• (View Cached feature not supported by Teoma)
• associates words in links pointing to the page
• (filetype: operator not supported by Teoma)

As you can see from my little comparison, MSN Search is the farthest behind when it comes to RSS feed indexing. Hopefully Scoble will read this and tell the MSN Search team to get on the ball. 😉

Even though the major engines have been slow to make RSS an integral part of their indices, I predict that the engines will, within the next year or so, wake from their slumber and overtake and even acquire their specialized RSS feed search engine competitors.

What that will mean for web marketers is that search engine optimizing RSS feeds will become a science unto itself (currently it’s limited mainly to optimizing the item titles for purposes of link text on syndicating sites) and that the feeds that are not optimized will get drowned out by those that are.

Comments