Spider trap for pages created on the fly
Q: Please explain again the “spider trap” as it pertains to pages created on the fly. How big of a disadvantage does this present? If redesigning the site isnt an option, what can be done to lessen this disadvantage?
A: A search engine spider can get caught in a “spider trap” if it keeps bumping into links to pages that are the same content but with different URLs that are varied dynamically (e.g. where the URLs contain a nonessential variable/flag or session ID in the query string). If caught in a spider trap, the spider would download the same pages over and over again, overloading the site’s web server and cluttering up Google’s index with a slew of duplicates. To circumvent such potential problems, Googlebot often chooses to skip over various dynamic pages. This can have very deleterious results, such as the majority of a dynamic site getting skipped over by Googlebot. If revamping the site isn’t an option, you might want to consider an outsourced “dynamic feed” service such as GravityStream.
Comments
Chapter 7:
Content Marketing
From the fundamentals of link building to the nuances of natural linking patterns, virality, and authority.
Related Posts
Thursday Three: Embrace Journaling, Tackle Tardiness, and Explore Our Energetic Echo
Here’s what I found inspirational, challenging, or just downright hilarious this week. What caught your eye? And, remember to check out this week’s great podcast episodes: Scaling a SaaS Company with Jason Morehouse “A crucial factor to business success is to find and take the personal path that works best for you.” — Jason Morehouse […]
Read More
Thursday Three: Harrison’s harmony, conquering a blank canvas, & gut health hacks
Here’s what I found inspirational, challenging, or just downright hilarious this week. What caught your eye? And, remember to check out this week’s great podcast episodes: Be a Sales Game Changer with Fred Diamond “True elite sales professionals develop a dedicated mindset, proactive client interaction, and continuous self-preparation. They understand their client’s needs and enable […]
Read More
Thursday Three: Rebirth of sleeper trains, 4,000 weeks is a long/short time, and golden age for medicine
Here’s what I found inspirational, challenging, or just downright hilarious this week. What caught your eye? And, remember to check out this week’s great podcast episodes: A Story Worth Retelling with Luke Storey “Aligned values are the cornerstone of successful partnerships, whether in business or life, as they shape our moral code, define our priorities, […]
Read More
