| Top Web Hosting Provider of The Month: |
 |
| Visit Bluehost.com | Read Bluehost Review |
|
|
| >> Web Hosting Geeks // Web Hosting Articles // SEO - Search Engine Optimization |
|
|
 |
Playing in Googlebots Sandbox with Slurp, Teoma, & MSNbot - Spiders Display Differing Personalities
There has been endless webmaster speculation and worry about the so-called "Google Sandbox" - the indexing time delay for new domain names - rumored to last for at least 45 days from the date of first "discovery" by Googlebot. This recognized listing delay came to be called the "Google Sandbox effect." Ruminations on the algorithmic elements of this sandbox time delay have ranged widely since the indexing delay was first noticed in spring of 2004. Some believe it to be an issue of one single element of good search engine optimization such as linking campaigns. Link building has been the focus of most discussion, but others have focused on the possibility of size of a new site or internal linking structure or just specific time delays as most relevant algorithmic elements. Rather than contribute to this speculation and further muddy the Sandbox, we'll be looking at a case study of a site on a new domain name, established May 11, 2005 and the specific site structure, submissions activity, external and internal linking. We'll see how this plays out in search engine spider activity vs. indexing dates at the top four search engines. Ready? We'll give dates and crawler action in daily lists and see how this all plays out on this single new site over time. * May 11, 2005 Basic text on large site posted on newly purchased domain name and going live by days end. Search friendly structure implemented with text linking making full discovery of all content possible by robots. Home page updated with 10 new text content pages added daily. Submitted site at Google's "Add URL" submission page. * May 12 - 14 - No visits by Slurp, MSNbot, Teoma or Google. (Slurp is Yahoo's spider and Teoma is from Ask Jeeves) Posted link on WebSite101 to new domain at Publish101.com * May 15 - Googlebot arrives and eagerly crawls 245 pages on new domain after looking for, but not finding the robots.txt file. Oooops! Gotta add that robots.txt file! * May 16 - Googlebot returns for 5 more pages and stops. Slurp greedily gobbles 1480 pages and 1892 bad links! Those bad links were caused by our email masking meant to keep out bad bots. How ironic slurp likes these. * May 17 - Slurp finds 1409 more masking links & only 209 new content pages. MSNbot visits for the first time and asks for robots.txt 75 times during the day, but leaves when it finds that file missing! Finally get around to add robots.txt by days end & stop slurp crawling email masking links and let MSNbot know it's safe to come in! * May 23 - Teoma spider shows up for the first time and crawls 93 pages. Site gets slammed by BecomeBot, a spider that hits a page every 5 to 7 seconds and strains our resources with 2409 rapid fire requests for pages. Added BecomeBot to robots.txt exclusion list to keep 'em out. * May 24 - MSNbot has stopped showing up for a week since finding the robots.txt file missing. Slurp is showing up every few hours looking at robots.txt and leaving again without crawling anything now that it is excluded from the email masking links. BecomeBot appears to be honoring the robots.txt exclusion but asks for that file 109 times during the day. Teoma crawls 139 more pages. * May 25 - We realize that we need to re-allocate server resources and database design and this requires changes to URL's, which means all previously crawled pages are now bad links! Implement subdomains and wonder what now? Slurp shows up and finds thousands of new email masking links as the robots.txt was not moved to new directory structures. Spiders are getting errors pages upon new visits. Scampering to put out fires after wide-ranging changes to site, we miss this for a week. Spider action is spotty for 10 days until we fix robots.txt * June 4 - Teoma returns and crawls 590 pages! No others. * June 5 - Teoma returns and crawls 1902 pages! No others. * June 6 - Teoma returns and crawls 290 pages. No others. * June 7 - Teoma returns and crawls 471 pages. No others. * June 8-14 Odd spider behavior, looking at robots.txt only. * June 15 - Slurp gets thirsty, gulps 1396 pages! No others. * June 16 - Slurp still thirsty, gulps 1379 pages! No others. So we'll take a break here at the 5 weeks point and take note of the very different behavior of the top crawlers. Googlebot visits once and looks at a substantial number of pages but doesn't return for over a month. Slurp finds bad links and seems addicted to them as it stops crawling good pages until it is told to lay off the bad liquor, er that is links by getting robots.txt to slap slurp to its senses. MSNbot visits looking for that robots.txt and won't crawl any pages until told what NOT to do by the robots.txt file. Teoma just crawls like crazy, takes breaks, then comes back for more. This behavior may imitate the differing personalities of the software engineers who designed them. Teoma is tenacious and hard working. MSNbot is timid and needs instruction and some reassurance it is doing the right thing, picks up pages slowly and carefully. Slurp has addictive personality and performs erratically on a random schedule. Googlebot takes a good long look and leaves. Who knows whether it will be back and when. Now let's look at indexing by each engine. As of this writing on July 7, each engine also shows differing indexing behavior as well. Google shows no pages indexed although it crawled 250 pages nearly two months ago. Yahoo has three pages indexed in a clear aging routine that doesn't list any of the nearly 8,000 pages it has crawled to date (not all itemized above.) MSN has 187 pages indexed while crawling fewer pages than any of the others. Ask Jeeves has crawled more pages to date than any search engine, yet has not indexed a single page. Each of the engines will show the number of pages indexed if you use the query operator "site:publish101.com" without the quotes. MSN 187 pages, Ask none, Yahoo 3 pages, Google none. The daily activity not listed in the three weeks since June 16 above has not varied dramatically, with Teoma crawling a bit more than other engines, Slurp erratically up and down and MSN slowly gathering 30 to 50 pages daily. Google is absent. Linking campaign has been minimal with posts to discussion lists, a couple of articles and some blog activity. Looking back over this time it is apparent that a listing delay is actually quite sensible from the view of the search engines. Our site restructuring and bobbled robots.txt implementation seems to have abruptly stalled crawling but the indexing behavior of each engine displays distinctly differing policy by each major player. The sandbox is apparently not just Google's playground, but it is certainly tiresome after nearly two months. I think I'd like to leave for home, have some lunch and take a nap now. Back to class before we leave for the day kiddies. What did we learn today? Watch early crawler activity and be certain to implement robots.txt early and adjust often for bad bots. Oh yes, and the sandbox belongs to all search engines. Mike Banks Valentine is a search engine optimization specialist who operates http://WebSite101.com and will continue reports of case study chronicling search indexing of http://Publish101.com
MORE RESOURCES:
Google News |
|
|
|
RELATED ARTICLES
Link Reputation - How to Improve Search Engine Rankings One of the most overlooked strategies to improve search engine rankings is building your link reputation. This link popularity strategy is just as important or equal to creating keyword rich content pages. Does The Number Of Links On A Page Affect Ranking? Lots of research has focused on inbound links to a site, but little has focused on the number of links actually on a page (outbound or to other parts of a site). Many SEO gurus have recently been talking about something they call "PR Leak" which seems to be a theory that the more outbound links you have, the more your page rank on Google "leaks" away. Google vs. Yahoo -- How To Rank High On Each One Google likes incoming links, especially links from high-ranking, on-topic pages that include keywords in the link text. Google doesn't like over-optimized, high keyword densities and over use of keywords in headings, etc. Linking for Traffic not Positioning! With more and more experts and search engine enthusiastsclaiming the right way and the wrong way to handle linkswapping, link exchanging or reciprocal linking! You can tell something is important when there is more thanone name for it! GRIN! There are also two schools of thought on the reasons linkswapping. The first reason for link swapping has always been to carryfavour with Search engine rankings. Search Engine Optimization With Sitemaps I just wanted to share a little Search Engine Optimization experiment I ran to confirm the theory that Google likes content rich sitemap pages rather than just a bunch of links pointing to different pages on your site. I also wanted to look at a way of funnelling Google page rank to all the internal pages on my site as quickly as possible I have heard from a few search engine optimization companies that sitemaps are good ways of helping search engine spiders find all the pages on your site but have you every thought that using good quality sitemaps can also help your internal pages attain a very high Google page rank very quickly I was reading a Search Engine Optimization article about how Google likes pages with good quality relevant content and how they wanted to serve this quality content to their surfers. SEO and Google Indexing - Why It Requires A Complex Blend of Skills If it was easy, everybody would be doing it. Getting a company's name and products, or services, onto the first page of a genuine Google search isn't a trivial piece of work. Search Engine Marketing Hype Killing Small Businesses Think about the first thing you ever heard about "marketing a website" on the web. 99% of the time the first words anyone ever hears are "search engine marketing. The Importance of Search Engine Optimization Search Engine Optimization is a key to any successful internet marketing strategy. There are numerous definitions and interpretations as to exactly what Search Engine Optimization means. Importance of Keywords in Anchor Text or Title Text Keywords are indisputably, the single most important element of an anchor text.First of all, for those who are still learning the ropes let us define an anchor text. Marketing Articles: Getting A Better Search Engine Rank For All Of Your Pages! In one of my articles, I discussed how to market your web site link twice. It detailed out how to promote not only www. A Three Day Marketing Plan for Better Google Rankings If you're reading this article, you've probably discovered that simply building a website is not enough to ensure success with your small business. Competition on the web is fierce. Keyword Research And Overture Many of us who build websites get in the habit of using the Overture "Keyword Suggestion Tool" to do keyword research. It's convenient, and one of the few remaining such tools that is free. Link Exchange Tips, No Tricks Use text links, avoid image links. Anyhow, if you have used image links, then always make sure to put your keywords in the alt tags. Search Engine Marketing: 20 Nitty-Gritty Strategies To Compel People To Link To Your Website You don't have to be a rocket scientist to know that having quality inbound links from websites with high page ranks can help you attain high rankings in all the Search Engines.But achieving this can be more difficult to do than can be imagined. Advertise Locally Using Search Engines While search engine advertising has been a great advertising medium for businesses capable of or interested in marketing their products and services to a national or international audience, the effectiveness of this type of advertising was limited for businesses interested in advertising to a local market until very recently.For example, a realtor with a web site in Minneapolis is likely interested in advertising on search terms such as "homes for sale" and "sell my home. The Power of Topic Specific Search What are Topical Search Engines?Simply put, topical search engines are search engines focused on a specific industry, sector or topic.While many marketers are scrambling for links, any links, an area that is often overlooked is topic-specific search engines. Googles New SEO Rules Google has recently made some pretty significant changes in its ranking algorithm. The latest update, dubbed by Google forum users as "Allegra", has left some web sites in the dust and catapulted others to top positions. The Great Search Engine War, Where Content is King When search engines first appeared, they were simple affairs consisting of a relatively basic database containing small amounts of information about websites. The search engine database allowed web-surfers to search for specific words or phrases. Guide to Search Engine Optimization What is Search Engine Optimization?Search Engine Optimization or SEO for short is modification done in the web site design, coding, content and/or structure of a web site in an effort to achieve the higher ranking within search engines. Search Engine Optimization are done to attend the highest ranking in the search engine results for some targeted keywords or key phrases. Why Do You Want to Link With A Home Business And Affiliate Website? No, it's not a general question for all and sundry. Obviously, there are many who would want to link to a Home Business or Affiliate program related website and for good reason.
|
|
|
|
|
| © 2004 - 2008 "Web Hosting Geeks" | Web Hosting Reviews | Customer Reviews | RealMetrics Reviews | Hosting Articles | Directory | Partners | Contacts Over 7000 articles: web hosting, web development, domain names, ecommerce, web design, site promotion, ppc advertising, seo, site promotion and many others. Web hosting reviews, ratings and awards are not based on any incentives or commissions. Names and trademarks are the properties of their respective owners. A direct link to Web Hosting Geeks (http://webhostinggeeks.com) must be provided in order to use any of the above information. Contact us for more info. Partners: Hosts by speed, Cheap Website Hosting, Free Website Hosting, Cheap Web Hosting, Top 10 Web Hosts, Top 10 Web Hosting Deals, Best Website Hosting, Free Web Hosting, Free Web Hosting, Dedicated Server Hosting, Adult Web Hosting, Web Hosting Discussions, Dedicated Server Reviews, Best Web Hosting, Web Hosting Discounts, HostProfessor.com, rsuog, halyava, PHP Website Hosting Services, Web Hosting Reviews, Hosting Uptime, Best Web Hosting Reviews, Cheap Webhosting, Web Hosting, Flash Templates, CMS Templates, Web Hosting Reviews, Website Hosting Reviews, Web Hosting Providers, Best Web Hosting, Top Web Hosting, RSUOG Web Hosting |
