Comments on: Comments on 8055

Post number 8055

Nothing — Tue, 05 Jun 2001 03:50:15 -0800

From the googlebot FAQ: "For most sites, Googlebot should not access your site more than once every few seconds on average"

I thought it was a mistake at first, but they go on to say that you should contact them if "we are placing too high a load on your site" Do they really hit some sites that hard? If so, is it really necessary?

By: barbelith

barbelith — Tue, 05 Jun 2001 04:06:30 -0800

Yes - absolutely - I onced worked at a database driven site - I was assembling the search-engine strategy ironically enough - running Vignette and Oracle, and the site kept collapsing completely after launch. It was determined that this was in fact caused by a Google bot, and the site eventually had to stop all search engine robots spidering it. Disaster for the huge company concerned. VERY bad.

By: chrish

chrish — Tue, 05 Jun 2001 05:13:10 -0800

What upsets me more than anything, is the handful of companies which have set the user-agent of their spider to match a popular browser. ex. Mozilla/4.0 (compatible; MSIE 5.0; Win32)

By: ParisParamus

ParisParamus — Tue, 05 Jun 2001 05:22:09 -0800

Chrish: please explain. Its like the Borg. Cool.

By: greensweater

greensweater — Tue, 05 Jun 2001 05:42:24 -0800

What about filtering googlebot's IP block at the router?

By: mattw

mattw — Tue, 05 Jun 2001 06:10:10 -0800

I had a similar problem with Altavista. Decided to make a script and database intensive site of mine all funky, you know, the-url-is-the-command-line and all that nonsense. So: switched from ?key=value to a /key/value scheme. Forgot to put a robots.txt file in. Upshot: Fat Perl/database script got hit upwards of 7000 times in 11 hours by the Altavista spider, killed the entire web server, and I got booted off my hosts for being antisocial a couple of weeks later. Horrible horrible episode. Had its upside in the end though. One alarming development is that bots are starting to spider obviously database backed sites so people are going to have to learn more about how to exclude them.

By: jessamyn

jessamyn — Tue, 05 Jun 2001 06:20:57 -0800

Background reading available at robotstxt.org.

By: beefula

beefula — Tue, 05 Jun 2001 07:25:02 -0800

In answer to the original poster's question: They only hit sites every few seconds that have thousands or millions of pages, they don't mean that they're going to hit the same page every few seconds. You can exclude them, or you can just plan ahead and have enough server power to handle it. Google, and most spiders, only checks a given page every few days tops, so all you need is enough server power to handle it if some actual human decided to read every page on your site for a few days. I have a server that Google hits a lot (it indexed 137,000 pages in the last 4 days) and it's not a problem at all, Google have always had very nicely behaved spiders. Anyone who has to block them needs to look at their site setup again, cause they're probably losing a ton of search engine hits needlessly.

By: PWA_BadBoy

PWA_BadBoy — Tue, 05 Jun 2001 07:43:28 -0800

It's not just Google's spiders that cause the problem. Multiple spiders running through my former employer's frames-based website brought down the web server every night until I added some stuff to our robots.txt file and we went to a non-frames based design. Each page had like 5 frames(!) in it, thanks in large part to a pretty web designer they had there. All those frames can make for some pretty big http requests.

By: jennak

jennak — Tue, 05 Jun 2001 08:47:36 -0800

Keynote Systems pounded the presidential campaign sites from January til about March. Then they called the campaigns and offered their services: "We can improve your performance!" They ended up posting the results of their "survey" in the Spring, and then did the process over again in the Fall. I don't think any of the campaigns actually utilized their services; we were all pretty pissed about become their unwilling guinea pigs.

By: Dreama

Dreama — Tue, 05 Jun 2001 11:23:33 -0800

There are far far worse potentially system crashing spider-and-bot-type things out there.

By: dhartung

dhartung — Tue, 05 Jun 2001 12:42:07 -0800

For one real-world example, read up on why Userland-hosted Manila Sites aren't on Google (more detail). Short answer: Google's bot was forcing Frontier/Manila to generate pages every few seconds, and they couldn't handle the load very well. Critics have noted that one problem is that Frontier generates all sorts of other kinds of files than HTML (e.g. WAP) that Google doesn't need to index, and they weren't doing a good job of excluding the bot from that extraneous surfing. Winer's said that he and Google have had technical discussions and will let the bot return, but for the time being those of us with sites at e.g. EditThisPage.com are getting no, nada, zero, zilch referrals from Google. For all intents and purposes, if you search for us on Google, we do not exist.

By: arf

arf — Tue, 05 Jun 2001 13:37:11 -0800

in userland's case it wasn't google, but inktomi. however, they had blocked (more or less accidently) all bots and crawlers.

By: nedrichards

nedrichards — Tue, 05 Jun 2001 14:14:11 -0800

erm google does need to index wap sites. check out wap.google.com very cool (on your Psion Revo and IR mobile of course, not a wap phone, waste of money!). It does also translate html sites to wap with often interesting results (lets say i'll be rearranging my divs :-] )

By: mutagen

mutagen — Wed, 06 Jun 2001 10:01:51 -0800

One problem with Google and Userland sites is that the ETP sites are hosted off a single server. So if google is only hitting once every few seconds but is hitting a number of sites at once, down comes the server. Another problem with ETP sites is the sheer number of duplicating links. I once tried indexing my site with Teleport Pro and had to stop if after a few hours and nearly 50MB of duplicated pages (all different views of the same material). A carefully designed web app takes spidering into account, only allowing robots to hit static pages (and giving them enough static content that hopefully reflects the site contents to keep the spider happy) and steers it away from the dynamic database stuff. So go so far as to dynamically create custom pages for the robot but then this often leads to spamming the search engine. As the Google FAQ points out, they even offer customization on a per link basis, with a little care you can isolate parts of your site from legitimate robots that follow the standard.

By: davewiner

davewiner — Wed, 06 Jun 2001 17:42:22 -0800

Dan, it had nothing to do with the other formats we generate. The way Manila works is that all the WAPs and other XMLizations are dynamic pages generated on-demand. As arf says it wasn't googlebot that caused us all the grief, it was a combination of all the crawlers and some of them very buggy. One was requesting the same page in an infinite loop. Bugs in their software give headaches to dynamic servers.

By: yarf

yarf — Thu, 07 Jun 2001 03:10:19 -0800

Gee, crawlers have been around oh, for what, 7 or 8 years, and only now are people deciding they are a "problem?" I think the real problem is that site designers or architects simply forget that such crawlers can make up a significant minority of their site's traffic (5-15%), and in return, the site gets indexed in the search engine (which is the place most people turn to first to find information online). If your site or architecture can't handle a technology and behavior that's been around so long, perhaps it's time to look at the real problem and find a robust and stable technology that can. There are hundreds of thousands of dynamically-generated, database-backed sites online today, and you don't hear them complaining that being indexed by search engines is a "problem." Perhaps that's because they're using software that understood this is an issue and built their program with it in mind.

By: rodii

rodii — Thu, 07 Jun 2001 07:59:15 -0800

There are hundreds of thousands of dynamically-generated, database-backed sites online today, and you don't hear them complaining that being indexed by search engines is a "problem." You don't?