The Web is 500x larger than we think?
July 26, 2000 9:36 PM Subscribe
The Web is 500x larger than we think? While plausible, it also smacks of a press release...
Really, it's a matter of how you define "the web". If a webpage offers you access to dynamically generated reports from a SQL database ... is that "on the web", or not? I'm inclined to say not, but not everyone agrees with me.
posted by dhartung at 6:49 PM on July 27, 2000
posted by dhartung at 6:49 PM on July 27, 2000
Not a lot of people know how to submit their sites to search engines, not to mention the search engine spiders will index & spider limited-link pages ("foo.com - click here to enter"), and have so many pages that aren't linked anywhere on their sites.
Also, from what I see in my referrer logs and Apache Guardian reports, they hit "roadblocks" like dead links, back off and start over again. They'll never fully index a web site (or the entire web) at that rate.
posted by CyberPal at 6:57 PM on July 27, 2000
Also, from what I see in my referrer logs and Apache Guardian reports, they hit "roadblocks" like dead links, back off and start over again. They'll never fully index a web site (or the entire web) at that rate.
posted by CyberPal at 6:57 PM on July 27, 2000
« Older Dems blast Playboy as contradictory to equal and... | Novel Idea. Newer »
This thread has been archived and is closed to new comments
posted by Bryan at 6:39 PM on July 27, 2000