The seedy underbelly of the internet.

Stagger Lee — Mon, 13 Feb 2012 11:07:00 -0800

The "visible web" is what you can find using general web search engines. It's also what you see in almost all subject directories. The "invisible web" is what you cannot find using these types of tools. It's the internet that Google doesn't show us; some of it dull, some of it private, some of it deliberately hidden.

More beneath the surface. "The darkweb"; "the deep web"; beneath "the surface web" – the metaphors alone make the internet feel suddenly more unfathomable and mysterious. Other terms circulate among those in the know: "darknet", "invisible web", "dark address space", "murky address space", "dirty address space". It's not all sunshines and lolcats down there. In the 'deep web', Freenet software allows users complete anonymity as they share viruses, criminal contacts and child pornography Then there's Tor, the proxy network meant to protect the privacy of its users. But Tor can also provide anonymity to servers in the form of location-hidden services, which are Tor clients or relays running specially configured server software. Rather than revealing the server's IP address (and therefore its network location), hidden services are accessed through Tor-specific .onion pseudo top-level domain (TLD), or pseudomain. The Tor network understands this TLD and routes data anonymously both to and from the hidden service. Due to this lack of reliance on a public address, hidden services may be hosted behind firewalls or network address translators (NAT). A Tor client is necessary in order to access a hidden service. Put simply, through a Tor proxy users can access a growing network of hidden web space and services. The Deep Web, or at least a small part of it.

Mining the Deep Web

badstone — Tue, 02 Mar 2004 16:56:00 -0800

Mining the Deep Web. Google indexes 4 billion pages, but there are hundreds of billions of documents out there in the Deep Web that are effectively unreachable by search engines because they are locked in databases or are unsearchable media. It looks like Yahoo is going to start giving us a peek by providing unified access to a wide variety of sites that are ordinarily only searchable by their own custom search engines.

MetaFilter posts tagged with InvisibleWeb

The seedy underbelly of the internet.

Mining the Deep Web