How Google shapes everything on the web
January 8, 2024 11:50 PM   Subscribe

How Google perfected the web

Is this document interactive?
Yes! It shows animations as you scroll up and down, but appears to work in Firefox as well as Chrome.

What is it about?
How Google search optimizations have flattened the diversity of the web into a bland homogenity of bite-sized information chunks, classified by question.

Is it worth reading?
Very much so! Even if you know much of this already, it presents the information very well in a graphical and illustrated format.
Why you can trust TheophileEscargot
With decades of experience surfing the web, many thousands of words of online bloviating, and a degree of personal cynicism, TheophileEscargot is supremely well qualified to complain about stuff online, and link to other people complaining about stuff online.
posted by TheophileEscargot (48 comments total) 33 users marked this as a favorite
 
We’re publishing blog posts every day on our site, and we now have over 100 high-quality lizard articles.

Now I want this to be a real blog!
posted by chavenet at 1:01 AM on January 9 [3 favorites]


I'd really like a plain text version of this. All the cruft makes it unreadable. (Or is that the point?)
posted by Cardinal Fang at 1:11 AM on January 9 [5 favorites]


This article is a nice summary of the reasons for Google's enshittification (a topic we've had on this site a few times lately).

Slowly, all that's left is a hollow network of useless sites. From a company that became hugely popular for clearing spam from email inboxes, it's gotten really "good" at promoting spam content on the www.

Google search, at this point, would be more useful if they flipped a switch to down rank all sites that are optimized for Google.*

* Edit: seriously I think this would be a legitimate Google search alternative. Someone needs to reverse engineer their ranking algorithm to promote non-googlified websites.
posted by UN at 2:06 AM on January 9 [18 favorites]


You can't overlook the spammers seeking to drive traffic to their businesses. Alphabet/Google also complicit in selling adverts when the PageRank paper says "ofc these measures of what information is more and less authoritative on a subject will be ruined if you're selling advertising." So that's probably both malice and incompetence when scoring the malice-vs-incompetence scale at good old 'do no evil' Google.

Reputation still matters for speaking authoritatively; to some extent a search index will solidify and amplify existing rich and authoritative voices.
posted by k3ninho at 3:45 AM on January 9


This site crashed my Macbook. Like hold down the restart button crash. Which I guess is emblematic of the enshittification of the web.
posted by hydropsyche at 4:21 AM on January 9 [4 favorites]


Google search, at this point, would be more useful if they flipped a switch to down rank all sites that are optimized for Google.

Way back in the day (possibly when many of you were of juice-box-and-Barny age) I was dipping my toe into web design. And, it was kind of fun, really. But, when SEO reared its nasty head, I said “fuck this shit” and walked far, far away from web design. Never regretted the decision.
posted by Thorzdad at 4:38 AM on January 9 [15 favorites]


to some extent a search index will solidify and amplify existing rich and authoritative voices.

In real life, Bannon's 'Flood the zone with shit' ensures that nobody can tell which voices are authoritative and which aren't; everything gets lost in the noise. Meanwhile, if you're in the UK, typing anything newsworthy into Google will now very often cause the 'In the News' bar to be populated with links from GB News, which is basically just a far-right video blog whose owner has enough money to buy it onto the airwaves as a 'news channel' without any need for significant viewer figures or advertising revenue, and wait a few years until it goes 'mainstream' by dint of the public simply passively accepting its existence. This is how garbage gains positioning alongside legitimate voices, and hence equivalent credence. I would imagine the US is similar regarding Newsmax, OANN and other toilets.
posted by Cardinal Fang at 5:39 AM on January 9 [15 favorites]


While Google deserves a lot of blame for pandering to advertisers over users, remember that PageRank always was based on a stand-in for authority, namely popularity. As QAnon and Trump have shown in the real world, popularity is a poor stand-in, and in a medium like the web, easily gamed.

And because capitalism makes it profitable to create bogus science journals, even Google Scholar i was not immune, though not to the same degree.

It makes me wonder if Ted Nelson's Xanadu, which was citation-based might have avoided some of these problems. Though I suppose fake citations are not too different than link farms, so maybe not.
posted by CheeseDigestsAll at 6:05 AM on January 9


The spammy side can always issue enough noise to fill the channels where you'd test and confirm sources are good.

This is backwards -- the 'boy crying wolf' can only do it a few times before his credibility is near-zero, so we should be able to mute spammers who waste time with false claims.

Competing unverifiable claims need clustering around worldview so that proponents of FSM or greed-is-good can have things that matter to them, without fighting over the supremacy of one particular worldview in our shared consensual hallucination of the world in front of us.
posted by k3ninho at 6:52 AM on January 9


> Someone needs to reverse engineer their ranking algorithm to promote non-googlified websites.

25 years of unsleeping effort by highly-motivated people has not succeeded in reverse-engineering their algorithms, which likely also change frequently.

> ... the 'boy crying wolf' can only do it a few times before his credibility is near-zero, so we should be able to mute spammers who waste time with false claims.

The difference between the boy and the spammers is, you know who the boy is so that you can ignore him. As long as you receive SMTP email, you will get spam, because SMTP didn't specify any means of sender authentication.
posted by Aardvark Cheeselog at 7:00 AM on January 9


Jon Postel's reach for robustness with "be conservative in what you send, be liberal in what you accept" shouldn't apply here. The SMTP horse has bolted, so SPF / DKIM have closed this open email system to trusted mail exchangers (without really reducing the spam level).
posted by k3ninho at 7:21 AM on January 9 [2 favorites]


While Google deserves a lot of blame for pandering to advertisers over users, remember that PageRank always was based on a stand-in for authority, namely popularity.

And, with the decline in independent sites, there are fewer links to power PageRank. It was created under the assumption of a thriving World Wide Web; when sites are mostly trying to be Reddit, Kotaku or Stack Overflow, you're going to get a much lesser quality of link, and PageRank falls apart.
posted by JHarris at 8:36 AM on January 9 [2 favorites]


search engines are inherently bad and eventually everyone who doesn't agree with me on this will agree with me on this.

but wait, you ask, without search engines how do people find websites? to you i answer: word of mouth.

but wait, you ask, what? please elaborate? to you i answer: literally word of mouth.
posted by bombastic lowercase pronouncements at 8:41 AM on January 9 [1 favorite]


If you know what chameleon feet are supposed to look like, the AI-generated chameleon feet in this article are quite funny.
posted by peeedro at 8:44 AM on January 9


The increase of spam sites is fully a result of the search monopoly status quo. Because Google promotes shitty websites, it leaves little incentive for people and organizations to make good ones (and drives them away from the web towards other platforms where they can be found).

It's not the spam sites that flood Google's algorithm, it's Google's algorithm that floods the Internet with spam sites.

Google (and Apple) accelerated the trend by using their smartphone duopoly status to drive users towards apps where they charge massive rent.

It's not an accident that the web suffers and spammers rule the virtual world.
posted by UN at 9:28 AM on January 9 [2 favorites]


it's not about the search monopoly. it's about search, full stop. this isn't a google problem, this isn't a monopoly problem, it's a problem inherent to Internet search and can only be fixed by abandoning the search paradigm for better, more robust, less intrusive methods of Internet content discovery.
posted by bombastic lowercase pronouncements at 9:31 AM on January 9


That's like telling a lobster that boiling water isn't the problem, water is the problem.
posted by UN at 9:34 AM on January 9 [3 favorites]


Surprisingly, the article is quite readable on my wheezy lynx browser. Far better than I expected. But the source is an impenetrable morass...
posted by jim in austin at 9:37 AM on January 9 [1 favorite]


> That's like telling a lobster that boiling water isn't the problem, water is the problem.

it's not though. if anything, it's like telling a lobster that boiling water isn't the problem, prolonged exposure to temperatures above 100 c is the problem.

the pattern of web browsing that search engines encourage — go to the search engine, search for keywords and phrases, click through links, see if they're good, if not go back to the search engine and click additional links / make additional searches until you find one that satisfies you — tends to surface sites that are familiar in design and are marginally useful to a wide range of people. when search becomes the primary way to discover web content, sites that break from the "be familiar and marginally useful to a wide range of people" template become increasingly hard to find but also> hard to make — if you're making content that you want people to see, or are trying to make money off of your content, and if your content doesn't adhere to the "be marginally useful to a wide range of people" you will become quite discouraged indeed to discover that no one is seeing the stuff you're making, and then you will give up.

this is, again, nothing that can be fixed by undoing the monopoly or by tweaking content surfacing algorithms. it's a problem that occurs when people find things through search engines.

again you ask: oh come on, we can't have a web without search, how do you think people will even find stuff at all without search engines? and i once again give you my 100% serious not a joke answer: word of mouth.

getting from an Internet mostly blighted by search to a generally worthwhile Internet seemed impossible until relatively recently. fortunately, we have llms now, and the best use case for llms, the thing that they are perfect at, is producing specifically the type of gibberish that poisons search engines / undermines the fiction that search engines are good at surfacing good content, and as the search engines die people are increasingly realizing that word of mouth really is the only reliable method for finding stuff on the Internet.
posted by bombastic lowercase pronouncements at 9:56 AM on January 9 [4 favorites]


> That's like telling a lobster that boiling water isn't the problem, water is the problem.

and like, your metaphor here really gets at the point of how badly search has screwed up the web: search engines are an element within the web environment, but the google monopoly has encouraged us to see their search engine as the environment itself.

it's analogous to the fiction that google allows access to the sum total of human knowledge; people don't think that because google gives access to the sum total of human knowledge, but instead because we've learned to overlook or just ignore the much much larger body of knowledge that google hasn't digitized/indexed.
posted by bombastic lowercase pronouncements at 10:01 AM on January 9 [1 favorite]


and like, your metaphor here really gets at the point of how badly search has screwed up the web: search engines are an element within the web environment, but the google monopoly has encouraged us to see their search engine as the environment itself.

Yes, and this is exactly what I'm trying to say. I mean, I don't know that breaking up Google's search monopoly (and the Google/Apple smartphone OS duopoly) will set the lobster free back to thrive in its biodiverse habitat— but I'd like to see what happens and I'm pretty sure it'd help, not harm, the web. [I guess a spider analogy would've been more fitting? Anyway...]

Even for the "non-search" ways of finding things on the internet, I can only imagine that not having a Google input on every screen would help, not hurt that cause.
posted by UN at 10:13 AM on January 9


Google was rapidly adopted when it debuted because it worked, really well. Those of us who remember search before Google know what a groundbreaking and seemingly magical experience it was when the thing you were looking for was nearly always returned when you searched, and the really cool tricks you could do to narrow it down even further. Then they got greedy. First they killed the ability to use operators to force-include or force-exclude results, and started selectively ignoring some input, because they were convinced they knew better than you did what you really meant when you searched for things. The increasingly shitty things they do on the back end now mean that searching for a term won't even submit a search for that term - they modify the query to add or remove things, specifically to ensure that the results returned meet their goal, not your intent.

Google search used to be designed to locate matching content. Google today is designed to drive you to advertising. Google owns the ad network, they own the ad distribution system, and they own the search engine that sends you to the places they serve ads so that they can monetize your clicks endlessly.

There is no goal at all in mind except delivering you more ads. They couldn't care less about whether the ads are tied to good content or linkfarm garbage so long as you are exposed to the ads. Linkfarm garbage turns out to be easier to produce than good content, go figure.

TikTok as a solution makes me cringe. The last fucking thing I need is for every goddamn search result to be in video format.
posted by caution live frogs at 10:27 AM on January 9 [19 favorites]


We often think of "perfect" as meaning it can't be any better, but it also means "completed" (as in the past perfect tense). So yeah, search has perfected the web. It was a good run.
Anything which has ended is finished
That which is perfect is finished
posted by thecaddy at 10:28 AM on January 9 [2 favorites]


This is a good article in that it explains something I already pretty much knew in a way that makes it clearer while also making me feel smart for already knowing it.
posted by jacquilynne at 10:42 AM on January 9 [5 favorites]


> Yes, and this is exactly what I'm trying to say. I mean, I don't know that breaking up Google's search monopoly (and the Google/Apple smartphone OS duopoly) will set the lobster free back to thrive in its biodiverse habitat— but I'd like to see what happens and I'm pretty sure it'd help, not harm, the web.

oh, yeah, don't get me wrong: i am totally on-board with smashing the google search monopoly.

but also: search will remain a bad method of content discovery even after the monopoly is gone. smashing google is necessary for the reëstablishment of a worthwhile web, but smashing google won't fix the underlying badness of search as a web content discovery paradigm.
posted by bombastic lowercase pronouncements at 10:51 AM on January 9 [1 favorite]


I don't have patience to wade through that website. Google is a major problem. Websites that can't just present the information without bullshit is another.
posted by BlueHorse at 10:56 AM on January 9 [1 favorite]


What we really need is the return of Dave's Cool Site of the Day.
posted by jacquilynne at 10:58 AM on January 9 [2 favorites]


see i used to say stuff like that when delivering earlier versions of this weird stand — i'd make reference to webrings and hand-maintained directories and such — but this isn't a nostalgia thing and really doesn't need to be discussed in backwards-facing terms. most of the web content that's been generally useful or generally interesting to me i've found either through word of mouth or things that closely approximate word of mouth (i.e. posts on small, long-established web fora). i think people only think search is useful because most of us don't realize that we actually find most of our favorite/most useful content through word of mouth rather than via search engines.
posted by bombastic lowercase pronouncements at 11:09 AM on January 9


i.e. posts on small, long-established web fora). i think people only think search is useful because most of us don't realize that we actually find most of our favorite/most useful content through word of mouth rather than via search engines.


Favorite, cool or useful? I most assuredly find my most useful content via google - it's not even close.

But my favorite content (the stuff I seek out to read month after month) came from word of mouth.
posted by The_Vegetables at 11:17 AM on January 9 [1 favorite]


see, for me i thought it was word of mouth for the interesting stuff and google for the useful stuff, but once i started paying attention to how i use the web i discovered that if i'm using google to find something useful i'm almost always using it to search for content within sites that i already discovered but that don't themselves have a robust search function.

basically, if i'm doing something like writing code or baking or doing simple home repairs, using a search engine to discover content on sites that i don't already know is a last resort, and not a particularly reliable one.
posted by bombastic lowercase pronouncements at 11:22 AM on January 9 [1 favorite]


What we really need is the return of Dave's Cool Site of the Day.

Or maybe a website where users curate posts to introduce other users to things that they might be unfamiliar with. Something like Plastic.com. I wonder if such a thing exists?
posted by Ashwagandha at 11:23 AM on January 9 [7 favorites]


A bit tangential, but it's going to be (morbidly) interesting to see how quickly the early enthusiasm around the promise of AI is going to turn into universal loathing.

"ChatGPT, give me a recipe for sheet pan salmon and broccolini"

"Sure thing. But first, maybe you should consider ordering it from Doordash. The advantages or ordering from Doordash are:
* it saves you time and energy
* you can be confident your delicious meal is going to be professionally prepared by highly-trained chefs
* all of the top-quality restaurants available via Doordash have excellent food safety standard ratings
* there's going to be no mess to clean up

And did you know that many home fires are started by cooking? Ordering sheet pan salmon with broccolini via Doordash can reduce the risk of such disasters and keep your family safe.

Here are ten restaurants that can make your meal, all within 45 minutes of delivery if you order now. The first is..."
posted by treepour at 11:52 AM on January 9 [2 favorites]


Some attempts at good old fashioned directories are available right now: Curlie and ooh.directory. I haven’t found them to be greatly useful, but I like that they exist.
posted by Eikonaut at 11:55 AM on January 9 [1 favorite]


The problem with the word of mouth approach is that it severely limits new sites. Search is dead, you have a new website that walks people though the fascinating realm of toilet bowl design. How do you let people who are interested in this know? I remember how forums used to be relentlessly spammed, these days it's reddit (the php replacement that won't be your fault if it gets hacked). But without search, we're relegated to going the person who published the site we're seeking is a capable viral marketer. Maybe they get a thousand people from the mechanical turk on Amazon to drop links to their useful and amazing site where people might find them. Which low quality spammers can also do. And instead of an algorithm playing good, we're back to people who can pretend to be authoritative policing which sites get traffic and which do not.

Search in some form is necessary. Having multiple search engines with differing criteria for what gets promoted to the top would at least give us more than one style of web page.
posted by Hactar at 12:01 PM on January 9 [4 favorites]


"The problem with the word of mouth approach is that it severely limits new sites."

Not really. It just slows down the rate of spreading in the populace. You might have to wait for several "contagion" incidents from one "word of mouth" site/group to the next...to the next... to hear of it. So, a "slower" web.

Of course, the original web site has to *survive* till you get to hear of it since the propagation is slowed down. And if the site relies on being "heard" to survive then you get competing rates of change.
posted by aleph at 12:14 PM on January 9 [2 favorites]


Curated indexes of the web exist. They are outdated as soon as they are updated, but they are there. A search engine could search into those. YaCy (yacy.net)is a search engine that you can run locally and specify how and from whence it builds its index. If you have some control over the construction of the index, you can avoid some of the bullshit.

If your view includes a button for users to demote or remove entries from search results, you can work that like a spam filter to exclude "similar" (in same statistical fashion that emails in your Spam folder are similar to one another) results to the ones the user has already chosen to demote or remove. More junk gone, taking some "good stuff" with it, no doubt- but a periodic review of the user's bullshit pile can potentially solve that.

Search doesn't have to be oppressive. Things provided "as a service" are subject to rent-seeking. Services provided by rent-seekers serve them. Services provided by a trustworthy party are as trustworthy as the provider is competent.
posted by Rev. Irreverent Revenant at 12:18 PM on January 9 [6 favorites]


Something like Plastic.com. I wonder if such a thing exists?

Would it be okay to like that?
posted by jacquilynne at 12:23 PM on January 9 [6 favorites]


basically, if i'm doing something like writing code or baking or doing simple home repairs, using a search engine to discover content on sites that i don't already know is a last resort, and not a particularly reliable one.

What?
Those are the opposite for me. I don't know of any specific (non-paying) website that is worthwhile for code samples, home repairs, or baking.

For home or car repairs, some random dude on youtube with my same model of cars via a google search.

For coding: Random websites. StackOverflow is ok, but was always too full of "read the documentation!" dudes for me to regularly read it. Nobody reads the documentation because it's terrible.

For baking: completely random.

What websites do you regularly use for these things? I'm curious about what it is...
posted by The_Vegetables at 12:34 PM on January 9


Every time the topic of search comes up here, I end up reading some comment or other about how we have to do away with search and make the whole organically findable through association (web rings, indexes, and today word or mouth) or some such nonsense, and it makes me shake my head. Look, if you don't need to use the Internet for anything business-related, that's fine, but that's not the world I live in.

Research is part of my job. I never know from one day to the next what I'm going to need to be researching. Today, for example, I needed to find out who was lobbying a particular government to change a particular piece of legislation so that the company could engage in a business practice that had previously been rejected, and then figure out what government officials are directly involved in representing people who have a stake in this matter. (Note this is all for an area that is not local to me.)

Earlier, I had to find experts who could explain the psychological impacts on people who are forced to interact with a certain piece of technology in a certain way, and also find other people who create or use that technology.

I dare anybody to tell me how to use the web without search and through word of mouth. (And that's taking into account that word of mouth is also part of my research strategy.)

If people want to treat the Internet as a playtoy for their amusement, have at it and devise all of the blue sky ways that it can work without search. For the rest of us, who need the Web to work and search to work, we'll be here trying to do our best coping with the tools that we have--tools whose usefulness had been highly degraded due to poor behaviour by Google--and figure out workarounds to enable us to keep doing our jobs. Just don't tell me how nobody needs search. That's the kind of word of mouth that is zero benefit or interest to me.
posted by sardonyx at 12:44 PM on January 9 [14 favorites]


and i once again give you my 100% serious not a joke answer: word of mouth.

And my response to that is simple: search is also "word of mouth". Because all "word of mouth" winds up being, in the end, is having some other entity giving you recommendations - whether it's a friend telling you about a cool new website, a curated list of resources, or a search algorithm that uses your query to pull in responses to that query. And as it turns out, each of those modalities has its own strengths and issues, which is why we need all of them.
posted by NoxAeternum at 1:45 PM on January 9 [2 favorites]


First they killed the ability to use operators to force-include or force-exclude results...

Oh, gods, I remember the first time I ran afoul of this nasty development. Google was dead to me on that day.
posted by Thorzdad at 3:10 PM on January 9 [5 favorites]


> Something like Plastic.com. I wonder if such a thing exists?
Would it be okay to like that?


I'm glad this little piece of MeFi lore is remembered!
posted by JHarris at 3:53 PM on January 9 [3 favorites]


I'm glad this little piece of MeFi lore is remembered!

I realised the other day that it is my 20th year of having a membership on Metafilter (tho I had been lurking for a couple years before that). So... that's to say that I come by that lore honestly?
posted by Ashwagandha at 5:15 PM on January 9 [1 favorite]


Google is shit now because, among other reasons, they made a decision to be an ad business, not search business. It doesn't matter to them if you can't find what you're looking for because you're still clicking and generating ad money. It doesn't help that all the old organic linkages that Google used to trawl to organize and decide which sites were important all disappeared, lots of interaction moved to large platforms like Facebook, Reddit, etc. I also hate the condescension applied when they substitute your search with similar but not synonymous queries - maybe it helps the lowest common denominator, but it makes it almost impossible to find very specific things. Google no longer searches, it herds.

I think walled gardens like MeFi will see some more traction as the public internet is flooded with AI "content" and people seek out things from, you know, actual people.
posted by ndr at 3:57 AM on January 10 [4 favorites]


I came across it honestly too, Ashwagandha! To fill everyone else in--

One of the first really popular websites was Suck (suck.com), which was quite a sensation for a few years. It even generated at least one book.

A spinoff of Suck was Plastic (plastic.com), a community discussion and webfind site not too dissimilar to a certain blue thing nearby. To fix the timeline-- MetaFilter: July 1999; Slashdot: October 1997; Plastic: January 2001

There was more of Slashdot in Plastic's makeup than Metafilter; Wikipedia tells me it even used Slashdot's content moderation system, Slash. It had comment threads, and more images in its design, and voting. It was one of the three sites I generally read at the time, along with Slashdot and Metafilter (I didn't get my MeFi account until 2005 though).

Metafilter used to have, for a short while, a few measly years, a whimsical subtitle beneath the MetaFilter logo in the upper-left, that was chosen randomly from a small collection of possibilities. One of those subtitles was: "The Plastic.com it's okay to like"

Now, it's "The Plastic.com that still exists, and is still even remembered." I miss Plastic, even the later days when owner and main moderator Carl Steadman had checked out from the site's running. I do remember hearing, shortly before the end, that he had returned from doing other things online to try to right the ship, but I also seem to recall it wasn't long after that that it sank. Plastic disappeared in 2011.

Steadman's still out there on the internet. No hard feelings Carl. Thanks for giving us a little branch to perch on for a while, even if it never panned out financially.
posted by JHarris at 5:31 AM on January 10 [2 favorites]




once i started paying attention to how i use the web i discovered that if i'm using google to find something useful i'm almost always using it to search for content within sites that i already discovered but that don't themselves have a robust search function.
I'm not trying to catch your bombastic lowercase pronouncement in a gotcha, and I've seen enough of these threads and your comments on them to not only believe how genuine you are, but also start to think you may have a point.

That said, I'm wondering if you can provide an explanation of how search within a site is not the same sort of killer that Web Search is. Maybe I'm misunderstanding here, but wouldn't there be some of the same issues with promoting certain content and not others?

I guess what I'm asking is if you see a site having a search function vs. a whole-Web search as a difference in kind rather than a difference in degree, and if so, why.
posted by cardioid at 6:37 AM on January 10 [1 favorite]


A bit off topic. My friend's daughter works for Google. Her rent is $3,800 a month for a one bedroom. I am sure she is making well into 6 figures or she ought to be with that rent.
posted by DJZouke at 12:59 PM on January 10 [1 favorite]


« Older Sort of Soundies: A Musical History by Michael...   |   DON'T DATE ROBOTS Newer »


This thread has been archived and is closed to new comments