Comments on: Iraqfilter

Iraqfilter

condour75 — Mon, 27 Oct 2003 14:30:44 -0800

Iraqfilter. "Sometime between April 2003 and October 2003, someone at the White House added virtually all of the directories with 'Iraq' in them to its robots.txt file, meaning that search engines would no longer list those pages in results or archive them." The robots.txt file is here. And here's the Slashdot discussion. I guess it's hard to restore integrity to the Presidency when people can compare your statements over time.

By: bz

bz — Mon, 27 Oct 2003 14:41:52 -0800

Yikes. That looks bad regardless of the reasoning.

By: specialk420

specialk420 — Mon, 27 Oct 2003 14:43:32 -0800

one would imagine /cakewalk got deleted sometime back....

By: twiggy

twiggy — Mon, 27 Oct 2003 14:46:01 -0800

One wonders if anyone out there is saving a copy of this robots.txt file for future comparison.. hmmm... Seriously though, this disturbs the shit out of me.. and what disturbs me more is the fact that the first coworker I showed this to said "well, I dunno.. it seems some of that stuff is understandable.." Attempting to deny the public the right to cache this stuff both 1) for history, and 2) for before/after comparison is at the very least "a bit shady", and more likely "quite shady"... I cannot think of a single valid reason to block search engines and other "bots" from caching historical copies of public documents on the whitehouse's website regardless of subject matter. Throw in that everything pertains to "iraq", and that's just plain scary... I'm no conspiracy theorist, but it doesn't take a radical nutball to notice they've already gone back and edited the content of bush speeches, etc...

By: the fire you left me

the fire you left me — Mon, 27 Oct 2003 14:54:34 -0800

Interesting. But wait, is Google keeping people from a history of preference? http://www.google.com/robots.txt

By: monju_bosatsu

monju_bosatsu — Mon, 27 Oct 2003 14:55:01 -0800

I think somebody screwed up the perl script for generating the robots.txt. Look at the directories; most don't exist at all. Disallow: /firstlady/photoessay/bookfestival/iraq Disallow: /firstlady/photoessay/welcometowh/iraq Disallow: /firstlady/recipes/iraq Disallow: /history/africanamerican/iraq Disallow: /history/photoessays/easter/one/iraq Disallow: /history/valentines/iraqDisallow: etc.

By: specialk420

specialk420 — Mon, 27 Oct 2003 14:58:12 -0800

speaking of iraq: the good. the bad and ugly.

By: the fire you left me

the fire you left me — Mon, 27 Oct 2003 14:59:51 -0800

Then again, there is a history of revisionism within this White House.

By: internal

internal — Mon, 27 Oct 2003 15:03:47 -0800

looks like they disallowed almost the whole site to my uneducated eye...

By: rusty

rusty — Mon, 27 Oct 2003 15:12:19 -0800

Let's all repeat the Slashdot discussion! First you say "This is scary! I hope someone is archiving this!" Then I say "Look, man, /firstlady/photos/2003/01/iraq? /president/holiday/decorations/iraq? They never existed. Some server monkey screwed up a perl -e." Then you ignore that because it's more fun to make Ministry of Truth jokes.

By: whatnotever

whatnotever — Mon, 27 Oct 2003 15:13:18 -0800

It's just "/text" and "/iraq" added to every directory. *Looks* like a goof to me. They have valid reasons to want to block indexing of "/text" and "/iraq" (as root-level directories). /text simply mirrors the content of the rest of the site, so there's no need to index it. (Avoiding indexing redundant content is a good thing.) /iraq simply redirects to /infocus/iraq, which is also linked from elsewhere. If they hadn't goofed, /infocus/iraq would be indexed normally, with no need for any crawlers to go in /iraq. There is the question of how "/afac/index.htm" got into the blunder. It doesn't seem to exist (.html does), but maybe there's some big conspiracy to keep people from finding out about the money we're giving to Afghan children! Now, you can argue all you want over whether or not the "goof" is coverup to block indexing of /infocus/iraq (I dare you to find any other directories ending in "/iraq") or not, but it looks like a dumb blunder. So just laugh at the dumb-dumbs or something.

By: eddydamascene

eddydamascene — Mon, 27 Oct 2003 15:13:47 -0800

Disallow: /firstlady/recipes/iraq "We know in what areas they eat Lis-san El Qua-Thia. They're in the area around Tikrit and Baghdad and east, west, south and north somewhat."

By: rusty

rusty — Mon, 27 Oct 2003 15:20:51 -0800

whanotever: How can you say that! After the well-known and respected journalist Dan Gillmor has urged us all to download those restricted nonexistent directories every day? "In the blogosphere, my readers can fact-check my ass, cause God knows I'm not going to do it myself."

By: tuxster

tuxster — Mon, 27 Oct 2003 15:48:11 -0800

Now, you can argue all you want over whether or not the "goof" is coverup to block indexing of /infocus/iraq (I dare you to find any other directories ending in "/iraq") or not, but it looks like a dumb blunder. So just laugh at the dumb-dumbs or something. Sure, but for someone to commit a blunder in robots.txt file with regards to the word "iraq", they must have been trying to do something involving the word "iraq" and the robots.txt file to begin with. So the question is, what were they really trying to do when they actually messed up?

By: kayjay

kayjay — Mon, 27 Oct 2003 16:13:16 -0800

speaking of iraq: the good. the bad and ugly. posted by specialk420 at 4:58 PM CST on October 27 specialk420- What is that? The same person with two blogs, or one original and a faker? I read the first one on a regular basis, but I'd never seen the second until today.

By: ROU_Xenophobe

ROU_Xenophobe — Mon, 27 Oct 2003 16:15:11 -0800

tuxster: The rest of the post you quoted mentioned reasons why you wouldn't want /text/ and /iraq/ to be spidered. This is what happens when you let Dubya exercise his MaD 1eet sKillZ at perl, I guess?

By: specialk420

specialk420 — Mon, 27 Oct 2003 16:25:34 -0800

i think the links to the israeli backed: memri.org and this blog which was linked on the suspicious copy until just a few hours ago might be a few clues as to who is behind the baghdad burning rip-off.

By: adamrice

adamrice — Mon, 27 Oct 2003 16:36:46 -0800

I'm always happy to blame GW for, well, just about anything. This may be evidence of the generally secretive approach at GW's WH, but when directories like http://www.whitehouse.gov/president/holiday/whtree/text/ are banned (that's legit, btw), you know they're just banning everything. Actually, that'd be an interesting project--write a spider that doesn't respect robots.txt, and find out which, if any, directories have been left open.

By: adamrice

adamrice — Mon, 27 Oct 2003 17:06:38 -0800

Damn, specialk420, those links are fascinating!

By: specialk420

specialk420 — Mon, 27 Oct 2003 17:11:16 -0800

total derail. the rest of the discussion is over at atrios. the bush adminstrations efforts to control information in this case - and dilute information in the baghdad burning case (we suspect) is ... well... i think people can make up their own minds.

By: specialk420

specialk420 — Mon, 27 Oct 2003 17:16:05 -0800

(further derail) this cat seems to be on the hunt.

By: mecran01

mecran01 — Mon, 27 Oct 2003 18:07:42 -0800

webreaper will ignore spiders, I believe. If not, the htweb might. One of them does. And Wget might be configurable in that regard.

By: palancik

palancik — Mon, 27 Oct 2003 18:31:00 -0800

Something interesting about this strategy, if it deserves that label. Go to Google, and search whitehouse.gov + Iraq, and you get two White House entries plus all the sites they would probably prefer you not see: Stumbleupon,, Whitehouse.org, the Evil Eradication Office, and a lot of weblogs (e.g., such as this one. So why is this such a good idea for the White House? Don't they know that someone will notice, and publicize it?

By: whatnotever

whatnotever — Mon, 27 Oct 2003 18:37:59 -0800

adamrice, I wasn't aware that "/text" showed up everywhere. But that just makes even more sense, for the most part. Throw "/text" on the end of any directory and you get the plain-text version (for accessibility, generally). That's a fine way to do things. But there's no need for anyone to index it, because it has the exact same content as the non-/text page. Take any "/text" url, remove "/text", and you should get the same page, but with pretty pictures. People can feel free to check all of them every day for themselves, but it seems a bit wasteful. So blocking [everything]/text makes perfect sense. How [everything]/iraq got in there...? I'll still chalk it up to stupidity. Really, does anyone remember "insert something meaningful here"?

By: UKnowForKids

UKnowForKids — Mon, 27 Oct 2003 19:01:46 -0800

palancik: Your reported Google results aren't due to the robots.txt file - you're only getting two whitehouse.gov results because Google hides all the others. Searching whitehouse.gov for "iraq" gives a ton of hits (at least for now).

By: inpHilltr8r

inpHilltr8r — Mon, 27 Oct 2003 19:26:33 -0800

Let's all repeat the Slashdot discussion! Don't you have a site to maintain somewhere?

By: soyjoy

soyjoy — Mon, 27 Oct 2003 21:09:10 -0800

Yeah, this same discussion has been had at Atrios and other places all over the Net already. BUT it was worth posting just in order to call the link "Iraqfilter." How many chances are we gonna have to make that pun? Kudos, condour75.

By: rusty

rusty — Mon, 27 Oct 2003 23:38:47 -0800

inpHilltr8r: I read MetaFilter to forget. :-)

By: bwerdmuller

bwerdmuller — Tue, 28 Oct 2003 02:35:10 -0800

Wget's been mentioned above, but it might be a worthwhile endeavour to configure something like it to download exactly those pages. Might be interesting to read around, say, the next election.

By: Mitheral

Mitheral — Tue, 28 Oct 2003 07:48:32 -0800

HTTrack can be configured to ignore robots.txt files and it's got a nice clicky interface for those on windows who can't handle a command line. You could even tell it to only get things mentioned in robots.txt (or any text file of links for that matter)