That Time the City of Seattle Accidentally Gave a Guy 32m Emails for $40

October 22, 2018 8:45 AM

The first large batch of requests for email metadata were sent to the largest cities of fourteen arbitrary states in a trial run of sorts. In the end of that batch, only two cities were willing to continue with the request - Houston and Seattle. Houston complied surprisingly quickly and snail mailed the metadata for 6m emails. Seattle on the other hand... 2800 words from Matt Chapman.
posted by cgc373 (24 comments total)

The passive aggression is thick.

Welcome to Seattle, buddy.
posted by lunasol at 9:07 AM on October 22, 2018 [18 favorites]

I don't like this very much. It turns out that Seattle's IT department is not very competent and not very nice, and it sounds like the resignation of the CTO was warranted -- but ultimately they *did* comply with the request reasonably quickly and at a reasonable price. This seems to me like government basically working like it should.

I'm more concerned with the 12 other cities Chapman contacted with FOIA requests who blew him off. They seem to get off scot-free in this narrative -- Chapman doesn't seem very concerned with them, and he doesn't even name them. In short, there's little upside in complying with a FOIA request (all you can do is screw it up), and apparently no downside in denying one. Don't be surprised if next time Seattle simply preemptively denies the request.
posted by crazy with stars at 9:20 AM on October 22, 2018 [15 favorites]

It turns out that Seattle's IT department is not very competent and not very nice

Based on the stories I've heard from people who've worked for the city of Seattle, this is an understatement. They are continually reorg'd and generally run the competent people out. I've worked in government, and usually it's not like this (you get a lot of mission-driven people mixed in that counteract the shitheels), but Seattle's whole IT and computing department is one of the worst I've ever heard described to me.

I'm more concerned with the 12 other cities Chapman contacted with FOIA requests who blew him off.

Washington has a strong Sunshine Law -- so open, in fact, that the kerfuffle you see him mention towards the end was all about the Legislature trying to exempt themselves from the extreme levels of public disclosure. If you don't comply, you can get your ass handed to you in court. I've known some malevolent conservative orgs who'd file public records requests on university research groups because they know it'd disrupt their work (the data they might get out of their computers was just gravy). It doesn't surprise me Seattle was so willing to send him a note. It also doesn't surprise me they were woefully incompetent.
posted by dw at 9:54 AM on October 22, 2018 [4 favorites]

Technically this request can done with a single line powershell command.

Everyone's an expert. This is like complaining about your local baseball team because they. just. don't. get. it.

"Jeez, it's so easy! If only I were in charge, there wouldn't be any problems and we'd win the World Series every year!"
posted by Cool Papa Bell at 9:54 AM on October 22, 2018 [6 favorites]

One of the things that bothers me the most about the Public Records Act in Washington is that wholesale lists of data about regular people is just available like this. I also like the goals of the PRA and I don't know how to "fix" it, or if there even is a fix. It just bothers the crap out of me that I can't do something simple like signing up for the Seattle P-Patch waiting list or entering the "One Trip A Day" commute reduction campaigns the county runs without making all of the information I give to the government open for anyone to request in bulk.

Yeah, the verbiage always says "not to be used for commercial use" but that relies on people following the rules, which marketers and scammers tend to not do. Or it relies on the rules being followed on the sending end, like when Sound Transit accidentally and improperly released all of the e-mail addresses to a pro-ST3 campaign of all of us who'd signed up for ORCA card accounts and we all promptly got spammed.

I know people who, for various reasons, simply don't communicate with local government here because they don't want their names, home addresses, mobile phone numbers, and e-mail addresses all in an easily-requested, often-requested database (because FOIA proponents seem to be enjoying ramping up the "give me every record you have...I fear you may have heard 'I want a lot of records,' but what I actually want is every record in this establishment"-type requests from places with very permissive PRA laws). They don't vote or sign up for library cards out of fear of being found.

Sorry for how long this got. I think the law's goals are great but I don't know how to fix what I see as a rough edge.
posted by fireoyster at 10:04 AM on October 22, 2018 [9 favorites]

I've responded many many times to the equivalent of FOI requests (1-4 per month for a decade). A review of 30s per email is actually low, if asking for content. We typically ask for 5 minutes per page/250 words. There are a bunch of judgement calls to make---privacy, information classification, health information, information classification by out partners, etc... If it's a document or an email trail that comes from a third party, we have to go back and consult and notify them as well.

Honestly, I've never been asked for just metadata before, I don't know if that would be different or not under the laws I work under. I suspect not---our privacy laws tend to be fairly inclusive.

The ones that truly bug me are the requesters that refuse electronic copies. For big requests we'll typically go back and verify that choice with a recipient. We have recently had to ship out literally a tonne of paper on two pallets as a result of some fishing trip requests. You know that unless they've got a few years themselves to review, the vast majority of those print out of emails and reports and draft reports aren't ever going to be looked at by human eyes again. Electronic versions are far more useful to the end user and far less costly and wasteful to generate. I am certain some requesters do this out of some misplaced revenge they're pretending to accomplish.
posted by bonehead at 10:04 AM on October 22, 2018 [8 favorites]

Everyone's an expert. This is like complaining about your local baseball team because they. just. don't. get. it.

The difference is that baseball (or any sport, for that matter) does not have actions that will consistently work 100% of the time to provide the same result. The batter cannot possibly hit the ball the exact same way 100% of the time because humans are not perfect. Also, there are different pitchers, each of whom pitch differently; even with the same pitcher, they can vary their pitch. Even if the batter does manage to hit it the same way every time, their success in making it safely to a base is highly variable dependent on his restedness, the players in the field, and so on. And if the batter becomes ridiculously consistent in always hitting it to a certain place, the other team is going to plan for that.

Given the same type of computer system, a command which does X on my computer should do X on your computer. Even given different types of computer systems, there are usually analogues (such as dir on Windows systems versus ls on *nix systems; both list directory contents).
posted by tubedogg at 10:22 AM on October 22, 2018

Yeesh. I find it unsurprising that he hasn't gotten compliance from SPD. I find it very surprising and disappointing that he got all that bs pushback from the city IT people. Seattle has some pretty neat online info resources that are extremely useful, including the parking map and the SPD "Crime Dashboard". I have an account on the data.seattle.gov site and a few years ago used it to examine the integrated datasets for automobile accidents in my area of town. It looked neat! You could set up views on a map that used heat mapping to show density of accidents and so forth.

But the data was terrible. It only covered 2010 to 2013, and looking around, I was able to find data from after 2013 available elsewhere on a city subsite that included accident reports and locations (hundred block, street address, lat/long) but which had not been integrated into the dataportal. Playing around with the heat map visualization tool also revealed that the map wasn't even being generated using the entire set of data, about 2500 records, but was instead only showing the data reflective of the data currently being displayed in-browser, about 50 rows of records. So that meant if you sorted the data by date the heatmap would change, which was surprising.

I went back and looked at my account just now and examined the maps I had made. The data has not changed since last looked at it in 2015, which means that they aren't adding anymore reportage to the datasets. I attempted to generate shareable links to the maps, and the maps are claiming that they have been set to a public view, but testing the links has not generated a view that I can provide a link to which can be accessed by a non-logged-in user on the data.seattle.gov site.

I mean, you can see the outlines of a laudable civic transparency project, but the execution is just... lacking.
posted by mwhybark at 10:23 AM on October 22, 2018 [3 favorites]

Technically this request can done with a single line powershell command.

The hardest thing about technology is people.
posted by davejay at 10:36 AM on October 22, 2018 [2 favorites]

Technically this request can done with a single line powershell command.

Everyone's an expert.

Possibly also known as: "Engineers disease"... I refer to statements like the above as: "famous last words"... heh... does he even know what their mail server software is? Even if it is Exchange and could potentially be manipulated by PowerShell, which version... Is it just one environment, or many? Is it on-premises or cloud-hosted - or hybrid.

And then - how competent are the admins doing the work? Are they even employee's, or is this maintained by a vendor/service-provider? Locally? Nationally? Internationally?

Yes, it is the people...
posted by jkaczor at 10:42 AM on October 22, 2018 [1 favorite]

Hah, delightful! I was puzzled as to why I couldn't share the data; it seems that my account no longer has data-set creation and publishing rights. Sure looks a lot like this wise and thoughtful use of my tax dollars is no longer going to be supported moving forward. aargh.
posted by mwhybark at 10:59 AM on October 22, 2018 [1 favorite]

The difference is that baseball (or any sport, for that matter) does not have actions that will consistently work 100% of the time to provide the same result.

You're missing the point, though. You couldn't possibly know everything about the environment down at city hall, so the flip suggestion of "just do X because X always works" is short-sighted and self-absorbed. Maybe X won't work here. Sure, X is a great idea literally everywhere else, and it's absolutely foolish to not being doing X. But here we are. And this guy's getting annoyed because the world isn't working exactly how he expects it to. Clearly, they must be idiots down at city hall. Maybe he can help enlighten everyone, if only we listened. Talk about passive aggressive.

Reminds me of the scene from She's Out of My League. "Underwear would be fine. If I were..."
posted by Cool Papa Bell at 11:30 AM on October 22, 2018 [4 favorites]

2800 words from Matt Chapman.

Not that Matt Chapman. Although he could be considered something of an expert on the subject of emails.
posted by Strange Interlude at 12:24 PM on October 22, 2018

does he even know what their mail server software is?

At this point he probably does know from browsing through the emails they mistakenly gave them. He looked close enough to chide the IT department about disk use.
posted by paper chromatographologist at 12:44 PM on October 22, 2018 [2 favorites]

At this point he probably does know from browsing through the emails they mistakenly gave them

After reading further, yes - he must know by now - and he also seems familiar with some of their other storage decisions. (And, it wouldn't be a bad guess that Seattle might be using Microsoft products, even if they do some hosting with Amazon...)

(Plus, I am sure you can get some low-level header information just by sending/recieving messages - possibly even attempting to connect via their public MX records)

(And, yes, I realize that I also suffer from both 'engineer' and 'consultants' diseases... Ironic...)
posted by jkaczor at 1:39 PM on October 22, 2018 [1 favorite]

And this guy's getting annoyed because the world isn't working exactly how he expects it to.

I think you are misreading the article. The original FOIA request did not include this line, which is what we are discussing:
Technically this request can done with a single line powershell command.
This is followed in the article by:
At a policy level, though, it usually gets a lot of pushback.

He's not telling Seattle how to respond to his request. He's illustrating that providing responsive data to his request should be relatively trivial, and the pushback he often gets (including in this case) is that his request is ridiculously burdensome. He's annoyed because "omg computers" is used as a shield to avoid being responsive, either because IT is incompetent, the FOIA responder doesn't understand it, or the FOIA requester is trying to get out of being responsive, not because they aren't doing it his specific way. If city hall is incompetent, he also has every right to be annoyed. Why should the tax dollars of citizens of that city pay for people who don't know what the hell they're doing?

You're missing the point, though.

See above. He didn't provide a technical guide (or even the "single line ... command") to respond to his request; he was trying to make a larger point that it seems we both missed.

That said, your baseball analogy still doesn't hold up, even allowing for differences in implementations of commands across computer systems (which I explicitly acknowledged, and even provided an example of, in my original comment).
posted by tubedogg at 5:50 PM on October 22, 2018 [1 favorite]

Really conflicted after reading this post. On the one hand, this guy is right. This should be a pretty straightforward request. On the other hand, my line of work wouldn’t exist if it always were that straightforward (my line of work should probably not exist).

But mostly I’m feeling a bit sheepish because, hi!, I’m the guy who would send you an email like this:

The information that you requestedis located in columns:
From address = column J
To address = column K
bcc address = column M
cc address = column L
Time and date = column R of the reports.
The records were generatedfrom a system report and I am unable to limit the report to generate only thefields you requested.

I hate my job and don’t want to deal with this shit at 6:30 on a Friday. So, yes, it is the people. Sometimes it’s very specifically me.
posted by GalaxieFiveHundred at 8:07 PM on October 22, 2018 [3 favorites]

As for how the content of the messages got out, I’m sure something like “email body” is one of the fields in the canned report and when you’re dealing with 32 million records, you can’t just put that shit in Excel and delete the relevant column. It’s not an insurmountable problem but whoever ran the export probably didn’t Q/C the resulting file. That also accounts for the fucked up delimiters. I sympathize.
posted by GalaxieFiveHundred at 8:22 PM on October 22, 2018

I have no idea about the extent to which the city relies on MS email products but a former co-worker of mine at the now-shuttered cable access channel alleges that the cable-licensing set-asides used to fund the access channel were redirected to MS email-product seat licenses, which appears entirely likely to my mind.
posted by mwhybark at 9:42 PM on October 22, 2018

I fundamentally disagree with the idea that simply asking for Metadata about every email sent through a government server would have been without privacy implications if Seattle hadn't messed up what they sent.

Even without the contents of the email, there are privacy implications if employee X sent a whole bunch of email to outpatient rehab facility Y when employee X's job has nothing to do with licensing or supervising rehab facilities, for example, and you don't have to know what the emails said to draw some conclusions about employee X or a member of their family.

Yeah, I'm sure every government employee signs a piece of paper that says they can't use the email system for personal use, isn't private, etc, etc. But what is legally covered and what is actually private are two different things in many cases.
posted by jacquilynne at 5:02 AM on October 23, 2018 [2 favorites]

As with any large enterprise, I'm sure there's people who didn't get the memo, but my (limited) experience with people who have a seattle.gov email address is that they're pretty aware that their emails are not only not private but like, not private in the "could become public record at any time" sort of way. As mentioned above, we have really strong public records requirements, and it's not a super new system.
posted by vibratory manner of working at 11:17 AM on October 23, 2018

If you literally asked the person, "Hey, do you mind if I make it public that you email your mistress or AA sponsor four times a day?" would they say yes?

The fact that they were aware that it could be made public doesn't actually make it okay to make it public.
posted by jacquilynne at 2:22 PM on October 23, 2018

Technically this request can done with a single line powershell command.

The problem with this whole line of thinking is that it's focusing narrowly on the thing that doesn't really take the time, the extraction of the data from a database.

To use a perhaps strained analogy, technically, buying a house is as simple as handing over a cheque and signing your name a couple of times. That's technically correct, but ignores all the machinery of getting mortgages, looking for properties, getting legal work done on title, the dozens of things like services, taxes, inspections, etc... that need to be figured out too.

Similarly, there's a lot more to a FOI request than simply the act of collecting the information. As I alluded to above, that has to be reviewed, by hand, for every single record for compliance to any number of protections and rights those affected might have. And that, in my experience, greatly outweighs any time collecting might take.

That's why this is an "engineer's disease" problem. He handwaves away the greater part of the work, and only focuses on that part he knows about.
posted by bonehead at 2:30 PM on October 23, 2018 [1 favorite]

And the fact the city fucked up as badly as they did says to me that not enough time was spent on review.
posted by bonehead at 2:53 PM on October 23, 2018 [1 favorite]