Mining and mapping comments to the FCC on Net Neutrality
August 13, 2014 1:52 PM Subscribe
Despite the comment collecting engine crashing on the last day to submit comments on the very popular topic of Network Neutrality, the system worked well enough to collect 1.1 million comments, which the FCC has made available to the general public as six XML files, totaling over 1.4 gigs of raw data. Mailed comments postmarked prior to July 18 are still being scanned and entered, so this isn't everything, but it's a lot of data. TechCrunch graphed the frequency of certain words, with the high score going to Comcast, with 4,613 mentions. NPR shared the visualized results of Quid's analysis of a sample of 250,000 comments, and Quid's analysis of a sample of 317,000 comments to map geographic sources of the public comments and adjusted them based on state populations to depict which states care more about net neutrality, while The Verge dug deeper, mapping comments by zip code.
I think it's interesting that only form letters seemed to be used for anti-net neutrality comments. Astroturfing?
posted by oceanjesse at 2:06 PM on August 13, 2014 [6 favorites]
posted by oceanjesse at 2:06 PM on August 13, 2014 [6 favorites]
I like how "free speech" was just a little more common than "Netflix" in the comments.
The zip code breakdown is really interesting. 44 people in my zip code wrote in. We just had a primary here, and I was browsing through those results, so trying to see how many people in my zip code voted in the primary. Definitely more than 44.
posted by RainyJay at 2:45 PM on August 13, 2014
The zip code breakdown is really interesting. 44 people in my zip code wrote in. We just had a primary here, and I was browsing through those results, so trying to see how many people in my zip code voted in the primary. Definitely more than 44.
posted by RainyJay at 2:45 PM on August 13, 2014
None of the people in my zip code wrote in, which is... well, wrong by at least 1. Dubious and dubiouser.
posted by Emperor SnooKloze at 2:47 PM on August 13, 2014
posted by Emperor SnooKloze at 2:47 PM on August 13, 2014
I like how "free speech" was just a little more common than "Netflix" in the comments.
This is kind of an aside, but I'd caution against these kinds of kneejerk observations based on word frequency. For example, there's a lot more ways to get across the idea of free speech in text than the idea of Netflix, which frequency analysis would fail to account for. It's a lot like saying people in general care more for X than Y because it gets more Google hits, which is a similar kind of mistake.
posted by JHarris at 3:16 PM on August 13, 2014 [2 favorites]
This is kind of an aside, but I'd caution against these kinds of kneejerk observations based on word frequency. For example, there's a lot more ways to get across the idea of free speech in text than the idea of Netflix, which frequency analysis would fail to account for. It's a lot like saying people in general care more for X than Y because it gets more Google hits, which is a similar kind of mistake.
posted by JHarris at 3:16 PM on August 13, 2014 [2 favorites]
Anyway, my own zip code is the quasi-rural 31523, which somehow got 8 responses.
posted by JHarris at 3:22 PM on August 13, 2014
posted by JHarris at 3:22 PM on August 13, 2014
I would bet anything in the world that this is going to end in some horribly nerfed law that sort-of-but-not-really defends net neutrality. Like something that makes provisions for load balancing to "maintain network integrity" and allows for different QoS priorities or something. Basically, they manage to sneak a bunch of really technical language 8 pages deep by a bunch of old congresspeople.
And then we're right back where we started, but now the rot is entrenched even deeper in the system.
I'd love to be wrong, and maybe i've gotten too cynical with age, but i think we're far too deep into the black hole of regulatory capture here. Possibly even past the event horizon.
At the very most we'll stave this off until after the 2016 election, and then it's off to the races. This, like drilling in national parks or something, is not an if so much as a when. It's sad, but it really feels true at this point.
We can huff and puff, and they'll set up a cardboard house for us to blow down... while they finish construction their hardened fallout shelter elsewhere.
posted by emptythought at 4:29 PM on August 13, 2014
And then we're right back where we started, but now the rot is entrenched even deeper in the system.
I'd love to be wrong, and maybe i've gotten too cynical with age, but i think we're far too deep into the black hole of regulatory capture here. Possibly even past the event horizon.
At the very most we'll stave this off until after the 2016 election, and then it's off to the races. This, like drilling in national parks or something, is not an if so much as a when. It's sad, but it really feels true at this point.
We can huff and puff, and they'll set up a cardboard house for us to blow down... while they finish construction their hardened fallout shelter elsewhere.
posted by emptythought at 4:29 PM on August 13, 2014
Well, it hasn't happened yet emptythought. Let's cross that bridge when we come to it. In the meantime, it's heartening that so many Americans care about this.
posted by JHarris at 5:52 PM on August 13, 2014 [1 favorite]
posted by JHarris at 5:52 PM on August 13, 2014 [1 favorite]
If this isn't a sure sign that the common man cares about this in the USA, I don't know how to help ya.
posted by oceanjesse at 7:31 PM on August 13, 2014 [1 favorite]
posted by oceanjesse at 7:31 PM on August 13, 2014 [1 favorite]
All this work just to get around having to read the comments?
Ha, if only. I've worked for the regulatory arms of two different Federal agencies--and will be getting out verrrry shortly, hallelujah--and trust me, we always have to read (and classify) (and respond to*) all the comments. If comments are not addressed**, we run the risk of being sued by interested parties or stonewalled by OIRA*** or investigated by the GAO****, any of which stops the regulation dead. I understand that regulatory capture is a legitimate concern, but the Obama administration in particular has ensured that the regulatory process is sloooow (you can browse the proposal dates at Regulations.gov if you're really bored) and puts the burden of proof on the agencies to show that new rules are necessary and beneficial. Say hello to E.O. 13563 [pdf].
Basically, they manage to sneak a bunch of really technical language 8 pages deep by a bunch of old congresspeople.
Just to back up, the process is like this: Congress writes the laws, the Executive writes the regulations to interpret and clarify the provisions of those laws (and the Judiciary, if necessary, determines their constitutionality). Some laws are self-executing (e.g., the Federal minimum wage law) but most of them are of the discretionary "There shall be a program that oversees X" variety, which leaves the Executive Branch a lot of leeway to determine how to carry out X.
Which is a roundabout way of saying that Congress doesn't write regulations. It can be a real pain in the ass to regulators, but it's not doing the writing.
---
*The upside is that those that are clearly off-topic or display an incorrect understanding of the proposed regulation [e.g., if the EPA proposes a rule regarding water fluoridation and someone writes in "I don't want my water to come from Florida"], they can be easily dismissed with language explaining such.
**This can mean either (a) providing a convincing justification for why the agency disagrees with the argument of the comment, (b) changing the provisions in the final rule in agreement with the comment, or (c) dismissing the comment as irrelevant, as above.
***This is the office under the White House that decides whether the regulations proposed by Federal agencies get approved and published (and hence given the force of law). It is probably one of the most powerful offices in the USG you've never heard of. I was just notified that one of the rules I started working on at my old agency four years ago, which every stakeholder on every side of the industry we covered agreed was a necessary regulation, has just been approved. I still have nightmares about the rewrites on that document.
****They are rottweilers, and despite existing to undertake congressional inquiries, this is probably the most ethical and professional organization in the USG.
posted by psoas at 1:55 PM on August 14, 2014 [4 favorites]
Ha, if only. I've worked for the regulatory arms of two different Federal agencies--and will be getting out verrrry shortly, hallelujah--and trust me, we always have to read (and classify) (and respond to*) all the comments. If comments are not addressed**, we run the risk of being sued by interested parties or stonewalled by OIRA*** or investigated by the GAO****, any of which stops the regulation dead. I understand that regulatory capture is a legitimate concern, but the Obama administration in particular has ensured that the regulatory process is sloooow (you can browse the proposal dates at Regulations.gov if you're really bored) and puts the burden of proof on the agencies to show that new rules are necessary and beneficial. Say hello to E.O. 13563 [pdf].
Basically, they manage to sneak a bunch of really technical language 8 pages deep by a bunch of old congresspeople.
Just to back up, the process is like this: Congress writes the laws, the Executive writes the regulations to interpret and clarify the provisions of those laws (and the Judiciary, if necessary, determines their constitutionality). Some laws are self-executing (e.g., the Federal minimum wage law) but most of them are of the discretionary "There shall be a program that oversees X" variety, which leaves the Executive Branch a lot of leeway to determine how to carry out X.
Which is a roundabout way of saying that Congress doesn't write regulations. It can be a real pain in the ass to regulators, but it's not doing the writing.
---
*The upside is that those that are clearly off-topic or display an incorrect understanding of the proposed regulation [e.g., if the EPA proposes a rule regarding water fluoridation and someone writes in "I don't want my water to come from Florida"], they can be easily dismissed with language explaining such.
**This can mean either (a) providing a convincing justification for why the agency disagrees with the argument of the comment, (b) changing the provisions in the final rule in agreement with the comment, or (c) dismissing the comment as irrelevant, as above.
***This is the office under the White House that decides whether the regulations proposed by Federal agencies get approved and published (and hence given the force of law). It is probably one of the most powerful offices in the USG you've never heard of. I was just notified that one of the rules I started working on at my old agency four years ago, which every stakeholder on every side of the industry we covered agreed was a necessary regulation, has just been approved. I still have nightmares about the rewrites on that document.
****They are rottweilers, and despite existing to undertake congressional inquiries, this is probably the most ethical and professional organization in the USG.
posted by psoas at 1:55 PM on August 14, 2014 [4 favorites]
That's great information psoas! Flagged as absurdly helpful.
posted by JHarris at 7:39 PM on August 14, 2014
posted by JHarris at 7:39 PM on August 14, 2014
« Older BGP Hijacking for fun and profit! | Walk a mile in her face. Newer »
This thread has been archived and is closed to new comments
posted by oceanjesse at 2:00 PM on August 13, 2014 [2 favorites]