Feature design versus trolling
December 5, 2016 2:22 PM   Subscribe

“Harassers are very clever. They take advantage of tools that are very innocuous and use them as vectors for abuse,” Ehmke told me. “If you’re creating products and not thinking about how it could be used for abuse, you are not doing your job.” - How Github is dealing with its troll problem.
posted by Artw (49 comments total) 39 users marked this as a favorite
 
"It might surprise you that a website built for programmers to share code could become a hotbed of online harassment. "

No. It would surprise me if any unmoderated space online didn't become a hotbed of harassment.
posted by maxsparber at 2:26 PM on December 5, 2016 [65 favorites]


I came here to quote that exact line. Not only that it's an unmoderated online space, but that it's a space for programmers.

Software has a major sexism problem. Everything listed as problems in the article makes me angry and/or sad, but none of it surprises me.
posted by tocts at 2:32 PM on December 5, 2016 [15 favorites]


You have a not insignificant percentage of humanity who are a**holes but who are too chickensh*t to do it face-to-face. The internet and its anonymity allows them to finally LIVE THE DREAM!
posted by jim in austin at 2:35 PM on December 5, 2016 [13 favorites]


This piece made me finally draw the connection that aside from its other ills, a "non-hierarchical structure" internally allows the plausible deniability of external responsibility - no one's in charge so it's no one's fault.
posted by PMdixon at 2:38 PM on December 5, 2016 [29 favorites]


Yup. That's the reality of the internet. And given that, I think this...

Their job isn’t just to build new anti-harassment tools, but to vet new GitHub features and anticipate how they might be used for abuse.

...is vital. This should be part of everybodies product design process.
posted by Artw at 2:38 PM on December 5, 2016 [12 favorites]


Not every space on the internet has to be squeaky clean.

“There need to be spaces on the internet with different social norms,” said Amy Bruckman, a researcher at Georgia Tech who studies online communities. “There should be online spaces that are a little more rough and tumble. That’s okay as long as they are open about what they are and don’t cross a line that’s actually dangerous.”

In other words, it all comes back to rules. Just like the real world, there are different spaces for everyone. Maybe Reddit is your neighborhood dive bar, and Facebook your corner coffeeshop. We can all agree that what happens in a dive bar at 4 a.m. is not always appropriate in a coffee shop.


I hate the implication that spaces that allow for racist/sexist/tran-and homophobic abuse and harassment are just "rough and tumble," which is how this comes off in the context of the piece, and is not (I hope) what Bruckman actually means. A place that doesn't allow its participants to harass other people is not "squeaky clean."

Bigoted abuse and harassment are not appropriate at either a dive bar or a coffee shop. Or they shouldn't be.
posted by rtha at 2:39 PM on December 5, 2016 [64 favorites]


That pull quote in the post description is fascinating to me. Because Chris Wanstrath and PJ Hyett rebuilt Chowhound and its moderator tools back in 2006, and that would not at all describe the approach taken then. It's good to see things evolving.
posted by jacquilynne at 2:43 PM on December 5, 2016


The GitHub Community Guidelines have been made public domain, so anyone can reuse and adapt them. Which I think is rather excellent.

Asshole-proofing communities is a fascinating topic. I like these guidelines because they don't just tell you what not to do, they also provide suggestions on positive behaviour.
posted by davidwitteveen at 2:52 PM on December 5, 2016 [5 favorites]


(also VP of Social Impact is a baller title)
posted by PMdixon at 2:53 PM on December 5, 2016 [2 favorites]


This piece made me finally draw the connection that aside from its other ills, a "non-hierarchical structure" internally allows the plausible deniability of external responsibility - no one's in charge so it's no one's fault.

Related:
https://en.wikipedia.org/wiki/The_Tyranny_of_Structurelessness


Though I think the proper conclusion is not that lack of hierarchy automatically means lack of accountability-- it's that an improper or nonexistent structure will allow de facto hierarchies to arise, and erode accountability, especially for organizations (GitHub) existing inside of already oppressive structures (programming culture, corporate culture, the world).
posted by abrightersummerday at 2:55 PM on December 5, 2016 [13 favorites]


“There need to be spaces on the internet with different social norms,” said Amy Bruckman, a researcher at Georgia Tech who studies online communities. “There should be online spaces that are a little more rough and tumble.

And yeah, I want to call this out as well. I really hate that writers of articles, researchers, etc, feel the need to say something like this whenever harassment is discussed -- as if it's not self-evident. Yeah, no shit, different spaces can and should have different norms, and nobody is arguing that that isn't the case. However, some behavior is unacceptable literally anywhere, and we shouldn't have to yet again soothe the anxieties of the privileged just to be allowed to meekly make the point that hey, online spaces have to be built with harassment prevention in mind from the get-go, and not have it tacked on as an afterthought.

Shit like the above quoted is basically just another face of #notallmen, and has no place in an article trying to seriously address harassment issues.
posted by tocts at 3:01 PM on December 5, 2016 [27 favorites]


I hate the implication that spaces that allow for racist/sexist/tran-and homophobic abuse and harassment are just "rough and tumble," which is how this comes off in the context of the piece, and is not (I hope) what Bruckman actually means. A place that doesn't allow its participants to harass other people is not "squeaky clean."


Well, there are places on the web where people go specifically to fight/troll each other, and thus some things that would constitute harassment most places would not be out of step when everybody has signed up for it. But there are also places in the real world people go literally to fight each other - and in both cases it's actually particularly important to make clear what "too far" means.
posted by atoxyl at 3:06 PM on December 5, 2016 [10 favorites]


Fusion.net trolled me just now! I like seeing stories about how people encourage civility, then they ajaxed one of those "get worked up by seeing this person's terrible tweets" stories onto the same page, so it appeared when I scrolled down. I don't need to get any more worked up, Fusion!
posted by yorick at 3:22 PM on December 5, 2016


a "non-hierarchical structure" internally allows the plausible deniability of external responsibility - no one's in charge so it's no one's fault.

Responsibility could be proportionate to the money earned, where money is earned.
posted by clew at 3:23 PM on December 5, 2016


It might surprise you that gasoline-soaked rags are flammable ...
posted by tocts at 3:41 PM on December 5, 2016 [14 favorites]


"It might surprise you that a website built for programmers to share code could become a hotbed of online harassment. "

Every dev/it community that I've been part of has been absolute shit. These days the only dev community I visit is Hacker News and surprise surprise it's shit.
posted by Foci for Analysis at 3:46 PM on December 5, 2016 [6 favorites]


Ahhh... Not only can I download potentially malware ridden code, but I can be exposed to vicious polyphobic crap too. The internet. The more I know the less I really want to do there.
posted by njohnson23 at 3:49 PM on December 5, 2016 [2 favorites]


“If you’re creating products and not thinking about how it could be used for abuse, you are not doing your job.”

I interviewed the maker of a "smart home" locking system where the idea was that one person in the house controlled the app & they, in turn, could determine who could enter and exit the house based on their smartphones (which also had apps). In other words, one person could literally decide whether or not to open a door to let someone in or out, and they didn't even have to be home to do it.

I asked, "So what are your safeguards against preventing abusers from locking out the kids they disown or locking in the spouse who's trying to flee?"

The man literally gaped for a minute before saying, "I don't think the tech would be used like that."

It was both a staggering encapsulation of privilege and an example of not thinking through abuse cases for your tech.

(p.s. I talked to two other smarthome doorknob/lock makers. They didn't have answers either.)
posted by sobell at 3:53 PM on December 5, 2016 [92 favorites]


Shit, I even dunno what's Schlage's plan to deal with abusive individuals changing the physical keys on a dumb lock.
posted by save alive nothing that breatheth at 3:58 PM on December 5, 2016 [4 favorites]


""So what are your safeguards against preventing abusers from locking out the kids they disown or locking in the spouse who's trying to flee?""

What are the current safeguards against this with traditional locks? And if abusers lock their disowned kids out, I don't think letting the kids to be back in the house to stay with the abusers is what's important.
posted by I-baLL at 5:18 PM on December 5, 2016



What are the current safeguards against this with traditional locks?

Hmmm, let's see.... they actually have to call someone to change the locks? or physically do so themselves? Also physical locks can be overridden? Unless there is a physical override on the "smart home" lock - which I did not see the last time I looked at one, surely you can see how there might be a problem?
posted by Gyre,Gimble,Wabe, Esq. at 5:23 PM on December 5, 2016 [20 favorites]


Physical locks can be overridden in a way that smartlocks can't? What do you mean by that?

And I don't understand what the issue is. You're saying that locks that you purchase for your home shouldn't be secure or controlled by you because somebody might abuse it? That's like saying that we shouldn't have homes because abusers will abuse people in them. I mean, seriously, how would the demands of "So what are your safeguards against preventing abusers from locking out the kids they disown or locking in the spouse who's trying to flee?" be achievable without a third party having access to everybody's house and everybody's smartlock?

And, let's say that you have a house and you have locks on your house and a stalker or abuser gets a hold of a key. You don't want the key to be easily changeable? How many abuse victims would be glad to know that they're not going to be allowed, by this logic, to have locks that they can have control over? I don't understand this logic.
posted by I-baLL at 5:31 PM on December 5, 2016 [6 favorites]


Try thinking about it as a matter of effort, which is important. It takes time and effort to physically change the locks, and, importantly, there is no hiding the fact that it's being/been done, and that it's potentially as inconvenient for the lock master as the lockee.

An app where an abusive individual can change whether the door can be opened on a whim? Another toy for them to torment their victim.

That little bit of distance is often the difference between something that's OK and something that's awful. Think of an open comment system vs. a $5 to comment system.
posted by maxwelton at 5:41 PM on December 5, 2016 [35 favorites]


And I guarantee you an app to open doors also contains a nifty "logging" feature so you can see how many times the door was opened. "Why did you open the door at 3? Who was it!"

Dude, you don't have to think hard about this stuff.
posted by maxwelton at 5:42 PM on December 5, 2016 [31 favorites]


Or - ever been trapped in the back seat of a car where the driver has engaged (in my case on accident) the child safety lock. The rear doors cannot be opened. In innocent instances, hilarity ensues...... When I looked at the "smart home" lock a while ago, there wasn't a contingency for physical override of the lock. If the internet went down - I was gonna be standing outside in the rain until it came back up - seemed like a significant design flaw, to me. Now, imagine I were intent on denying exit.....
posted by Gyre,Gimble,Wabe, Esq. at 5:48 PM on December 5, 2016 [6 favorites]


Mod note: A few deleted; I-baLL, probably better to just step away for a bit.
posted by LobsterMitten (staff) at 5:50 PM on December 5, 2016 [2 favorites]


For anyone who honestly needs an example of where this kind of lack of consideration of the ramifications of a feature can put people at risk, look at Google Buzz as a case study. Google literally rolled out a opted-in-by-default feature that would reveal your most frequent contacts to the world. Apparently not a single person involved pre-launch thought through whether this might e.g. reveal that an abused spouse was talking with a divorce lawyer, or that a closeted LGBT child was talking with a support group, etc.

This was in 2010, and in my estimation not a ton has changed. This kind of careless design still happens all the time, and we (the software development community) learn slowly if at all from the mistakes. Smart homes are a fucking minefield of privacy and abuse-enabling concerns, few if any of which are being considered by the companies rushing to make money from them.
posted by tocts at 6:39 PM on December 5, 2016 [15 favorites]


And no one is saying "don't make thing Z", they're saying "thing Z can be used as a weapon, you need to mitigate that." And yes, if you're making a thing, whether an online repository or a door lock, part of your job in making it is making sure it doesn't harm someone. Frankly, I don't need products from inventors who can't be arsed to spend five minutes imagining the ways bad actors might use them. (How much human effort, time and misery would have been saved if the people who invented email had spent even a few minutes considering human nature?)
posted by maxwelton at 7:01 PM on December 5, 2016 [13 favorites]


Kudos to GitHub. This wasn't lip service to the problem. They literally changed their corporate structure and hired a team that included some of their strongest critics-- and that team now vets all new products for how they will impact these issues.
posted by gwint at 7:19 PM on December 5, 2016 [13 favorites]


"Well, there are places on the web where people go specifically to fight/troll each other, and thus some things that would constitute harassment most places would not be out of step when everybody has signed up for it. But there are also places in the real world people go literally to fight each other."

And most of the time, those rough and tumble places aren't places like libraries or family restaurants or offices or elementary schools. When schools in the south got a little too rough and tumble during desegregation, we sent the National Guard in. Some places just shouldn't be rough and tumble. If you want to be rough and tumble online, you shouldn't be able to do it at a place like Github (or Metafilter, or Facebook, or any other site I've ever heard of because I don't got to places where it's expected to be assholes).
posted by kevinbelt at 7:28 PM on December 5, 2016 [3 favorites]


I'm glad they're making this effort now, but like...too late? This is their community now. I know who lives there. I wouldn't use social features on github or stick my neck out at all (I have an account for stuff, but I'm not a developer and don't need it professionally.)

How long does it take new norms to settle in? To genuinely settle in, so that the people who comprise the community are changed too?
posted by schadenfrau at 7:39 PM on December 5, 2016


"Well, there are places on the web where people go specifically to fight/troll each other, and thus some things that would constitute harassment most places would not be out of step when everybody has signed up for it. But there are also places in the real world people go literally to fight each other."

Why you didn't complete the quote is curious:

--and in both cases it's actually particularly important to make clear what "too far" means.

I agree the analogy of #notallmen is a valid criticism and a vital compartment, while my first reference for Amy Bruckman's distinction was Fark, versus the goto of 4Chan. The longevity of any site speaks to a cultural relevance, its internal changes as well. I noticed around the time Drew Curtis had a family and Google Analytics made him a very rich man, the front page boobies tag was retired.

And, on preview, in matters of protection, equality, and safe places, I'm pressed to cite any effort that has not been too late.
posted by lazycomputerkids at 7:50 PM on December 5, 2016


schadenfrau, I think it depends on the community. Witness the evolution here. I joined the site about 16 years ago, and the boyzone level was almost impossible at times. I had to switch to a sockpuppet during gamergate, and this is the only site I kept active when I dropped fb and Twitter and everywhere else I had a public profile. Because metafilter has evolved as a strong, supportive, tolerant community willing to listen and learn and grow.

So, change can happen, but it's a lot of work on the part of both members and mods.
posted by SecretAgentSockpuppet at 8:11 PM on December 5, 2016 [15 favorites]


"Why you didn't complete the quote is curious"

Because I wasn't talking about spaces where it's ok. I was talking about places like schools and offices where just about anything is too far.
posted by kevinbelt at 8:40 PM on December 5, 2016


Alternatively
posted by fallingbadgers at 8:46 PM on December 5, 2016


FYI, I've never seen a smart lock that overrode a physical key. The benefit is that i don't need to fish out my keys from the bottom of my purse, and I can let people in remotely. It does not mean that I'm locked out of my house when the internet goes down. It just reverts to a dumb lock.

I can't believe we're fighting about features that don't exist.
posted by politikitty at 11:00 PM on December 5, 2016 [6 favorites]


I did something like this. I worked with a bunch of game designers on an online game. As they'd come up with mechanics I would Red Team it and figure out how I'd use it to troll or grief or otherwise use it for unintended purposes. Naturally, they took my advice to heart. Just kidding, they fired me for being too depressing and "nobody would ever think of those things or do that."

They were genuinely shocked when people thought of those things and did that.
posted by Ghostride The Whip at 11:43 PM on December 5, 2016 [44 favorites]


That reminds me of Bruce Schneier's anecdote at the start of his essay The Security Mindset
Uncle Milton Industries has been selling ant farms to children since 1956. Some years ago, I remember opening one up with a friend. There were no actual ants included in the box. Instead, there was a card that you filled in with your address, and the company would mail you some ants. My friend expressed surprise that you could get ants sent to you in the mail.

I replied: "What's really interesting is that these people will send a tube of live ants to anyone you tell them to."
posted by Joe in Australia at 11:52 PM on December 5, 2016 [23 favorites]


Speaking of having a lack of adversarial design thinking in your software development, Google has just released an app called Trusted Contacts:
This new personal safety app lets you share your location with loved ones in everyday situations and when emergencies arise — even if your phone is offline or you can’t get to it.
Yup, this just happened.
posted by Foci for Analysis at 12:45 AM on December 6, 2016 [3 favorites]


Oh dear.

I like how they have a hypothetical situation involving "Elliot" and "Thelma", but Thelma is checking up on Elliott.

Maybe Elliott wants to check up on Thelma, because she said she was going to the grocery but it doesn't take that long? Maybe she's meeting someone she hasn't told him about, like a friend from work? Now Elliot can check on Thelma 24/7, and if she doesn't answer (or has turned her phone off, like she promised not to) he can tell the rest of the prayer circle and they can check out her last recorded location.

Just to make sure.
posted by Joe in Australia at 1:16 AM on December 6, 2016 [5 favorites]


The smartphone is the ideal tool for total surveillance & that “feature” is inherent to the technology - you can’t have the upsides without that deeply unfortunate downside. We tolerate this because the utility they offer is so great of course, but it’s not news that it’s not all sweetness and light in tech-land.

Honestly, it would be nice to swing this conversation back to the original topic of online social spaces & what can be done to limit harassment there rather than a generic dumping ground for 'ugh, tech bros are terrible amiright, look at *this* awful thing they’ve just done'.
posted by pharm at 2:29 AM on December 6, 2016 [4 favorites]


Speaking of having a lack of adversarial design thinking in your software development, Google has just released an app called Trusted Contacts

This sounds like Google's me-too version of Apple's "Find My Friends" feature, which I have yet to see any hand-wringing about.

(An interesting problem, though -- how do you keep abusers from forcing their victims to "opt-in" to compromising their privacy? Don't really see a way around this, aside from allowing users to fake their location.)
posted by neckro23 at 5:30 AM on December 6, 2016 [1 favorite]


even if your phone is offline or you can’t get to it.

At least it looks like this just means giving up your last known location, and not anything sneakier...
posted by ominous_paws at 5:33 AM on December 6, 2016


If you like the GitHub look and feel, but want control over the social features, then check out its open source clones GitLab and Gogs.
posted by jeffburdges at 6:42 AM on December 6, 2016 [4 favorites]


I've always been interested in the increasing toxicity of online communities as one approaches technical subject matters. It's not necessarily true in RL (although it sometimes definitely is).

E.g.

(in ascending toxicity from NONTOXIC to DO NOT EAT)

[Website about geoducks] --> [Website about geoduck databases] --> [ DB Programming]
posted by mrdaneri at 7:38 AM on December 6, 2016 [2 favorites]


A blank wall will always attract graffiti!!
posted by Burn_IT at 8:33 AM on December 6, 2016 [1 favorite]


The Postgres crowd are pretty chill in my experience.

Which is to say: biology is not destiny. Or something.
posted by pharm at 11:45 AM on December 6, 2016 [1 favorite]


Though a graffiti'd wall attracts graffiti more, so that analogy doesn't exactly...

Actually, you know, it still does. Twitter is the graffiti'd wall, and people flock to it to graffiti it more. Other places, where that shit is cleaned on the regular, tend to suffer less at the hands of trolls.

Again, Mefi is the best $5 I've ever spent, I think.
posted by Imperfect at 11:47 AM on December 6, 2016 [6 favorites]


"Don't really see a way around this, aside from allowing users to fake their location",

Nope, won't work, follow someone, in secret, so you know where they are, then request their location. Or just do it repeatedly until they are somewhere they can't be. Like people don't already do this phoning someone and asking where they are to catch them out.

In any case allowing fakes just gives someone one more thing to accuse someone of.
posted by tallus at 10:11 PM on December 7, 2016


« Older The Real U.S. Embassy in Accra is White, not Pink   |   "...and we'll have a burial for him tomorrow.” Newer »


This thread has been archived and is closed to new comments