Don't be evil
August 8, 2018 4:15 PM   Subscribe

"The last thing you want is to get blindsided by a future YOU helped create. The Ethical OS is here to help you see more clearly."

"If the technology you’re building right now will some day be used in unexpected ways, how can you hope to be prepared? What new categories of risk should you pay special attention to now? And which design, team or business model choices can actively safeguard users, communities, society, and your company from future risk?"

The full toolkit

Wired: Silicon Valley writes a playbook to help avert ethical disasters

Medium: Introducing the (world's first) ethical operating system
posted by Paragon (47 comments total) 28 users marked this as a favorite
 
“Come read our pamphlet on fire safety,” say the people standing inside the house they lit on fire.
posted by not_the_water at 5:00 PM on August 8 [44 favorites]




Meanwhile, what’s the source of the post title up to these days?

Google plans return to China search market with censored app
posted by Sangermaine at 5:08 PM on August 8 [2 favorites]


Is this some kind of deep parody? Because... man... it reads like one.
posted by Nanukthedog at 5:23 PM on August 8 [6 favorites]


Silicon Valley: the tech we make keeps getting used in surprising ways for evil by people ok with breaking the rules.

Also Silicon Valley: we have an answer! We'll make more tech, but with rules!
posted by eustacescrubb at 5:31 PM on August 8 [13 favorites]


If you opt to follow this AND you fail to properly anticipate a use which is evil, you need to be ready to answer the questions: is this way your product is being used for evil actually admissible, and more importantly - are you officially complacent in whatever evil was committed because you actually did prep? Does prepping for some evil inoculate you from other evil you didn't anticipate in some sort of Good Samaritan sense, or are you still responsible for egregious usage?
posted by Nanukthedog at 5:43 PM on August 8


The China reveal is currently causing an uproar inside of Google (the execs have even cancelled a Friday Q&A session they had planned).

I can totally see this EthicalOS thing as a employee pushed project that the execs didn't care for as anything but cover. But I think we ought to distinguish between Google as run by execs, and the employees they hornswaggle.

Co-determination could help correct the course of companies by giving employees more voice. We've got some senators and reps that are pushing the Reward Work Act that would make a percentage of every company's board of directors ones that were voted in by the employees of the company. There's been some recent polling saying it has a ton of support. It's standard in Germany.
posted by jmhodges at 5:47 PM on August 8 [16 favorites]


1. Let's not have the future overrun by mindless people who've given up on individual thought for technology.
2. WE HAVE AN APP FOR THAT!
posted by DirtyOldTown at 5:53 PM on August 8 [3 favorites]


In my empirical experience there are two or more people in silicon valley and they sometimes disagree about ethics
posted by idiopath at 5:56 PM on August 8 [2 favorites]


I need this the way I need an app to identify pornography for me.
posted by East14thTaco at 6:18 PM on August 8 [4 favorites]


When I read "OS" at the end of a word, I expect to see an operating system, not a PDF full of buzzwords. A lot like when I read the word "toolkit", I expect to see some actual tools, not a selection of writing prompts from Charlie Brooker's notebook.
posted by Juso No Thankyou at 6:33 PM on August 8 [40 favorites]


Clippy sez: It looks like you're laying the groundwork for a culture of anonymous harassment, sheathed by competitive victim-hood. Would you like to see a template?
posted by es_de_bah at 6:39 PM on August 8 [16 favorites]


Reminds me of the "values clarification exercises" they made us do in elementary school back there in the 70s
posted by thelonius at 6:39 PM on August 8 [2 favorites]


in startup world, for a long time, anything but relentless manic positivity about technology has been frowned upon. if it’s no longer taboo to speculate about negative outcomes alongside positive ones, that’s a good thing, I think.
posted by vogon_poet at 6:54 PM on August 8 [23 favorites]


given the state of ethical training for IT and computer scientist graduate in the United States, which I have heard from multiple sources is "jack shit", this feels like a step forward, so I'm not sure why we're so quick to shit on it.

Or you could hire graduates from countries that include ethics training in their IT qualifications, like mine did! I'm available for employment and my rates and ethics are both reasonable
posted by Merus at 7:13 PM on August 8 [19 favorites]


"Everyone wants to do better, but we heard feedback when we were speaking to VCs and tech co-founders that they didn't know how. They didn't know what to do.'"
RAINA KUMRA, TECH AND SOCIETY SOLUTIONS LAB


Poor things, they just don't know what to do! Fortunately this little fig leaf of non thought will make everything crystal clear.
posted by haemanu at 7:16 PM on August 8 [2 favorites]


My Information Technology degree had a requirement for ethics in technology, but that’s one university out of a thousand so... 🤷‍♀️

But it isn’t just IT that needs to think of ethics, it’s leadership and their company’s friggin’ legal team. Look at Wells Fargo, their leadership is ridiculously unethical and essentially force employees to choose between acting unethically and being fired. (Or hell, why not both? WF fired the whistleblowers, too.) While it behooves the people actually creating the product to think of ethical applications as the first line of defense, the final line of defense should come from the people who own the responsibility for the company’s mission and culture.
posted by Autumnheart at 7:29 PM on August 8 [7 favorites]


OS in this case stands for Overstated & Stupid.
posted by Nanukthedog at 7:49 PM on August 8


Can I install it from a USB drive or do I need to burn a CD? /hamburger
posted by ambulocetus at 8:26 PM on August 8 [1 favorite]


Yea, people have been raising these concerns since at least the 90s, that I am aware of, maybe even earlier. Silicon Valley didn't want to hear it. I don't think they'll change a thing until they're loosing serious money and power. As public opinion turns on them more and more, we'll see more "fig leaves" like this, but it might be like if Standard Oil had said at some point "hmm, maybe we shouldn't have killed off public transportation in America." The infrastructure is already in place, and would be hard do dismantle and rebuild.

If anyone is looking for an actual Ethical OS, I would suggest checking out a Linux distro, such as an Ubuntu fork, and learning a bit about what about open source means. I think of it as the "socialist OS."
posted by ethical_caligula at 8:44 PM on August 8 [3 favorites]


Charlie Brooker's toolkit contains devices straight out of Cronenberg's Dead Ringers.
posted by rokusan at 8:57 PM on August 8 [1 favorite]


So on the one hand... sure. People who work in technology are frequently more focused on how cool their achievements could be, and don't always think about misuse of their work. If this gets some optimists to step back and spend a bit of time on how things could go wrong, it's not a bad thing. At some small startups, this could do some good.

But most of the organizations doing problematic things on the Internet are huge. The developers working there mostly have goals focused on relatively short term improvements on small parts of the overall system. Reduce the latency for requests to a specific ads API, increase reliability for the container scheduler, make sure the app crashes 5% less frequently. People largely focus on the job in front of them, unless there's a really fucking huge example of horrible things being done right now. (Such as, for example, the Google employees revolting about rolling out censored search in China.) There are people making decisions about the Big Picture, but the numbers are comparatively small, and you aren't going to reach them with an online slide deck.

Poking individual engineers to Do Better is still probably a good thing, but it's not going to move the ball very far down the field. As we've learned in many other industries, the only thing that makes a material difference is government regulation, monetary liability, or criminal charges. Heck, witness the scramble at many of these companies to make changes ahead of GDPR; those improvements were limited, but it definitely motivated some changes.

Make there be consequences. Otherwise you're depending on the occasional employee protest, but I don't think that will get you far.
posted by fencerjimmy at 9:00 PM on August 8 [14 favorites]


I am dismayed by the brutal reception this is getting here, but I suppose I shouldn't be surprised (c.f. "should nevertrumpers get cookies?"). Many of the "ethical risk zones" listed in the OP have been fertile subjects for FPPs in the past. The message here isn't bad. And if you somehow don't think this stuff needs to be said, you should see how places like hackernews treat the question of whether we should consider the downsides of technological advances.

TLDR; of course not- we're engineers, not philosophers; no one can predict what bad people will do with what we make; our fingers are in our ears, so shut up already.
posted by Jpfed at 9:09 PM on August 8 [9 favorites]


I found the Ethical OS kind of blind in its assumption that bad outcomes only happen because of a lack of thought about consequences. But there IS a tension between ethics and maximizing profits. The limit is usually a legal matter, because in their search for profits companies are not really that good at self policing.

Imagine installing this OS onto different industries. "Everyone wants to do better", "they just did not know what to do" is even more jarring when talking about civil engineering or medicine. In those cases, there are regulatory bodies that make sure consumers are not overly mistreated or harmed by profit driven companies. One can dream that tech companies could be subject to the same level of oversight at some point.
posted by haemanu at 10:22 PM on August 8 [11 favorites]


the brutal reception this is getting here

Much of it seems to be blowback at the gimmicky "OS" framing. Now it's not nothing if many people are so annoyed by the presentation that they never look at the content, but it's not a very serious problem ( it's fixable).
posted by thelonius at 11:09 PM on August 8 [1 favorite]


It seems like a useful exercise to go through, even if the "OS" framing is a bit, well, gimmicky. I had a quick look through and will read it more thoroughly later. I feel that some of it directly applies to me as I'm working on automating away a small number of (boring, repetitive, but still...) jobs.

However, I was ready to dismiss it as satire when one of the first recommendations on the front page were from someone in the "flying car industry".
posted by Harald74 at 12:02 AM on August 9 [3 favorites]


the brutal reception this is getting here


My main issues with it:

(a) It's not an OS. That's not what OS means. I know I'm being pedantic, but they're making a document for developers, who are almost all more pedantic than me. And calling it an OS makes it sound too technical for the CEOs and CTOs who should be reading it. Especially when words like "toolkit" and "framework" and "standards" are perfectly usable instead.

(b) Their Toolkit is so poorly formatted I can barely read it. It looks like a dump of a PowerPoint presentation, and it probably is one. It doesn't make its case clearly and its "scenarios" veer into science fiction, which makes the whole thing look like a joke. There are plenty of ethically questionable platforms and technologies right now that they could use as examples. Also, to be pedantic again, It isn't a Toolkit. A toolkit is a grouping of useful tools. Call it "Our Presentation about Ethics" and it would make more sense.

By contrast, I like their Checklist a lot. It's essentially a quick summary of the same content as the Toolkit without the fluff, and I would definitely see it as something inventors/developers should read through, if only as food for thought.

This isn't really my area, but I haven't seen all of the future-proofing risks of technology summarized as well as that checklist before. I hope they can improve this and get some people to pay attention to it.
posted by mmoncur at 12:20 AM on August 9 [19 favorites]


If anyone is looking for an actual Ethical OS, I would suggest checking out a Linux distro, such as an Ubuntu fork, and learning a bit about what about open source means. I think of it as the "socialist OS."
posted by ethical_caligula at 10:44 PM on August 8
I tried for years to get a linux OS up and running but I didn't have the chops to get it going on any puter I tried it on. Really frustrating, because I loved what it stands for, but I just couldn't make it fly. Then Ubuntu came along, and it was incredibly easy to install, and ran perfectly fine on a Dell laptop I had at that time, and it was the maybe the ugliest thing I'd ever seen -- gawd. It really was gruesome.

Then one day I tried linux Mint. Which, if I recall correctly, does branch off of Ubuntu. And it installed as easy as Ubuntu, and pretty much just worked, just like Ubuntu does. But Mint is absolutely gorgeous, even right out of the box. And the really fun part -- I know what I like, I have certain ways I like to lay things out, from all the Windows variants I've had, since WIN95 forward. And while I was not able to get Ubuntu to do anything but run great and look horrific, Mint runs great and was/is easy-peasy to customize to exactly what I want. It's just such a great OS.
posted by dancestoblue at 12:43 AM on August 9 [6 favorites]


An extreme rush job, why? What's the hurry?
posted by infini at 1:33 AM on August 9


My death ray was built for peaceful purposes!
posted by The Underpants Monster at 1:43 AM on August 9 [1 favorite]


The framing of this is in terms of unintended/unforeseen consequences. However many of the eight 'risk zones' they identify are pillars of the Silicon Valley design book. They're intended and designed to be there, and they're not going to go away as long as they 'produce value for shareholders' etc.
posted by carter at 3:25 AM on August 9 [2 favorites]


Note that this is being put out by a startup accelerator, basically an entity that invests in and mentors startups at a very early stage, before they’ve even completely decided on their business model.

It’s a situation where this kind of thinking would be genuinely helpful, and DEFINITELY has not gone on in the past.
posted by vogon_poet at 3:35 AM on August 9 [5 favorites]


TECH AND SOCIETY SOLUTIONS LAB

That may be your problem right there.
posted by adamgreenfield at 3:49 AM on August 9 [1 favorite]


More seriously, though: several years ago, when both I and the world were more innocent, I wrote a book largely dedicated to exploring the idea that interactive networked technology could be designed with ethical care (here's much of its argument distilled into article form, for tl;dr purposes).

There were two things I knew would frustrate any such ambition, even at the time. The first was that the "networked" aspect of our technologies tends to make a hash of any attempt to hardwire an ethical stance into any one component. Networked technologies, by definition, work as an articulated ensemble, in which functionality is smeared out across a sprawling, heterogeneous, unstable, contingent mesh of devices and services; unless you somehow have end-to-end control over every aspect of that mesh, you have relatively little final control over the posture (or the choices) it presents to a user.

The second frustrating factor is more damning still, if theoretically more tractable and open to change. It is that, at present, all technological devices and services are made inside the context of late capitalism, and the implacable demands of late capitalism will in the end erode and override any attempt to protect the user's ethical prerogatives that interferes with any way with the aims of revenue generation, data capture, brand building, etc. These demands, further, have been so internalized by the class that develops technology that it seems natural to mention "business model" alongside other design desiderata.

The problem is capitalism. It's not technically impossible to make ethically conscious technology inside capitalism, but it is very, very difficult, and such attempts as do emerge are generally not sustained for very long.
posted by adamgreenfield at 4:06 AM on August 9 [11 favorites]


dancestoblue: Then one day I tried linux Mint. Which, if I recall correctly, does branch off of Ubuntu. And it installed as easy as Ubuntu, and pretty much just worked, just like Ubuntu does. But Mint is absolutely gorgeous, even right out of the box.

The Aesthetical OS. Is't it funny though, how much opinions differ?
posted by Too-Ticky at 4:07 AM on August 9


People who work in technology are frequently more focused on how cool their achievements could be, and don't always think about misuse of their work.

Maybe. My experience with the tech crowd, though, leads me to believe that they actually do understand the potential for misuse of their works, but seem to reject the probability out of hand. Hubris?
posted by Thorzdad at 4:14 AM on August 9 [2 favorites]


I'll one up that Thorzdad: My experience is that leadership knows and understands certain evils, but through corporate speak we've eliminated dissension and whistleblowing through marketing schlock and 'tech evangelism.' In reality these issues are handled closer to 'No Snitching!' campaigns by gangs... We may be in a post-Snowden world, but Snowden released information about an *at least* 50% unpopular public big-bad. These companies are walled off privately with "rumors out of Redmond" or whatever leaking headline that prevents society at large from really being educated on the companies goals and ethics... You look at the scandal with Wells Fargo - they blamed it on software, they blamed it on a specifc few bad seeds, they blamed it on just a section... The only reason folks think anything more nefarious than that actually went on was because at some point *everybody* gets screwed by a bank - if only for $20 and a few late fees. But Apple? How many people actually realize that when they stop a production run for parts - even key ones like shitty 2002 PowerMac power supplies that product is a fucking obsolete paperweight. And google and facebook? people only started talking about *maybe giving them every bit of information isn't the safest plan ever...
posted by Nanukthedog at 5:04 AM on August 9


I went to log this in Pinboard and my helpful client, in amongst the previous tags that had been used by others for this URL, suggested 'technomessianism', which is both new to me and sounds right.
posted by davemee at 5:25 AM on August 9 [2 favorites]


Meanwhile, what’s the source of the post title up to these days?

Google plans return to China search market with censored app


They scratched the first word out of their motto a while back.
posted by duffell at 5:34 AM on August 9 [1 favorite]


dancestoblue: Then one day I tried linux Mint. Which, if I recall correctly, does branch off of Ubuntu. And it installed as easy as Ubuntu, and pretty much just worked, just like Ubuntu does. But Mint is absolutely gorgeous, even right out of the box.

The Aesthetical OS. Is't it funny though, how much opinions differ?

posted by Too-Ticky at 6:07 AM on August 9 [+] [!]
I went and read what those ppl wrote, and after careful consideration, I came away with the clear understanding that they are all completely wrong. And wrong-headed, too, and probably they have gas.

Ubuntu: you always know you are in Ubuntu. Open up a piece of software -- browser, word processor, whatever -- and Ubuntu is wrapped around it. And Ubuntu is ugly. Ubuntu won't play nice, it won't disappear so that I can use a piece of software and not have to know that I"m using Ubuntu.

Mint: Easily able to be customized to exactly what I want. People complaining about the wrong color scheme -- they've got to be on drugs. It's so easy to change colors in Mint. It's so easy to change anything in Mint. It doesn't force me to notice all day every day that I am in Mint -- Mint has manners, and will get the hell out of the way so I can dig into a browser or whatever else and not even know what OS I'm in, and not need to care. It's software that's actually soft, I don't need to be able to program in order to use the software. Spend a couple of hours getting it how you like it and then it gets out of your way.
posted by dancestoblue at 5:53 AM on August 9 [4 favorites]


Their Toolkit is so poorly formatted I can barely read it. It looks like a dump of a PowerPoint presentation, and it probably is one.

The label "PowerPoint Presentation" in the upper left and the title bar gave it away. Horrendous.
posted by lazaruslong at 6:13 AM on August 9


I'm staggered that nobody has yet mentioned that this is coming out of the Omidyar Network, and who Omidyar is. That would be Pierre Omidyar, the founder of eBay.

While I do appreciate that it's a good idea to suggest that these companies get with the ethics programme, I can't help but roll my eyes that this push is ultimately coming from that person's organisation. You really only have to have the most cursory knowledge of eBay's history with small-time sellers (and of course everyone's favourite monopoly friend, Paypal) to be giving this the side eye.
posted by Juso No Thankyou at 6:54 AM on August 9 [5 favorites]


eBay is no Amazon.
posted by infini at 8:42 AM on August 9


"Meanwhile, what’s the source of the post title up to these days?

Google plans return to China search market with censored app"

Well, ignoring the blatant absurdity of a business ever claiming to not do evil while still continuing to be a business -- you have to consider which is less evil, denying Chinese people a powerful tool to seek out information or providing the search engine with the legal restrictions of where it will be used. AFAIK, the results and pages will be censored even if Google went rogue and tried to make an uncensored one for China.

I won't say Google isn't evil (I believe capitalism to be fundamentally evil at it's core, which makes everyone involved in some way and degree also evil.* However, of evil things Google does notable enough to get a round of news about, the censorship thing seems a bit daft. It is unreasonable to expect businesses operating in the United States to do "good" for the sake of it, it is ridiculous to expect companies here to defy the government to do what they think is moral (if a company is defying the government, typically it means they can afford to break the law, pay the penalties, and still profit[wages stolen from the worker]) -- so I don't know what kind of fantastical expectations people have that Google would defy China about this.

* I also believe humans are born with great capacity for and natural tendency toward evils. I'm not sure what to call that besides the Christian analogue of "Original Sin"
posted by GoblinHoney at 11:24 AM on August 9 [2 favorites]


So I guess I have to explain the porn joke. Potter Stewart in one of the worst pieces of legal writing ever said that he'd "know it when he sees it" as if that's a useful piece of legal analysis. But there is a grain of truth buried in that asinine statement that we all live our lives by which is: "Only we can define what fundamentally offends us."

Stwart, if he was half as good a Justice as he claimed to be, would have understood his "I know it when I see it" for what it is: a statement that people interpret different speech different ways.

Now, does this mean I support legalized hate speech? Or harassment? It does not. But I do not think that a privately owned app is by any means the way to shield yourself from the worst the internet has to offer. I think that your average person is capable of figuring out their own mental and emotional health without silicon valley's help.

So yeah, I know both hate speech and porn when I see them.
posted by East14thTaco at 12:26 PM on August 9


people interpret different speech different ways.

It's ad hoc all the way down.
posted by eustacescrubb at 5:18 PM on August 9


Item: I installed Malwarebytes for my browser.

Item: Malwarebytes for my browser tried to scare me off The Daily Beast two days ago with a fullscreen “clickbait” warning overlaid on the main page. So much red and WARNING!

Really, though? It did that to me on a couple of left-leaning news sites. I’m now looking to replace this awful thing. What do they think I am, stupid or something?
posted by droplet at 7:09 PM on August 9


« Older One hundred years ago the tide finally turned   |   Kiddo, the first cat to attempt crossing the... Newer »


You are not currently logged in. Log in or create a new account to post comments.