Complex Systems Break in Complex Ways

August 1, 2015 6:01 AM

The RISKS Digest Turns 30: In February 1985 Adele Goldberg, the President of the Association for Computing Machinery (ACM), published a letter in the Communications of the ACM expressing concern with humanity’s “increasingly critical dependence on the use of computers” and the risks associated with complex computer and software systems. On August 1st 1985 Stanford Research Institute's Peter G. Neumann responded by creating RISKS@SRI-CRL.

"Its intent is to address issues involving risks to the public in the use of computers. As such, it is necessarily concerned with whether/how critical requirements for human safety, reliability, fault tolerance, security, privacy, integrity, and guaranteed service (among others) can be met (in some cases all at the same time), and how the attempted fulfillment or ignorance of those requirements may imply risks to the public. We will presumably explore both deficiencies in existing systems and techniques for developing better computer systems -- as well as the implications of using computer systems in highly critical environments.the collection, analysis, and discussion of computers"

For the last 30 years, Neumann and the subscribers of his digest have collected, archived and discussed reports of thousands of bugs, failures, amusing mishaps, major catastrophes, and other related issues as the modern computer age kicked into high gear and the internet took its first baby steps out into daily public use.

The list has witnessed historical events such as the Morris Worm, the Canter/Siegel Green Card Spam, the collapse of AT&T's telephone network, lost spacecraft, the Intel Pentium FDIV bug, the death of Knight Capital, and incidents that are now textbook reading for computer scientists such as the Therac-25 and the Mars Pathfinder priority inversion lockup .

Now in his eighties, Neumann still maintains the list and digest while continuing his research in computer security and reliability.

The entire RISKS archive continues to be available at http://catless.ncl.ac.uk/Risks/

posted by JoeZydeco (15 comments total)

As far back as Emerson we have been told that for every advance, every item of progress, there is a negative impact also involved. Hand driven equipment, replaced by steam driven, coal fueled, and now oil fueled. Horses replaced walking, and cars replaced horses, etc etc.
posted by Postroad at 6:27 AM on August 1, 2015

Is that's not true? Or are you saying we shouldn't pay attention to that aspect? Just looking to understand the reasoning behind your comment.
posted by kokaku at 6:40 AM on August 1, 2015

Merely noting that yes, I favor things getting better but am also trying to note that such advances also have a negative impact and those ought to be both recognized and perhaps, if it can be, dealt with. Example: people get cancer. They have always gotten cancer. We have made advances in detecting and in dealing with cancer. But dealing with it is expensive and that is an issue. And just perhaps our environment, and what we put into the environment might need consideration. Progress Yes. Recognize downside: Yes too.
posted by Postroad at 7:02 AM on August 1, 2015 [1 favorite]

That's what RISKS is all about.
posted by Obscure Reference at 7:16 AM on August 1, 2015 [1 favorite]

G-d, I'm old. The more things change...
posted by mikelieman at 7:34 AM on August 1, 2015 [2 favorites]

RISKS is one of the more important things which has kept me sane over the years.
posted by Bringer Tom at 7:45 AM on August 1, 2015 [3 favorites]

lobby groups trump risk assessments
posted by Postroad at 7:56 AM on August 1, 2015 [1 favorite]

RISKS has been one of the only unambiguously informative things I've found on the Internet since I subscribed in the 90s.
posted by kjs3 at 9:19 AM on August 1, 2015 [6 favorites]

Something recently (someone talking about disaster recovery, but meaning something very different) got me thinking about disaster recovery and similar problems and ended up with a couple of hours perusing the always fascinating and informative RISKS.

That prompted me to go ask a couple of people in my company - people in the right positions to ask - about our plans for disaster prevention and recovery. Came up with a couple of problem scenarios that were not awfully low probability but that would make quite a bit of difficulty for us. Their response was "Can't happen here, so not worth thinking about it." And none of them had even heard of the RISKS digest.

I thought a bit about regularly mailing out RISKS summaries as they happen to all and sundry, but that wouldn't do any good either.
posted by Death and Gravity at 9:47 AM on August 1, 2015

RISKs was the first (useful) public forum I contributed to.

It has also affected my career immensely by providing me with a solid cache of evidence to back up my cynicism about software design and development. It turns out that you can almost but not quite convince people to think ahead if you have good examples.

Most importantly though it has taught me to never ever take common sense for granted. You think people not backing up their theses is bad? Try a six person development team with 18 months of work who have never once done a backup. Or heavy duty industrial robots that have no hardware interdicts and rely on the software to get it right.

On the plus side it has made me much more accepting of movies were the evil plot has a hole that a five year old could drive a truck through. The Death Star designers probably *would* leave a proton torpedo sized hole direct to the reactor -- it's the kind of thing that happens all the time.

In any case I highly recommend RISKs if you have any faith in the engineers who designed the world you live in. It will cure you of that right quick.
posted by Tell Me No Lies at 11:32 AM on August 1, 2015 [10 favorites]

All their RSS and ATOM feeds show up blank for me.
posted by LogicalDash at 11:55 AM on August 1, 2015

"but if we allow code changes *directly* on the production server, we can skip the test/build steps and get those features out by cob. just this once."
posted by j_curiouser at 1:40 PM on August 1, 2015 [1 favorite]

here's a recent yt featuring neumann. aside: neumann's colleague at University of Newcastle upon Tyne, Lindsay Marshall, curates the excellent eclectica links collection at Bifurcated Rivets and manages the HTML RISKS site.
posted by j_curiouser at 1:50 PM on August 1, 2015 [1 favorite]

Fascinating resource. Thanks for sharing.
posted by amusebuche at 10:47 PM on August 1, 2015

I love RISKS and have been a regular reader since the 1990's. So good.
posted by rmd1023 at 10:07 AM on August 5, 2015