Join 3,497 readers in helping fund MetaFilter (Hide)


s/camel/_/
April 30, 2014 5:15 AM   Subscribe

Right now someone who works for Facebook is getting tens of thousands of error messages and frantically trying to find the problem before the whole charade collapses. There's a team at a Google office that hasn't slept in three days. Somewhere there's a database programmer surrounded by empty Mountain Dew bottles whose husband thinks she's dead. And if these people stop, the world burns. Most people don't even know what sysadmins do, but trust me, if they all took a lunch break at the same time they wouldn't make it to the deli before you ran out of bullets protecting your canned goods from roving bands of mutants.
posted by xqwzts (90 comments total) 61 users marked this as a favorite

 
Enlarging the text in Google Chrome 4.0.1847.131 on Windows 7 Professional Service Pack 1 causes the text to go out of the browser window on the left hand side.
posted by Brandon Blatcher at 5:28 AM on April 30 [8 favorites]


Have you tried turning it off and on again?
posted by sldownard at 5:28 AM on April 30 [18 favorites]


Relevant.
posted by Cash4Lead at 5:30 AM on April 30 [13 favorites]


Nobody's sure what Phil does, but it's definitely full of synergy and has to do with upper management,

Someone needs to see that Phil has an accident.
posted by GenjiandProust at 5:34 AM on April 30 [4 favorites]


Sounds like we need Simon and the PFY on the case to deal with these guys.
posted by Old'n'Busted at 5:37 AM on April 30 [7 favorites]


Also, the bridge was designed as a suspension bridge, but nobody actually knew how to build a suspension bridge, so they got halfway through it and then just added extra support columns to keep the thing standing, but they left the suspension cables because they're still sort of holding up parts of the bridge. Nobody knows which parts, but everybody's pretty sure they're important parts.

Sigh. Yep.
posted by escape from the potato planet at 5:41 AM on April 30 [15 favorites]


oh god this hurts me on a spiritual level
posted by specialagentwebb at 5:49 AM on April 30 [4 favorites]


I so miss the days of sysadmin worship, but it turns out most of what I did really could be automated or outsourced.
posted by gilrain at 5:49 AM on April 30 [4 favorites]


Why hello there tech burnout, my old friend. You're looking well.
posted by Spatch at 5:54 AM on April 30 [15 favorites]


Every time I encounter a fresh-faced developer passionately over-engineering a simple task because patterns!! I want to punch a penguin in the face.
posted by grumpybear69 at 5:56 AM on April 30 [16 favorites]


Mock Ye the Arcane Arts of Unix Server Administration!?

Read it in the voice of Doctor Orpheus.
posted by Pope Guilty at 6:01 AM on April 30 [7 favorites]


Also, the bridge was designed as a suspension bridge, but nobody actually knew how to build a suspension bridge, so they got halfway through it and then just added extra support columns to keep the thing standing, but they left the suspension cables because they're still sort of holding up parts of the bridge. Nobody knows which parts, but everybody's pretty sure they're important parts.

I'm not a programmer or a sysadmin, but currently my whole job involves working with a web-based application that was clearly built in exactly this fashion.
posted by dnash at 6:05 AM on April 30 [6 favorites]


And then, when you dicks and idiots have done your best at making something out of that cat urine soaked heap of snow, bailing twine and duct tape, some asshole too stupid to be a programmer but with an uncanny sense of knowning just where to prod to let the whole thing collapse in on itself, comes round to do just that.

They're going to tell you that you did something wrong and if they're in the right mood, they may even tell you where and how you did something wrong, but they can never tell you why nor help you solve.

Become a software tester. Make programmers cry.
posted by MartinWisse at 6:14 AM on April 30 [13 favorites]


To explain the joke in the title, and in the article:

s/camel/_ means:
search & replace CamelCase with an underscore.

Essentially the reply is that it's not called arrayReverse but array_reverse, which is helpful.

And yes, that's a very specific way to think.
posted by olya at 6:15 AM on April 30 [8 favorites]


a fresh-faced developer passionately over-engineering a simple task because patterns!!

::twitches::

That guy. The one that makes you want to invent a time machine and prevent his parents from ever getting together just so he won't be conceived, born, grow up (just a little) and then write that code.
posted by Foosnark at 6:16 AM on April 30 [1 favorite]


That's not really a joke, it's more of a dog turd hurled in your face
posted by thelonius at 6:18 AM on April 30


I'm a project manager for a big, complicated e-commerce, ticketing, and event management system and as I was reading this article, a senior member of my team just happened to stop by my desk to tell me that "[important bit of functionality] is broken because it was relying on a bug to work that's now been fixed."
posted by cilantro at 6:23 AM on April 30 [37 favorites]


I so miss the days of sysadmin worship, but it turns out most of what I did really could be automated or outsourced.
It always seemed like a dangerous job to have. As an in-house sysadmin, if you're constantly working on the company's network it means someone on your team is incompetent, so your job is at risk, whereas if you're surfing the internet while the stuff you've automated hums along untroubled it means you're a master, but you look like you're slacking off and your job is at risk.

Programming has its own risks, though. That first "bridge" design takes thousands of man-hours, but if you really understood the problem domain then every subsequent "bridge" can just use another copy of the first design with new parameters, so as soon as you finally master what you're working on then it's time to switch to something else. No wonder programmers like the article author feel like they're always patching together junk.
posted by roystgnr at 6:27 AM on April 30 [8 favorites]


Sales always makes ludacrious promises to get a client to purchase, even to the point of outright fabrications. Sales doesn't care that we've never even thought about that concept because they have their commission check and can move on to the next Candy and Sodas deal that they got fermenting.

Implementation managers over promise when something can go live so we're left scrambling and pulling hell marches to make sure we hit the deadlines so the company doesn't loose money in penalties.

Development never has enough time to build the requirement the right way because we're always 2 months behind on deliverables so it's "Slap whatever you can together so that we can move on to the next emergency".

Middle Management comes by with conflicting priorities and says "Stop whatever other projects you're working on and work on my request" despite the fact that all the middle management coordinated the priorities for the work period and the emergency request was not brought up.

Day long meetings in which the same items are reviewed over and over with no progress to report.

Time to purge my brain with some ethanol...
posted by Hasteur at 6:28 AM on April 30 [11 favorites]


Actually, no. Things really aren't that brittle. It's just when they do go pear-shaped, they go pear-shaped at inconvenient times in surprising ways.

Sysadmins are alternately bores to tears or terrified. Like tech support folks.
posted by clvrmnky at 6:37 AM on April 30 [1 favorite]


A friend of mine once did sysadmin-style work for a hedge fund's trading systems - as he put it, 'keeping an eye on the machine that prints money'. He said it was the best job he ever had, because since the systems were responsible for generating that much cash they were well-engineered and tended not to break too badly too often, but since they generated SO MUCH cash the firm didn't mind paying a bunch of people a salary to mostly sit around and read blogs, just in case something DID go wrong.
posted by Itaxpica at 6:42 AM on April 30 [3 favorites]


I empathize with TFA almost 100%, and yet yesterday I sat down to work on a reported bug in some code I wrote about a year ago: somebody noticed a couple of cases out of a few hundred where they weren't getting the expected output. I dug through the code to get to the relevant bit, almost immediately spotted the glaring oversight plus a couple more instances of stupidity. I cleaned everything up, moved some stuff around for general sanity, compiled it and did some very basic testing. Then I went through it again, simplified and removed almost half of the code, and everything was Right and Beautiful, and I experienced a moment of bliss.

Of course now some poor bastard must test this rigorously, and make sure I haven't fucked up behavior for the other 99% of cases - but the point is not all of a programmer's work is grief.
posted by Dr Dracator at 6:42 AM on April 30 [5 favorites]


Thanks for this. I work in a job where I teach 170 teenagers how to write more wonderfully and read more closely. I am over 60. I stopped being able to help my daughter with math homework in about 5th grade.

Obviously, I have had no idea whatsoever what this job (these jobs) is like. I like the attempts at describing what happens to the human mind in this hellish occupation. Is it really this bad?
posted by kozad at 6:47 AM on April 30 [3 favorites]


Fuckin' synchronous.

I was going to mention "Nice try, I know all about teh Google Ultrons" and that thread so I know what sysadmins *really* do...

But when I go to link to imgur to get the thread image, I get...

"Imgur is over capacity!

Sorry! We're busy running around with our hair on fire because Imgur is over capacity! This can happen when the site is under a very heavy load, or while we're doing maintenance.

Please try again in a few minutes."

Here we go: 1
Here we go: 2
posted by symbioid at 6:51 AM on April 30 [5 favorites]


It is not quite this bad, but it is very close, and really does drive you crazy in particular ways that are very hard to describe. Reading it, I realized I have a bit of stockholm syndrome; my response was that of course every piece of software is shitty, why would you expect anything else? The trick is accepting that and still trying to write anything good ever.
posted by contrarian at 6:52 AM on April 30 [3 favorites]


Kozad, it's like having a brain full of the most beautiful prose, visualizing how it could flow onto the page and tell wonderful stories that would make people happy, and being forced instead to write apartment listings for other people's visions -- listings where you are required to use clichés and abbreviate everything because apparently if you write out "hardwood floors" the universe will collapse because of the column-width restrictions set in formatting newspapers in the late 1800s, despite the unacknowledged but globally understood truth that this is going to be electronically published on a web page where the print is justified and will just flow to the next line. Also they make you use papyrus because some manager thought it looks like the next new thing. If it wasn't for the brain full of beautiful prose, you'd hang it all up because it's not worth the pain, but you live in hope that someday, your day will come!
posted by sldownard at 6:57 AM on April 30 [21 favorites]


contrarian, I (sysadmin) feel the same way. People always ask me what the best OS is, but I typically respond, "all operating systems suck. The trick is finding the one that sucks the least for your application." Then they stare at me like *I'm* the weirdo...
posted by sldownard at 6:59 AM on April 30 [7 favorites]


Also, they tend to pay you quite a lot of money, so that helps.
posted by Itaxpica at 6:59 AM on April 30 [6 favorites]


Yeah, coding and sysadmining is hard work and forced autistic and psychotic mind states, etc.

But I'm giving less and less of a fuck about the whining of the new gilded age technocracy. Try hard labor without enough food or sleep and see what that does for your sanity. No, that AWS service or that shopping cart for that bespoke widget store likely isn't actually life critical. No, people aren't likely to die or starve if iCloud or Siri goes down.

Sure, you might lose your job because you lost $TechCorp a zillion dollars, or you leaked a million credit card numbers or unsalted, unhashed passwords, and it might even kill the company dead.

Sure, some people work with technology that is life threatening. Your average web dev and sysadmin doesn't.

Sure, the sales department at $TechCorp might be clownish morons that promised a supersonic jetliner when the client can barely afford and operate a tow-line autogyro, but there are plenty of devs that muck things up with creeping features, ill advised attempts at cleverness and trying to be insanely great, and it's often their own damn fault because - as Oppenheimer put it, paraphrasing - the mad scientist isn't just a cliche, and they're willing to play dice just to see if something theoretical will work.

Try getting fired from flipping burgers because you called in sick to work and shouldn't handle food, or getting a limb torn off because someone zigged instead of zagged with an earthmover the size of a small apartment block or busting concrete by hand with a maul and prybar. You still have to think there, too, and if you fuck up you lose limbs or actually kill people.

I've seen both sides if this fence. Yep, being hardwired into baby sitting a bunch of mentally challenged machines and pulling three days without sleep sucks and it's stressful. Physical and menial labor is still worse, at extreme fractions of the pay, benefits and job stability.
posted by loquacious at 7:01 AM on April 30 [49 favorites]


Last night at 11:30PM I remembered that a manual process -- which we could automate, but for some people who insist on being asked for their approval every damn day -- hadn't been manually run.

I knew that I could run it this morning and no one would really know or care, but I just…. couldn't leave it undone. So I got out of bed, went downstairs, opened the laptop, VPN'ed in, opened a terminal window, logged in, ran a shell script, and then closed everything down.

Damn, do I need to get those people out of the process and get that script automated!
posted by wenestvedt at 7:04 AM on April 30 [3 favorites]


Damn, do I need to get those people out of the process and get that script automated!

Assuming they give their approval every day anyway, could you just cron it and not tell them?
posted by Itaxpica at 7:13 AM on April 30 [2 favorites]


I really love the test bed project that suddenly becomes mission critical and gets scaled to fucking enormous without ever going back to the design phases to make sure that the infrastructure is actually resourced correctly so instead you get a horrible mish-mash of architectures that were purchased because OMG we need more high speed disks right now or sure let's just clone a bunch of app servers on the VMs and hope that the network guys somehow provisioned enough bandwidth so that the retarded traffic flows don't bring the entire data center to it's knees.

But god forbid you size something for growth at the start of a project because that's just wasting resources and we really need that architect or sysadmin guru or heavy duty programmer to move on to the next critical project and thus we are going to move this into maintenance mode with a junior admin that doesn't understand any of the architecture but knows how to follow some vague instructions written down in bad handwriting by the architect during some meeting. Of course he develops some bizarre rituals that resemble superstitions in an attempt to fend off the day that the whole house of cards collapses and someone is forced to actually come in and fix shit because god knows nothing is adequately documented and the whole thing is a frankenstein.
posted by vuron at 7:16 AM on April 30 [3 favorites]


Very fucking relevant.
posted by ThatFuzzyBastard at 7:18 AM on April 30 [3 favorites]


Yep, being hardwired into baby sitting a bunch of mentally challenged machines and pulling three days without sleep sucks and it's stressful. Physical and menial labor is still worse, at extreme fractions of the pay, benefits and job stability.

Agreed, but: we all know that manual labor and food service and whatnot suck.

Most people who don't have tech jobs don't really grasp the insanity of what we do. They think we are magical wizard genius mathematician-linguist electronics gurus who are smart enough to command dead silicon to do our bidding. They don't know we're regular people doing stupid things because we (A) don't really know what we're doing half the time and (B) our bosses want things that make no damned sense and so we commit all kinds of evil and insanity in the line of duty, which fellow tech people will regret later.
posted by Foosnark at 7:20 AM on April 30 [15 favorites]


(And honestly, half the "we don't really know what we're doing" goes back to management having plumbers do carpentry work with a "it's all basically the same" and "how hard could it be?" attitude.)
posted by Foosnark at 7:22 AM on April 30 [5 favorites]


I just realised I do actually have that bit of code tucked away on my machine. It doesn't do much - recursive depth-first tree traversal - but it's a suspension bridge in comparison to the beaver dams I build in my day job.

I'd never write that rant though, because I'm a good ten years beyond burnout and well into BOFH-hood. Instead of tech enthusiasm these days I amuse myself by dangling bad ideas like kitten toys in front of grumpybear69's scrub-faced youths. Because, on a very fundamental level, none of this matters. Like loquacious says, nobody's going to live or die because someone clicked on an ad.

The whole industry's a shell game. Distraction and shiny, layer upon layer of abstraction, old ideas in new packaging, conference circuit, massive quantities of bullshit, and all we do is glue together some pre-existing components and paint 'em.

Fuck it, I'm gonna go dig the garden over.
posted by Leon at 7:25 AM on April 30 [8 favorites]


"could you just cron it and not tell them?"
Oh, wouldn't THAT be great? But we are not allowed to use cron, so automated processes have to be scheduled through another contractor, who will "onboard" the job and then send you an email when it doesn't run because the password expired. So instead of using the built-in, simple, and elegant solution for recurring jobs, we use the external, complicated, and kludgy replacement. Progress on All Fronts!
posted by joecacti at 7:25 AM on April 30 [4 favorites]


Wait what you aren't allowed to use cron in favor of some other likely homegrown job management software that is probably poorly maintained but somehow incumbent because some decision-maker like 2-3 CIOs ago decided that it was a good product and now you can't get rid of it even though that person left the company like 10 years ago?

I have no idea what you are talking about.
posted by vuron at 7:31 AM on April 30 [11 favorites]


Wait what you aren't allowed to use cron in favor of some other likely homegrown job management software that is probably poorly maintained but somehow incumbent because some decision-maker like 2-3 CIOs ago decided that it was a good product and now you can't get rid of it even though that person left the company like 10 years ago?

Show of hands. Who's seen Jenkins used as a replacement for cron in a production system?

Now... who's suggested it?
posted by Leon at 7:33 AM on April 30 [13 favorites]


Foosnark: agreed. I've been on both sides of the fence, which includes IT support for users and machines.

I just don't agree with the self important apocalyptic zombie scenario. A vast majority of devs and sysadmins could vanish and it wouldn't be a catastrophe. The world would be a less cool and interesting place, sure, but losing stuff like Flappy Birds, Facebook, the iTunes Store, YouTube, Reddit or even MetaFilter for hours or days or even weeks isn't going to lead to a systemic collapse.

Carrier grade telecom? Major civil banking? Embedded industrials and power? Sure.

Amazon Fresh? No.

Hell, losing a lot of the algorithmic conveniences we use today might actually be good for the economy as a whole, just not the technocracy, and more to the point, their managers and the $TechCorp leveraging said devs for maximal profiteering.
posted by loquacious at 7:33 AM on April 30 [2 favorites]


the systems were responsible for generating that much cash they were well-engineered and tended not to break too badly too often, but since they generated SO MUCH cash the firm didn't mind paying a bunch of people a salary to mostly sit around and read blogs, just in case something DID go wrong.

which is to say that this has nothing to do with computers, programming, or engineering but economics. Now, extrapolate it to the rest of our crazy eff'd up world and ask yourself if really things have to be this way.
posted by ennui.bz at 7:40 AM on April 30 [4 favorites]


I work at a large tech company which has been around since the mid 80s. A couple of months ago, we had an issue with a minor billing system modification. The file directory in question indicated that it had existed since sometime prior to 1989. A had to call the engineer in question at 2am. The man is in his 60s and, I shit you not was near tears, as he had to explain AGAIN to someone who had never seen the system in question, and which he had programmed over 20 years ago, how to fix an utterly trivial problem that should have been patched sometime in the early 90s. His plaintive wail: "Why won't they just let me retire?" will haunt me until the end of my days.
posted by empath at 7:44 AM on April 30 [53 favorites]


grumpybear69: "Every time I encounter a fresh-faced developer passionately over-engineering a simple task because patterns!! I want to punch a penguin in the face."

You Have Ruined Javascript

There are seriously people in the reddit thread in which I found this who insisted that it is only right and natural that a JS framework use design patterns and methodologies of Java.

Goddamned Corporate Java bastards. Wait, this is about sysadmins, not languages.
posted by symbioid at 7:45 AM on April 30 [3 favorites]


As someone now doing tech support for phones and streaming media devices all from the same monolithic ad and tech company, the phrases that hurt most are:

"Why is this so difficult", usually paired with "I don't know (much|anything) about computers". The attitude of it-should-just-work is endemic for these people. They willfully try not to understand how these devices get operated in the most basic ways (I had to tell someone how to find the Start button on Win 7, for the love of the Light), and then complain when they need to do something more than one click.

And somehow this becomes my personal fault; that I must ask them questions about their computer and they don't like having to answer the question and actually try to do something on their side. I was actually told that the Windows XP retirement was Obama's fault to bail out Microsoft.

This is my day to day: people refusing to learn the basics of operating a complex machine and being angry about it. I fear their driving and pray the people who fix their cars have good scotch.

(Oh, and I found out the company I work for was told by the company that we are supporting as an outsourced support line that paying us the original rate was no good and we're making more as a result. Yah, a company said that the outsourced call center phone mooks we're getting underpaid and would pay more. Weird.)
posted by mephron at 7:49 AM on April 30 [3 favorites]


Oh, wouldn't THAT be great? But we are not allowed to use cron, so automated processes have to be scheduled through another contractor, who will "onboard" the job and then send you an email when it doesn't run because the password expired. So instead of using the built-in, simple, and elegant solution for recurring jobs, we use the external, complicated, and kludgy replacement. Progress on All Fronts!

I am about to suggest something which is probably almost as awful, but you could always schedule a cron job on a server you control that remotely logs into said machine and runs the script.
posted by empath at 7:52 AM on April 30 [6 favorites]


Empath, if your company is smart they'll hook up said developer to life support machines pre-emptively so that instead of "why won't they just let me retire?" it can be "why won't they just let me die?". I mean keeping someone on 24/7 life support might still be cheaper than patching or replacing the code.
posted by vuron at 7:55 AM on April 30 [5 favorites]


Oh, I'm sure they've considered it.
posted by empath at 7:58 AM on April 30 [2 favorites]


Damn, do I need to get those people out of the process and get that script automated!

If they need to approve the running of the script, could they run the script themselves? Another approach is to build a dashboard or some interface where they push the button to approve (log this, CYA y'know) and then your automated thing checks periodically for the approval flag? This gets you out of the business of owning the button press and back into the business of automating stuff.
posted by dgran at 7:59 AM on April 30 [4 favorites]


The attitude of it-should-just-work is endemic for these people.

You're wrong. It should be that simple. Every time a user performs an operation and doesn't get the result they expected, we've failed, not them. It's called the principle of least astonishment, and it should be core to the worldview of anyone who writes software for humans.

I'll give you an example. On earlier versions of Windows if the user click-dragged an item in the task bar and dropped it somewhere else, the icons didn't rearrange themselves. That's a failure of the software, not the user. The Start icon isn't necessary. Your phone doesn't have one.

"Technology always develops from the primitive via the complex to the simple" - Antoine de Saint-Exupéry (if anyone knows exactly which book that quote came from, I'd love to know, 'cos I just stole it off the internet).
posted by Leon at 8:00 AM on April 30 [14 favorites]


Okay people, let's have a stand-up in the Zuckerberg room at 11 and whiteboard out a workflow for joecacti. I'm sure we can engineer a solution with a quick scrum.
posted by empath at 8:01 AM on April 30 [22 favorites]


I so miss the days of sysadmin worship, but it turns out most of what I did really could be automated or outsourced.

It depends. If you're working for someone like Facebook where you have a billion nodes doing one of four things (Hadoop, memcached, mysql, Cassandra) it becomes trivial to spin up a new standardized image doing what you really want it to do and add it to the pile. Then yeah, you can get hundreds if not thousands of servers per admin.

Working as a sysadmin for a company where you have a diverse set of services running across different physical and virtual servers running on multiple platforms? Not so bueno for one sysadmin to run a hundred servers. Even a dozen would push it.
posted by Talez at 8:04 AM on April 30 [3 favorites]


Also, this Mickens article is depressingly relevant here. It's long, but totally worth it (especially if you've never read Mickens before)

https://www.usenix.org/system/files/1403_02-08_mickens.pdf
posted by Itaxpica at 8:11 AM on April 30 [2 favorites]


if the user click-dragged an item in the task bar and dropped it somewhere else, the icons didn't rearrange themselves.

LOCK THE TASK BAR
LOCK THE TASK BAR
posted by thelonius at 8:12 AM on April 30 [8 favorites]


.Working as a sysadmin for a company where you have a diverse set of services running across different physical and virtual servers running on multiple platforms? Not so bueno for one sysadmin to run a hundred servers. Even a dozen would push it

Yeah, the difficulty has more to do with the different 'kinds' of servers. One guy could easily manage hundreds of webservers running the same stack, even if the actual apps running on the servers are vastly different from each other -- apache is apache, and if you know how to read the logs, you're pretty much sorted for 90% of issues, and you can always page the developer if you need backup.

Running a lot of medium-sized office setups is actually a lot harder, IMO, just because the breadth of knowledge you need to know is so much larger.
posted by empath at 8:13 AM on April 30 [1 favorite]


I'll give you an example.

I'll give you another one: "it says to click this link to start the setup. Do I need to do that?"

I do not lie that I have been asked that.
posted by mephron at 8:13 AM on April 30 [2 favorites]


I have made food grow out of the earth, I have slaughtered chickens and pigs, I have even brought babies back to life as an EMT.

Right now I am part of a Dev Ops team, which is just a fancy kind of sys admin. This work is exciting and fun, and I kind of have a knack for it.

Practically, though, my daily life is possible because hundreds of thousands of people I will never meet gave a damn. From the factory workers who made these wonderful machines to the construction workers who are improving the sewer outside my house right now, everything is connected and everything is vital.

The feeling that your work is important is absolutely true, but each person who enjoys the product of your labor and all the labor around yours cannot truly appreciate it because there would be no room left for doing their own.

Though there are challenges unique to each job and little failures and troubles every day, the best days feature a sense that you have made something worthwhile happen.
posted by poe at 8:14 AM on April 30 [10 favorites]


mephron: Ask yourself where that user's fear is coming from.

But also, you're the expert on the end of the telephone. Of course they're going to run everything by you before they do it.
posted by Leon at 8:15 AM on April 30 [1 favorite]


Also, this Mickens article is depressingly relevant here. It's long, but totally worth it (especially if you've never read Mickens before)

I always think of James Mickens when I read these essays from burned out programmers that cloak the absurdities and frustrations of our profession in metaphors of unicorns, zombie apocalypses and rivers of blood. I first encountered this style when reading his usenix columns, but have since seen the format multiply and flourish. When done well, it always seems like a bit of a non-sequitur until the author gets to their point and makes us realize that he's succeeded in getting both the engineer and the layman to gaze upon the countenance of Cthulhu together and recognize the madness in our shared existence using rampaging mutant bears as the bridge for our common understanding.

Which is to say, +1 to that article.
posted by bl1nk at 8:18 AM on April 30 [3 favorites]


"Hey, kaib, whatcha up to there?"

"just hanging out, listening to the screeching of the old ones."
posted by kaibutsu at 8:24 AM on April 30 [5 favorites]


Also, the bridge was designed as a suspension bridge, but nobody actually knew how to build a suspension bridge, so they got halfway through it and then just added extra support columns to keep the thing standing, but they left the suspension cables because they're still sort of holding up parts of the bridge. Nobody knows which parts, but everybody's pretty sure they're important parts.

Sigh. Yep.


Ah, I see that you've witnesed The Margaret Hunt Hill Bridge in person...

Sorry, being a snarky Dallas-ite
posted by Hasteur at 8:25 AM on April 30


ennui.bz: "the systems were responsible for generating that much cash they were well-engineered and tended not to break too badly too often, but since they generated SO MUCH cash the firm didn't mind paying a bunch of people a salary to mostly sit around and read blogs, just in case something DID go wrong.

which is to say that this has nothing to do with computers, programming, or engineering but economics. Now, extrapolate it to the rest of our crazy eff'd up world and ask yourself if really things have to be this way.
"

OMFG THIS!

I always say society is held together by Duct Tape. It took me quite a few years to understand this truism. But it's true. And a lot lot lot of it is due to the need to push out ever more for ever less, more work, less oversight, less quality, just to compete to get the lowest amount of money on a contract, but more and more done. Add in new requirements (from clients, from government, from consumer demands, etc....), and it's seriously a wonder civilization ever gets a damn move ahead at all without completely self destructing.

I take it to say that it shows the good nature of the majority of us. We push ourselves and do what has to be done so it doesn't completely 100% fuck up, but it's always the bare minimum, because to do an actual good job means we'd have to have the time and pay to do it well, and jesus fuck... And yeah, maybe it's not important, like loquacious says, in most cases, but the fact is, all these systems are interrelated, and a lot of people do depend on them, whether we like it or not.

If we were truly all greedy bastards, we'd start to fuck shit up and make a point, but we shuffle along, with a few brave souls trying to make a point or change for a small segment of a system that is overall steered towards this mode of production... until the impending societal collapse of all humanity, but hey...

at least we had flappy bird and candy crush saga and farmville to while away the time!
posted by symbioid at 8:34 AM on April 30 [4 favorites]


Leon: the person called to ask that question. That was the entire point of the call, to ask if they should follow the directions on the screen. How do I accept that as reasonable?
posted by mephron at 8:50 AM on April 30 [2 favorites]


I'm a network engineer. One of the ways I describe my job is that if I do my job well, most people have no idea what I do, and if I do my job badly, nobody can do any work.

(Since I currently work for a university, I also explain my job as "I make sure the kids can get to Facebook.")

In a previous job, I worked on failures that resulted in newspaper stories on page 1 above the fold. And even then, I kept in mind that nobody is dying because of any of my issues. I briefly had a coworker years ago who decided that the company we were with was too high stress and so he went to work for an HMO. And all I could think was "Dude, seriously? That's potentially dealing with actual people being actually sick!" So I guess that what people stress out about is different for different people.
posted by rmd1023 at 8:51 AM on April 30 [1 favorite]


Symbioid: so basically, no matter what anyone claims we who work in IT aren't going to follow a John Galt.

Rational.
posted by mephron at 8:51 AM on April 30 [1 favorite]


Well, that's certainly a nice way to spin it, mephron :)
posted by symbioid at 8:58 AM on April 30


mephron: "Leon: the person called to ask that question. That was the entire point of the call, to ask if they should follow the directions on the screen. How do I accept that as reasonable?"

Dear User:

This is your weekly reminder: DO NOT CLICK ON LINKS! THIS IS HOW WE GET VIRUSES, TROJANS, AND ANTS!

Signed,
Your Friendly IT Security Manager
posted by pwnguin at 9:07 AM on April 30 [16 favorites]


@mephron: Users are essentially trained to feel stupid and doubt themselves even in what should be the most unambiguous scenarios, such as yours. Certainly there are those who actively refuse to learn, but for the majority it is a damned-if-you-do, damned-if-you-don't situation. If they don't ask you for advice and something goes wrong they will incur the wrath of IT for being a moron. If they do ask you for advice they get an exasperated "THAT IS SO OBVIOUS YOU MORON" response. The relationship is fundamentally antagonistic..
posted by grumpybear69 at 9:13 AM on April 30 [4 favorites]


empath: Do I know you? Your knowledge of our SDLC is... unsettling... although I guess the scales have tilted and I would wager 80% of the industry would say they are using "Agile" now, MS Project usage notwithstanding.
posted by joecacti at 9:18 AM on April 30 [1 favorite]


Every time I encounter a fresh-faced developer passionately over-engineering a simple task because patterns!! I want to punch a penguin in the face.

I believe you first need to instantiate a PenguinPunchFactory and call getPenguin().

PunchService and FistProvider are left as exercises for the reader.
posted by Mr. Bad Example at 9:19 AM on April 30 [10 favorites]


But I'm giving less and less of a fuck about the whining of the new gilded age technocracy. Try hard labor without enough food or sleep and see what that does for your sanity. No, that AWS service or that shopping cart for that bespoke widget store likely isn't actually life critical. No, people aren't likely to die or starve if iCloud or Siri goes down.

Hi. I design, build and maintain security infrastructure for 911 call centers, hospitals, banks and retirement investment firms. People generally are going to die or starve if my systems go down.

But, man. That "technocracy" you're sneering at is the last vestige of the middle-class, being pinched thin and bled white. They're telling you that worker protections simply don't apply to anyone - the 40 hour work week, paid time off, job security, reasonable workloads because the work is shared among enough people, predictable wage increases for seniority, all gone. This is a severe problem affecting everyone outside the management class, and not just the nerd-babies nannying along the linux cluster.

Worse, much worse, is the steady drumbeat of resentment fanned by the monied elite regarding this "technocracy" - they don't deserve to make a decent living. They're coddled and lazy and make too much money - they need to be poor like everyone else. They're spoiled, soft cogs, replaceable with the right outsourcing firm or the right automation tool, they should watch their step.

In typical Gilded Age 2.0 fashion, all you can do is snarl at them for being all middle-class. As if working hard doesn't entitle them to a decent living, and they should all snivel in the dirt with the rest of the gutted and hamstrung working class, grateful for whatever scraps the bosses fling their way.

Mind your own business, and I mean that literally, and demand your own fair share from the bosses, not the other grunts. We don't have the power to give or take it from you. You're being ripped off, we're all being ripped off, and it's not my fault. Organize or die.
posted by Slap*Happy at 9:28 AM on April 30 [87 favorites]


Every time I encounter a fresh-faced developer passionately over-engineering a simple task because patterns!! I want to punch a penguin in the face.

Honestly, I will take that young man or woman over the alternative, which is an EE major who thinks "pattern" refers to fabric and just cannot countenance the idea that code should look like anything other than the cheat sheet you bring with you to a differential equations exam. The best way I can think of to describe his code is in terms of a trailer for a movie I haven't seen, World War Z. That part where zombies are just mindlessly crawling over and on top each other in single-minded pursuit of the humans on the other side of the wall, and they end up scaling the wall not as a result of any sort of conscious collective action but just out of a sheer lack of qualms about the damage they'll do themselves and everything around them in pressing forth, and it strikes you that it must take like a hundred zombies piling up to get even one over the wall whereas a single human with a ladder would have to sacrifice no one to get over the wall herself? That's what this person's code was hypothetically like, since I definitely have not ever worked with or had to suffer for years through the machinations left behind by them.
posted by invitapriore at 9:29 AM on April 30 [6 favorites]


Working as a sysadmin for a company where you have a diverse set of services running across different physical and virtual servers running on multiple platforms? Not so bueno for one sysadmin to run a hundred servers. Even a dozen would push it

In this AWS/OpenStack age, servers are cattle, not pets. They don't have quirky names or artisanally hand-edited configurations (with the possible exception of a box kept for experimenting and never used for production), are stamped out from standardised settings, and created and destroyed like they're going out of style.

Where I am now, there are vast flotillas of AWS servers created this way. Each does one thing and (with few exceptions) keeps no state locally. They don't even bother to make them come up on reboot; the applications on them are started in the AWS setup scripts, and the virtual server will only ever be booted once in its lifetime. Mind you, we are a functional programming shop, so maybe that comes from thinking of immutable objects and translating that into the context of infrastructure.
posted by acb at 9:46 AM on April 30 [4 favorites]


If they need to approve the running of the script, could they run the script themselves?

Horrors, NO! Look, I have to keep the lid on my laundry basket, so I will just intone, hollowly: NO LOGIN FOR THEM.
posted by wenestvedt at 9:55 AM on April 30 [1 favorite]


I just don't agree with the self important apocalyptic zombie scenario. A vast majority of devs and sysadmins could vanish and it wouldn't be a catastrophe. The world would be a less cool and interesting place, sure, but losing stuff like Flappy Birds, Facebook, the iTunes Store, YouTube, Reddit or even MetaFilter for hours or days or even weeks isn't going to lead to a systemic collapse.

Woah woah woah.

I can attest, that incidental fallout from the Planetside 2 servers being down for 26 hours while SOE bandaged up some issues did was that when I was finally ready to take a break from work, and relieve some stress and I couldin't. That was about as catastrophic as I could imagine, because that two hours of stress relief that I was counting on last night was instead replaced with me modifying code and writing presentations for 4 hours until I fell asleep in a pile of drool on the couch instead of my bed... for the third night in a row. At this point I'm sort of a shell of a person, having coded, slept, ran stuff through excel, made slides for a presentation, and repeated that cycle for... well... wait... what day is it?
posted by Nanukthedog at 10:14 AM on April 30 [1 favorite]


I find that more and more often, I'm that guy in that story who only works in wood. Because wood can do anything and you don't have to mess with welders and solder and there's a clean elegance to it or worry about alloys and corrosion and yeah sometimes termites and okay this analogy's really stretching.

But yes, this hit a little too close to home.
posted by Hamusutaa at 11:37 AM on April 30


Thank you people of metafilter for sharing and commenting upon this link. It is having the effect of talking me down from the ledge as it were today.

carry on
posted by garethspor at 1:51 PM on April 30 [1 favorite]


Also, the bridge was designed as a suspension bridge, but nobody actually knew how to build a suspension bridge, so they got halfway through it and then just added extra support columns to keep the thing standing, but they left the suspension cables because they're still sort of holding up parts of the bridge. Nobody knows which parts, but everybody's pretty sure they're important parts.


Oh come on the new bay bridge isn't THAT bad.
posted by kenko at 2:23 PM on April 30 [1 favorite]


This is a world where people eschew sex to write a programming language for orangutans.

I ruptured something laughing at this.

I'm a sales engineer. I'm convinced that we are the Referee Class that tries to protect Sales from themselves and the coders from everyone else.
posted by Thistledown at 2:33 PM on April 30 [7 favorites]


In this AWS/OpenStack age, servers are cattle, not pets.

Fortunate you are, contentedly looking across the herds grazing on your ranch.

Let me tell you what it's like on the range I ride. The settlement has been here a while, but it wasn't exactly planned...

Prospectors and missionaries came. They built according to their desires, each in their own way with whatever was at hand. The followers of the mainframe constructed a temple. Later the workstations arrived, followed by the desktops. They fought a bitter feud that left resentments to this day. The Wins and the Macs are here, in an uneasy truce. There's even a tribe of Sun-worshipers up in the hills. Sure, they speak Unix - but they don't like your accent and your fancy ps options. Kill -9 you as soon as look at ya.

All of them have their turf, and they'll defend it. Try grazing your cattle where their pets roam, you'll regret it. You have to learn to work with them all, care for their servers as you can. You see ones that should be put out of their misery, but you never know who's going to come howling for your head if you do. You'd have done things differently if you were here from the start, for sure.

It's not a bad life, keeps you on your toes. Feels good when you bring a new system into the world, or rescue a user being attacked by bandits. One more day of holding chaos at bay. I think I'd get bored just watching the cattle graze.
posted by bitmage at 5:22 PM on April 30 [18 favorites]


I dunno, in Amazon country you're free to experiment with things like alternative stomach arrangements and even composite cows, where the udders and the hooves and the mouth and the tail don't even need to be a part of the same animal. Not to mention that if you set things up right you can essentially will fully-grown cows into existence if the demand for milk suddenly spikes, and then vaporize them instantly when demand wanes again.

I feel like we can really milk this analogy.
posted by invitapriore at 5:34 PM on April 30 [5 favorites]


Looks like I picked the wrong week to quit smoking.
posted by Poldo at 6:23 PM on April 30


I dunno, in Amazon country you're free to experiment with things like alternative stomach arrangements and even composite cows...

No, AWS isn't that flexible, which is rapidly becoming an issue as Linux distros and Beastie flavors all diverge everywhere, some of them becoming very attractive as special purpose platforms. Also, more and more folks are noticing that even as their pricing goes down, costs continue to go up.

With the advent of stuff like SmartOS, Xen and LWX, indie cloud vendors and DIY clouds are a thing. It used to be small servers straight on a leased line, then it was big shared web hosts, then it was dedicated hosts in a CoLo, then it was virtualized hosts in the colo, then it was virtualized hosts as a service, now it's virtualized hosts as a service in a dedicated infrastructure. Eat up, drink up, tomorrow we may die we'll all be running serverless internet apps.
posted by Slap*Happy at 7:18 PM on April 30 [1 favorite]


In this AWS/OpenStack age, servers are cattle, not pets.

Yeah, but when you are held in thrall to slothful or callous vendors and capricious users, you can find that you end up with an archipelago of systems where the supportability of the software drives the versions of the underlying stack, and each application's island much be kept apart from the others -- sometimes by sharks or angry islanders in canoes with poisoned darts.

In organizations that do One Thing (and out-host a lot of the rest), then cookie-cutter hosts can be extruded whenever you like; when you're in a place that has lots of different software, then often you must cast aside those osft dreams of uniformity and just keep the goddamn lights on.
posted by wenestvedt at 10:07 AM on May 1 [3 favorites]


LOCK THE TASK BAR
LOCK THE TASK BAR

Dude, is there some clever Clash parody I'm missing out on? Because that has become a virtual earworm that demands attention.
posted by drowsy at 4:15 PM on May 1 [5 favorites]


The luuusers don't like it
LOCK THE TASK BAR
LOCK THE TASK BAR
posted by MartinWisse at 3:40 AM on May 2 [3 favorites]


Every time I encounter a fresh-faced developer passionately over-engineering a simple task because patterns!! I want to punch a penguin in the face.

I see this attitude a lot these days and I just want to unpack some of the assumptions baked into this:

a. I say this a lot but: There is no such thing as a simple project. I hear it time and again, "this is just a simple project, we don't need tests or anything". Fast forward 3 months and you're supporting a complex system without any automated tests. Yay!

b) Practices like design patterns, test-driven development aren't fancy extras, they're the bare minimum you should do. They have constantly justified been argued against and justified over the years and the reason they're still here is because the save time, even for mythical "simple programming tasks:.

c) Punching penguins is animal cruelty, you monster.
posted by axon at 5:43 AM on May 2


wenestvedt: "Damn, do I need to get those people out of the process and get that script automated!"

I don't know if I'm comfortable taking the men out of the silos.
posted by Chrysostom at 7:59 AM on May 2


It's not a simple matter of man versus machine - every system breaks down: the hard part is making it fail gracefully.
posted by Dr Dracator at 11:22 PM on May 2


Organize or die.

I posted about that the Apple/Google non-recruitment settlement was another sign that a *lot* more Americans needed unions. Even if the coddled techies can't get a fair shake, no one has any chance at all. it touched off quite a few comments about how Unions are horrible and they only let horrible people keep jobs. And how collective bargaining would make Silicon Valley disappear. It made me sad. Intelligent people I like cannot seem to realize the value of a million interconnected people is more than a million standing alone. Ya' know, like the computers.
posted by DigDoug at 5:09 AM on May 6 [3 favorites]


Unions need some serious rebranding, unfortunately.... I feel like a lot of the problems come from a combination of extremely good union-busting by organized crime business, but much of the bad sentiment comes from these perceptions of cronyism and cliqueishness. Maybe with a bit of short-sightedness on the value of union dues mixed in.

You'll get some wins for labor by figuring out ways to implement the functions of a union in a more decentralized, depersonalized way.

So start from the problem space: We're looking for ways for individual employees to ensure that they're getting a fair contract with their employers, who have most of the actual leverage in any kind of negotiation space.

A slightly more radical version of Glassdoor might be helpful for initial negotiations: Get a big pool of data, where I as an individual can negotiate from hard data on their industry that with X background for this kind of job, here's what the expected salary and benefits are. There's a lot of immediate value for workers just in making the labor market (such as it is) more transparent. Since this is essentially a market tool, you can possibly sell it to employers, too, as a tool for figuring out what's going on in the labor market. The subscription fee for business analytics comes with a requirement to upload labor data...
posted by kaibutsu at 6:16 AM on May 6 [2 favorites]


« Older “I just want to be normal,”...  |  A bump on the head, a mysterio... Newer »


This thread has been archived and is closed to new comments