Dark Side of the Force
August 2, 2019 9:33 AM   Subscribe

After Trump cites Amazon concerns, Pentagon reexamines $10 billion JEDI cloud contract process The White House has instructed newly installed Defense Secretary Mark T. Esper to reexamine the awarding of the military’s massive cloud-computing contract because of concerns that the deal would go to Amazon (Washington Post)

I'm getting tremendous complaints about the contract with the Pentagon and with Amazon; they're saying it wasn't competitively bid," Trump told reporters on July 18 (NPR) Those complaints reportedly included a flow chart created by Oracle alleging an Amazon “conspiracy” that made it to the president’s desk in recent weeks (Vanity Fair). Late last week, the Pentagon shot back at Oracle in a fiery statement following the court decision, calling the contractor’s allegations “the subject of poorly-informed and often manipulative speculation” (Politico) Oracle Corp lobbied aggressively and expressed concerns about the award process for the contract, including asking about the role of a former Amazon employee who worked on the project at the Defense Department but then recused himself, then later left the Defense Department and returned to Amazon Web Services (Reuters) Steve Kelman, a former federal procurement official now at Harvard’s Kennedy School of Government, said it would be improper for any president to influence the contracting process. “It’s not appropriate and it’s definitely not typical for a president to intervene on a contract,” he said. “That should be left to the Civil Service" (NYT).
posted by box (51 comments total) 10 users marked this as a favorite
 
"they're saying it wasn't competitively bid," Trump told reporters on July 18

Yes, please tell us how all government contracts during your administration were transparently, competitively bid, and not just government hand-outs to lackeys and sycophants (Need to Impeach.com).
posted by filthy light thief at 9:41 AM on August 2, 2019 [25 favorites]


This President? Do something improper? This is my surprised face.
posted by fifteen schnitzengruben is my limit at 9:46 AM on August 2, 2019 [3 favorites]


Let's not forget all this likely stems from the President's irritation with the Washington Post's coverage of him, which is owned by Amazon's Jeff Bezos.
posted by touchstone033 at 9:48 AM on August 2, 2019 [39 favorites]


If Bezos had paid proper obeisance to his lord and master 45, this wouldn't be an issue.
posted by Faint of Butt at 9:57 AM on August 2, 2019 [1 favorite]


It seems like if Bezos really wanted to, he has more than enough money to make like really, really complicated for Trump & Co for a long, long time. If he had that Peter Thiel-sort-of-streak in him, I mean. Amazon will certainly outlast Trump this term (and next, if it happens). What happens after that?
posted by jquinby at 10:13 AM on August 2, 2019 [8 favorites]


I mean he's almost Lex Luthor as it is.
posted by jquinby at 10:14 AM on August 2, 2019 [2 favorites]


The document that reached the US President contains a flow chart titled "A Conspiracy to Create a Ten Year DoD Cloud Monopoly," that suggests that senior military officials were potentially influenced by money. The chart includes images of two secretaries of Defense, Ash Carter (pictured alongside President Obama) and James Mattis, along with representations of dollar signs, arrows and a heart linking the various figures (Data Center Dynamics) On July 22, Trump retweeted a Fox News segment criticizing the JEDI contract as "The Bezos Bailout." It's not clear if the President had already seen the Oracle document by then (CNN)

[Fox News] Host Steve Hilton said: "President Trump is demanding more information on the JEDI contract and is considering intervening - quite right too. Draining the swamp means stopping this contract... It's not just appropriate, but vital that the President kills this contract...

"Mr President, if you want to keep draining the swamp, don't let the Bezos bailout fill it back up."
posted by box at 10:14 AM on August 2, 2019 [5 favorites]


Trump doesn't plan past lunch. He's always been reactive rather than a strategist. He bumbles into things. He's surprised by events constantly. It's a major weakness of his.
posted by bonehead at 10:15 AM on August 2, 2019 [2 favorites]


Doesn’t Trump have a beef with all the other cloud providers (Google, MS, etc) that can deliver at the required scale/availability and also have the necessary clearance? Seems like a no win situation for the project regardless of who they pick.
posted by delicious-luncheon at 10:25 AM on August 2, 2019


About the most impeachable thing he's done yet, really.
posted by jamjam at 10:25 AM on August 2, 2019 [1 favorite]


I don’t give two shits what Fox News host Steve Hilton has to say about any damn thing.
posted by Abehammerb Lincoln at 10:27 AM on August 2, 2019 [12 favorites]


Battle of the shitbirds.
posted by grumpybear69 at 10:27 AM on August 2, 2019 [3 favorites]


So Oracle leveraged Trump's anti-Bezos attitude to wedge their way back into the bidding process. You're surprised?
posted by JoeZydeco at 10:27 AM on August 2, 2019 [12 favorites]


Wow. Didn't Oracle used to make software?
posted by biogeo at 10:32 AM on August 2, 2019 [2 favorites]


Probably has family shorting Amazon stock prior to tweet. Corruption through and through. It is shocking to see any normal governance. I bet the Oracle CEO is going to get a huge bonus even if this scheme fails. Dirty, dirty, dirty.
posted by zerobyproxy at 10:33 AM on August 2, 2019 [5 favorites]


How about no tech companies help this administration's War Department? (Or any administration's, for that matter.)
posted by kmz at 10:51 AM on August 2, 2019 [12 favorites]


I would love it if this inspired Bezos to dig into his pockets to make life harder for Trump... I can't say I'm deeply sad if Amazon doesn't win this contract, or if it even puts off the government choosing a winner for this. It'll be a travesty if Oracle gets this, though.

I think the thing that, IIRC, disqualified Oracle is entirely reasonable - namely, that the contract can't be awarded to a company where it represents a majority of their cloud business. Say what you will about Amazon, this contract would be very good for them but not an outsize amount of their business. Oracle's cloud business is miniscule in comparison, and that's not really what I'd want if I were in the Pentagon's shoes trying to choose a provider.
posted by jzb at 11:02 AM on August 2, 2019 [10 favorites]


Based on the experiences had north of 49 in public sector sphere web spaces and aggregation efforts, I'd suggest that letting either Oracle or IBM anywhere near this contract dooms it instantly. We've had a number of really high profile crash-and-burns from both companies in the past years. It's incomprehensible to me why the Canadian Government lets them continue to bid on any contracts, given past levels of completion and competence.
posted by bonehead at 11:09 AM on August 2, 2019 [16 favorites]


I'm getting tremendous complaints about the contract with the Pentagon and with Amazon; they're saying it wasn't competitively bid," Trump told reporters on July 18
Translation: "Stop publishing my name in articles about Jeffrey Epstein or I will hurt you"
posted by sexyrobot at 11:23 AM on August 2, 2019 [2 favorites]


Doesn’t Trump have a beef with all the other cloud providers (Google, MS, etc) that can deliver at the required scale/availability and also have the necessary clearance?

Not Microsoft AFAIK. This Google beef even predates Trump, when they snubbed DARPA funding in 2014. People working adjacent to DoD or DHS are likely to call them hippie commie traitors (or similar) in private, especially after they dropped out of the JEDI bid last year.

With Trump, Thiel, Ellison, and the KSA (allegedly) all with knives out, Bezos has his hands full.
posted by RobotVoodooPower at 11:23 AM on August 2, 2019 [2 favorites]


On the one hand TRUMP!! On the other hand - how about we stop handing over huge swaths of public infrastructure to these Global Technology Oligarchs !? On the other other hand - F*ING TRUMP!!!
posted by helmutdog at 11:29 AM on August 2, 2019 [1 favorite]


With Trump, Thiel, Ellison, and the KSA (allegedly) all with knives out, Bezos has his hands full.

How dare you make me cheer for the villain!
posted by fatbird at 11:40 AM on August 2, 2019 [5 favorites]


This is that one time that a stopped clock tells the right time.
Handing over national security to a vampiristic corporate monopoly is about the stupidest thing I can imagine.
Whatever Trump's motivation may be.
posted by victotronics at 12:11 PM on August 2, 2019 [1 favorite]


That "vampiristic corporate monopoly" is reason you have Netflix, AirBNB, and the very website you are reading right now.
posted by sideshow at 12:15 PM on August 2, 2019


That "vampiristic corporate monopoly" is reason you have Netflix, AirBNB, and the very website you are reading right now.

Websites, including this one, existed way before AWS.
posted by grumpybear69 at 12:24 PM on August 2, 2019 [17 favorites]


Handing over national security to a vampiristic corporate monopoly is about the stupidest thing I can imagine.

He just wants to hand it to a different one.
posted by praemunire at 12:34 PM on August 2, 2019 [4 favorites]


It's a $10 billion contract, and the DoD doesn't want theirs to be the majority of the company's cloud business--at that scale, big companies are the only ones that can afford to play. The other companies that made bids were IBM, Oracle (who were both disqualified), and Microsoft. I agree that maybe national security cloud data should be handled in-house, but I don't think picking Microsoft over Amazon would result in any great reduction in vampirism.
posted by box at 12:34 PM on August 2, 2019 [1 favorite]


"That "vampiristic corporate monopoly" is reason you have Netflix, AirBNB"

I despise AirBNB and have never used it. I mostly get DVDs from Netflix, so if Amazon went away I'd easily give up streaming. (And I get almost as many DVDs from the public library as from Netflix, so I could easily give up Netflix altogether.)

Did you have a point?
posted by victotronics at 12:56 PM on August 2, 2019 [2 favorites]


Not that you really needed anything else to know that Oracle was a vile shell of a company, but there you go.

And yeah -- Amazon and Microsoft were the only non-disqualified bids left in JEDI. Given that the DOD is already beholden to Microsoft for its desktops, giving the cloud contract to Amazon actually makes government IT more diverse. Plus, there are a lot more developers around with AWS experience than Azure.

Oracle's argument, such as it is (which it is really not), is that the DOD should have picked a "multi cloud solution", i.e. they should have spread the money around more. And the inevitable result would have been an unusable, fragmented morass of systems, where probably everyone would try desperately to get on AWS (because that's what everyone wants to deploy to) and whatever other solutions they chose would be underutilized and basically a waste of money.

The competition the DOD wanted was probably AWS vs GCP, but when Google didn't throw their hat into the ring, it made things awkward because AWS was really the only solution anyone likely wanted. Certainly nobody wants whatever shit sandwich Oracle is serving.

Oracle and Trump seem to be of one mind on this: they're in it for themselves and they don't care about the cost to the public.
posted by Kadin2048 at 1:23 PM on August 2, 2019 [9 favorites]


ya, I can't imagine a worst outcome than "sharing" the cloud contract. Do you want people to be storing all their information of personal USB drives? Cause that's how you'd do it.
posted by Mitheral at 1:31 PM on August 2, 2019 [3 favorites]


Don't care about Amazon but this is straight out of The Fascist Dictator's Playbook. Every day there are fewer and fewer checks on Trump's power, and this seems action seems to be a bit past an invisible point of no return. It may seem like just desserts, yet if one of the world's largest corporations can be made to bend to his personal whim, what hope is there for the rest of us? How do we get back to a functioning democracy at this point? Shrugs and resigned sighs by the left-leaning commentariat don't seem to be accomplishing much .
posted by xigxag at 1:39 PM on August 2, 2019 [2 favorites]


All future DOD and federal contracts will start by issuing a RFM (Request for Memes).
posted by srboisvert at 1:40 PM on August 2, 2019


reason you have Netflix, AirBNB, and the very website you are reading right now

That comment seems familiar.
posted by aspersioncast at 1:43 PM on August 2, 2019 [3 favorites]


And yeah -- Amazon and Microsoft were the only non-disqualified bids left in JEDI. Given that the DOD is already beholden to Microsoft for its desktops, giving the cloud contract to Amazon actually makes government IT more diverse.

For the DoD, maybe, but Amazon already has a $600 million dollar contract with the CIA for cloud computing services.
posted by vibratory manner of working at 1:51 PM on August 2, 2019


He just wants the contract to go to a small enough company to become a Trump Organization subsidiary.
posted by oneswellfoop at 1:54 PM on August 2, 2019


Oracle is the Saul Goodman of government contractors, forever trying to get you to buy Danny's Laser Tag place. Because he is a good guy, and he has a place for you to set up your servers behind the Skeeball games.
posted by The Power Nap at 2:14 PM on August 2, 2019 [2 favorites]


> ya, I can't imagine a worst outcome than "sharing" the cloud contract.

I am presently running a high profile site in in AWS across multiple regions. I am also replicating data to GCP as both warm systems as well as just raw backups, enough to rebuild everything while the most important parts can limp along in a degraded fashion. I am not working with data that has anywhere near the criticality of government data.

There are plenty of ways to use both services in ways in which they will work well together. Using a provider-independent language for provisioning, such as Terraform, simplifies this. The networks themselves can be connected via VPN if needed, making it easy to share resources.

Where it becomes difficult is when you start using services that are AWS or GCP only that do not translate directly into other systems. Keeping critical systems on a proprietary cloud-based systems that cannot be easily migrated to other solutions even within the same provider is a giant mistake.

There are plenty of valid reasons to not keep all of your eggs in one basket. Having all of your data in one companies control is still a risk, even if it is a very large company and even if it is distributed between multiple locations. Risk thresholds should be much lower for federal systems than they are for private entities, and not providing a way or at least recovering systems via data stored in another provider should be unacceptable.

That being said - Oracle has no place in this deal. AWS, GCP, Azure at least all have dedicated resources and knowledge for a government "cloud" system, and they all meet existing minimal compliance standards and then some. There's nobody else from a could provider perspective who can meet the needed compliance standards - full stop. Those are your options.

This whole kerfuffle is happening over the same sorts of wrong and corrupt reasons that we have come to expect from this administration, but that doesn't mean that the actual correct answer is to hand everything to AWS. At minimum you'd want enough encrypted data kept on a second provider (with strong control over the keys) to recover everything critical.
posted by MysticMCJ at 2:24 PM on August 2, 2019 [4 favorites]


I am not working with data that has anywhere near the criticality of government data.

I'm with a really large bank, and your financial information AND MONEY is about as critical as it gets.

We built our own cloud.
posted by mikelieman at 2:50 PM on August 2, 2019 [2 favorites]


Can the feds just build their own cloud? Sure. But everything Congress does these days is for contractors.
posted by 922257033c4a0f3cecdbd819a46d626999d1af4a at 4:41 PM on August 2, 2019 [3 favorites]


That comment seems familiar m

I’ve said it a lot more than just that one time.
posted by sideshow at 6:14 PM on August 2, 2019


this whole beef is because bezos owns the post. which, despite the evils of amazon, is a good paper and is doing great work undermining trump.
posted by wibari at 10:21 PM on August 2, 2019 [1 favorite]


Can the feds just build their own cloud?

Probably? I mean, any large organization can. But that doesn't mean they should. It would likely cost more than just having an existing cloud provider add more capacity.

Apple, for instance, is certainly capable of building its own datacenters and buying servers and disks and whatnot, but they don't, because that's not really their thing and it's cheaper to build their services on top of S3 (and Google's services, IIRC). It's not really DoD's thing either.

The whole point of cloud architecture is outsourcing the physical ownership of datacenters, servers, disks, etc. to a specialized company and paying for a metered service. You can argue that it's a bad architecture and you're better off owning everything down to the bare metal including the real estate the servers sit on, and some people do argue that (and some companies still do it, and I'm sure the DoD will still keep some stuff in-house in any scenario), but it's getting to be a minority opinion these days because the advantages of cloud-based architectures are very real. I'm a pretty old-school "give me a 2U server and SSH credentials and STFU" kind of guy but even I have to admit that standing up a web app on AWS or GCP is pretty painless by comparison, and the reduced maintenance is really compelling by itself.

Personally, I think cloud services are becoming important enough economically that having the government become involved with them is probably a good thing. It's entirely possible that cloud services tend, due to economies of scale, towards a natural monopoly or duopoly, and strong regulation is the only thing that will prevent rent-seeking behavior. Having the government be a significant customer and consumer—thus giving the government a significant vested interest in not allowing a squeeze by a single major provider—might be the most direct route to an eventual regulated market. In the same way that AT&T became the subject of regulation (in 1914) at about the same time that the War Department was realizing the benefits of the telephone and becoming a significant user of telephony services, it could be that significant government uptake of cloud services is what leads to, or (if Amazon is smart) prevents the circumstances that might lead to, similar regulation. One business squeezing another is easy to write off as "just business", but you squeeze the government at your peril.
posted by Kadin2048 at 10:37 PM on August 2, 2019 [2 favorites]


We built our own cloud.

Along the theme of failed Trump properties like Trump University, Trump wines, Trump casinos, and Trump steaks, I'm trying to imagine the government using a Trump-approved vendor's version of a TrumpCloud™.

Say what you will about Amazon, but at this point it is basically a choice between a proven service and whatever the currently favored crony kissing up to the wannabe dictator offers. Buyer beware.
posted by They sucked his brains out! at 5:58 AM on August 3, 2019


I'd assume they would want vendor diversity...
posted by sfts2 at 1:31 PM on August 3, 2019


Apple, for instance, is certainly capable of building its own datacenters and buying servers and disks and whatnot, but they don't, because that's not really their thing and it's cheaper to build their services on top of S3 (and Google's services, IIRC). It's not really DoD's thing either.

I question this because there has to be some point of scale where it's cheaper to engineer something specific to your needs than to continue paying for a general metered service. I think it's a justifiable decision to stick with the metered service, assuming the economics work out; but I don't think it's strictly true that anything outside your "core competencies" is more cheaply sourced externally, in part because there's a hidden cost to not owning internal expertise in a critical dependency.
posted by fatbird at 2:17 PM on August 3, 2019 [1 favorite]


I question this because there has to be some point of scale where it's cheaper to engineer something specific to your needs than to continue paying for a general metered service.

There's some point of scale where that's true, but the vast majority of companies never hit it.
posted by ryoshu at 6:37 PM on August 3, 2019 [1 favorite]


I would certainly never say that nothing outside an organization's core competencies is worth bringing in-house; there are certainly times when it makes sense. But I doubt that Apple, being the size that it is, and having the resources that it does, and considering the rather diverse things that they do bring in-house (silicon, etc.), hasn't considered building its own DCs and self-hosting. That they don't is pretty suggestive of the numbers not working out as a capital investment.

In particular, I think it'd be pretty hard to compete with Amazon, since they basically have a whole cyber-retail operation that's subsidizing the fixed costs of their IT infrastructure. (Or put differently, all their AWS revenue is essentially the result of building excess capacity above what's strictly needed for their retail operation, which they'd need to have regardless.) Microsoft is somewhat similar: Azure is relatively small compared to their "commercial cloud" revenue (Office 365, etc.), and probably rides on much of the same infrastructure. The pricing to a customer in both cases just has to cover the marginal cost of adding the additional capacity.
posted by Kadin2048 at 9:39 PM on August 3, 2019


All of this talk of clouds reminds me of one of my favorite Onion articles: HP Offers 'That Cloud Thing Everyone Is Talking About'

"We have social sharing, 4G, 5G, 6G, all the Gs. We have app!"
posted by fifteen schnitzengruben is my limit at 10:56 PM on August 3, 2019 [1 favorite]


Along the theme of failed Trump properties like Trump University, Trump wines, Trump casinos, and Trump steaks, I'm trying to imagine the government using a Trump-approved vendor's version of a TrumpCloud™.

And in addition to building out our own cloud, We're operating 29% cheaper than any commercial cloud provider.

In this model, it would be more like... I dunno. LLNL building out datacenters.

Thing is, the Government's supposed to do the things that aren't a commercial advantage OR profitable.
posted by mikelieman at 3:41 AM on August 4, 2019


I cannot speak for the US experience, but a major problem with self-hosting/building our own cloud on the Canadian side is that the government has not authorized salary levels that are competitive with Amazon, Microsoft or the Googles. So getting enough staff to do so is a real challenge. All governments are chronically short on IT/CS staff. At the same, increasingly time crazy prices for external contractors seem to be just fine though. Easier to pretend the costs aren't incurred or on-going, I guess, if you're not paying operating costs or staff dollars, but rather contractors.

Contracting out is increasing encouraged in an environment of "controlling costs".

It's also a way to bust public service unions. Government is one of the few places large numbers of programmers are unionized.
posted by bonehead at 11:53 AM on August 4, 2019 [1 favorite]


I cannot speak for the US experience, but a major problem with self-hosting/building our own cloud on the Canadian side is that the government has not authorized salary levels that are competitive with Amazon, Microsoft or the Googles.
This is a huge problem at the U.S. federal level: the benefits aren’t bad but they’re not going to make up for the imbalance, and there’s a sizable voting bloc in Congress which is always trying to make the deal worse. 18F tried to hire staff to do work which would normally be [very inefficiently] contracted out, and they were still at a big disadvantage despite their positions being in the top two rungs of the non-executive payscale (GS-14/15), which you’ll note are not bad money but not competitive with what the large contractors or tech giants pay.

Most federal agencies have trouble hiring even at those levels since in most other fields that’s a senior position — rarely for senior technical specialists, more commonly managers with a fair amount of responsibility (when I was a NASA contractor, the person overseeing my companies contract and several others – at least hundreds of people – was a GS-14). That inevitably means a certain level of opposition or resentment when you need such a high percentage of the high grades in an entire agency.
posted by adamsc at 7:51 PM on August 4, 2019 [1 favorite]


« Older This is How Things Work Now at G/O Media   |   On a Roll Newer »


This thread has been archived and is closed to new comments