AI’s Appetite for Computation
July 16, 2020 2:17 PM Subscribe
Prepare for Artificial Intelligence to Produce Less Wizardry: A paper by MIT research scientist David Thompson et al. argues that the deep-learning approach to AI is "rapidly becoming economically, technically, and environmentally unsustainable".
[echo distortion effect guitar run intensifies]
(Sorry, I couldn’t come up with a better appetite for destruction/welcome to the jungle joke)
Interesting article; I’ll share it with our machine learning obsessed CIO
posted by das_2099 at 2:29 PM on July 16, 2020
(Sorry, I couldn’t come up with a better appetite for destruction/welcome to the jungle joke)
Interesting article; I’ll share it with our machine learning obsessed CIO
posted by das_2099 at 2:29 PM on July 16, 2020
my wild ass guess is that at least some of these applications would be cheaper if you owned your own hardware
Your wild ass guess is wrong.
posted by ambrosen at 2:32 PM on July 16, 2020 [12 favorites]
Your wild ass guess is wrong.
posted by ambrosen at 2:32 PM on July 16, 2020 [12 favorites]
Thompson believes that, without clever new algorithms, the limits of deep learning could slow advances in multiple fields, affecting the rate at which computers replace human tasks.
There are some examples of clever new algorithms. The paper doesn't appear to discuss AlphaGo, but it went from 48 TPUs requiring 10kW in March 2016 (the version that beat Lee Sedol; Elo rating 3,739) to just 4 TPUs requiring ~1kW a little over a year later, with considerably better performance (Elo rating 5,185).
Also, regarding this:
Of course Google Translate (or any other machine translation model) is not a substitute for a human translator in all circumstances. But for the many circumstances for which it is acceptable? $3 million in startup cost for a model that can run on a cell phone (offline even!) is a bargain.
But the paper does make a fair point that we are reaching the point where just throwing more computation at these problems will no longer see big gains. Algorithm design and better (rather than merely more) data will inevitably become more important again, just as happened when CPU clock speeds stopped doubling as fast as transistor count.
posted by jedicus at 2:43 PM on July 16, 2020 [6 favorites]
There are some examples of clever new algorithms. The paper doesn't appear to discuss AlphaGo, but it went from 48 TPUs requiring 10kW in March 2016 (the version that beat Lee Sedol; Elo rating 3,739) to just 4 TPUs requiring ~1kW a little over a year later, with considerably better performance (Elo rating 5,185).
Also, regarding this:
A translation algorithm, developed last year by a team at Google, required the rough equivalent of 12,000 specialized chips running for a week. By some estimates, it would cost up to $3 million to rent this much computer power through the cloud.$3 million sounds like a lot until you compare it to the fantastic cost:benefit ratio. How many millions would it cost to train the dozens of translators needed to translate between all of the languages that Google Translate handles? And how many billions would it cost to train enough translators that they could handle the over 100 billion words per day that go through Google Translate near instantaneously, if still imperfectly?
Of course Google Translate (or any other machine translation model) is not a substitute for a human translator in all circumstances. But for the many circumstances for which it is acceptable? $3 million in startup cost for a model that can run on a cell phone (offline even!) is a bargain.
But the paper does make a fair point that we are reaching the point where just throwing more computation at these problems will no longer see big gains. Algorithm design and better (rather than merely more) data will inevitably become more important again, just as happened when CPU clock speeds stopped doubling as fast as transistor count.
posted by jedicus at 2:43 PM on July 16, 2020 [6 favorites]
Obviously what needs to happen is you get all the deep-learning algorithms to look at all the other deep-learning algorithms, and determine which one is the most deep-learningy. I'm pretty confident that whatever happens after that will definitely be good.
posted by turbid dahlia at 3:05 PM on July 16, 2020 [20 favorites]
posted by turbid dahlia at 3:05 PM on July 16, 2020 [20 favorites]
Pretty sure it was misogynistic Isaac Asimov who described computers as the equivalent of being able to count really fast on an ever-increasing number of fingers and toes.
The description as deep-"learning" leaves me particularly dubious - most of it looks like deep-"processing" to me.
posted by Barbara Spitzer at 4:19 PM on July 16, 2020 [4 favorites]
The description as deep-"learning" leaves me particularly dubious - most of it looks like deep-"processing" to me.
posted by Barbara Spitzer at 4:19 PM on July 16, 2020 [4 favorites]
Your wild ass guess is wrong.
ok, great. why though?
posted by Dr. Twist at 4:42 PM on July 16, 2020 [2 favorites]
ok, great. why though?
posted by Dr. Twist at 4:42 PM on July 16, 2020 [2 favorites]
I don’t think that the “wild guess” is unreasonable, our company just purchased our own clusters for some analysis work because AWS and other easily available cloud services are poorly optimized for the computation in question, and academic clusters are hard to deal with IRT priority. Is available cloud computing a good match for machine learning tasks?
posted by q*ben at 5:51 PM on July 16, 2020 [2 favorites]
posted by q*ben at 5:51 PM on July 16, 2020 [2 favorites]
ok, great. why though?
It's actually not wrong. It's just that the scale where it get cheaper to get your stuff custom fabbed is very large, but Google definitely meets the threshold.
posted by sideshow at 6:02 PM on July 16, 2020 [2 favorites]
It's actually not wrong. It's just that the scale where it get cheaper to get your stuff custom fabbed is very large, but Google definitely meets the threshold.
posted by sideshow at 6:02 PM on July 16, 2020 [2 favorites]
I feel like things might be on the cusp of changing a whole bunch, given that a ML researcher I follow on Twitter posted over a year ago that the then-new iPad Pro had better TensorFlow performance than any desktop computer she'd ever used, presumably because the chip it runs on has dedicated machine-learning-oriented hardware (and also, separately, because Apple's been optimizing their chips to run JavaScript especially fast).
It's gonna be interesting seeing this sort of thing come to desktop ARM Macs, because imagine what they'd be able to do with chip design for devices that don't have to assume "no fan and running on a battery" when there's already this sort of ML performance on mobile devices.
posted by DoctorFedora at 6:59 PM on July 16, 2020 [2 favorites]
It's gonna be interesting seeing this sort of thing come to desktop ARM Macs, because imagine what they'd be able to do with chip design for devices that don't have to assume "no fan and running on a battery" when there's already this sort of ML performance on mobile devices.
posted by DoctorFedora at 6:59 PM on July 16, 2020 [2 favorites]
I'll put some numbers to the question "why use the cloud instead of buy your own hardware." If you are using a single NVIDIA Tesla V100, it's $2.48 / hour on google's cloud platform to use however you want. A single v100 goes for $7189 on Amazon. So, you need to be using the GPU for 2898 hours (about 4 months) continuously to "break even", which is neglecting the power bill or any other costs.
When training a model it's pretty common to use multiple GPUs. With GCP or AWS that's easy: just go to the gpu dropdown menu and select 8 GPUs instead of 1. It's similarly as easy to bump up the ram, memory, number of cpus. Sure, you're now getting charged $19.84 / hour, but you need those GPUs for less time (big gain if you have a deadline). For many startups and academics you aren't using the gpu consistently enough to justify the cost and hassle of managing the hardware that can quickly go out of date.
posted by getao at 7:09 PM on July 16, 2020 [9 favorites]
When training a model it's pretty common to use multiple GPUs. With GCP or AWS that's easy: just go to the gpu dropdown menu and select 8 GPUs instead of 1. It's similarly as easy to bump up the ram, memory, number of cpus. Sure, you're now getting charged $19.84 / hour, but you need those GPUs for less time (big gain if you have a deadline). For many startups and academics you aren't using the gpu consistently enough to justify the cost and hassle of managing the hardware that can quickly go out of date.
posted by getao at 7:09 PM on July 16, 2020 [9 favorites]
As we're constantly going through this at work, and while there are a lot of trade-offs in flexibility, fixed vs. operating cost, performance and intended use, but it's certainly not the case that any normal organization can be much cheaper than cloud. Using commodity cloud computing to represent true cost of processor power is pretty reasonable.
If you've never sat through exciting meetings on this topic it always seems like you can do it cheaper when you think in terms of small numbers, but by the time you've hit up management for millions for the second server room rebuild in five years it's a harder case to make. Definitely if you're in the sort of company where someone might say "I want to start on X project next month and I need 1000 processors" cloud gets appealing looking.
$3 million sounds like a lot until you compare it to the fantastic cost:benefit ratio. How many millions would it cost to train the dozens of translators needed to translate between all of the languages that Google Translate handles? And how many billions would it cost to train enough translators that they could handle the over 100 billion words per day that go through Google Translate near instantaneously, if still imperfectly?
Except the $3 million wasn't to to replace dozens of translators, it was the computational cost (only) to eke out an improvement over the already existing computer translation processes people have been using for eons. At least per Wired the paper is all about diminishing marginal returns on investment. Maybe this improvement was worth it for translation--heck, maybe $300 million will be worth it for the next generation of tiny improvement--but you're definitely getting into prices that rule it out for other uses.
I wonder if this seems like especially bad news for small research groups and academics, as for a while all you needed was a collaborator with a domain specific problem and data to pretty much guarantee a paper.
posted by mark k at 7:15 PM on July 16, 2020 [6 favorites]
If you've never sat through exciting meetings on this topic it always seems like you can do it cheaper when you think in terms of small numbers, but by the time you've hit up management for millions for the second server room rebuild in five years it's a harder case to make. Definitely if you're in the sort of company where someone might say "I want to start on X project next month and I need 1000 processors" cloud gets appealing looking.
$3 million sounds like a lot until you compare it to the fantastic cost:benefit ratio. How many millions would it cost to train the dozens of translators needed to translate between all of the languages that Google Translate handles? And how many billions would it cost to train enough translators that they could handle the over 100 billion words per day that go through Google Translate near instantaneously, if still imperfectly?
Except the $3 million wasn't to to replace dozens of translators, it was the computational cost (only) to eke out an improvement over the already existing computer translation processes people have been using for eons. At least per Wired the paper is all about diminishing marginal returns on investment. Maybe this improvement was worth it for translation--heck, maybe $300 million will be worth it for the next generation of tiny improvement--but you're definitely getting into prices that rule it out for other uses.
I wonder if this seems like especially bad news for small research groups and academics, as for a while all you needed was a collaborator with a domain specific problem and data to pretty much guarantee a paper.
posted by mark k at 7:15 PM on July 16, 2020 [6 favorites]
See? Like I’ve always said, we should stick to maximizing paperclips. Might turn the solar system into a paperclip Dyson sphere, but at least it won’t hog the AWS budget.
posted by No-sword at 3:01 AM on July 17, 2020 [6 favorites]
posted by No-sword at 3:01 AM on July 17, 2020 [6 favorites]
we should stick to maximizing paperclips
Microsoft tried this, but gave it up as a bad job with Office 2007.
posted by flabdablet at 5:23 AM on July 17, 2020 [4 favorites]
Microsoft tried this, but gave it up as a bad job with Office 2007.
posted by flabdablet at 5:23 AM on July 17, 2020 [4 favorites]
when your AWS spend at a large multinational company is measured in hundreds of millions of dollars a year 3 million dollars in a week is a small bump on the cost graph.
posted by noiseanoise at 7:09 AM on July 17, 2020
posted by noiseanoise at 7:09 AM on July 17, 2020
But my experience with all this is limited to building data correlation engines using pythonML, and I certainly have no expertise in data science other than building use cases and training very small data models.
posted by noiseanoise at 7:11 AM on July 17, 2020
posted by noiseanoise at 7:11 AM on July 17, 2020
I feel the article doesn't convey how fundamental the problem is. Deep learning was never a panacea for AI; it is an important milestone and breakthrough, but computer scientists never expected that it would scale forever for free. The answer is not merely more clever or efficient deep learning algorithms, or relying on eking out the last remaining drops of Moore's law. What the serious scientists are working on are gaining a fundamental understanding of AI in terms of computational complexity and what will improve the technology is going back to this as a basic science.
posted by polymodus at 7:58 AM on July 17, 2020 [3 favorites]
posted by polymodus at 7:58 AM on July 17, 2020 [3 favorites]
Breaking News: AI researchers discover concept of low hanging fruit.
Seriously in every field that I am aware of initial progress is huge and costs are low each further increment in productivity increases cost and decreases gain. It reads like researchers are surprised by this. Is that right?
posted by The Violet Cypher at 9:13 AM on July 17, 2020 [1 favorite]
Seriously in every field that I am aware of initial progress is huge and costs are low each further increment in productivity increases cost and decreases gain. It reads like researchers are surprised by this. Is that right?
posted by The Violet Cypher at 9:13 AM on July 17, 2020 [1 favorite]
They are concerned that it looks to be happening earlier than expected with deep learning methods--what really turned the tide on AI application was Google demonstrably solving chess, which really sold the idea to tech, but the broader scientific context has been the infamous Chomsky Norvig debate almost 10 years ago. Chomsky said that by turning to ML techniques AI researchers were giving up on trying to understand and make the study of AI intelligible, necessary for any legitimate scientific field. Today it looks like both sides were kind of right after all.
posted by polymodus at 10:23 AM on July 17, 2020 [2 favorites]
posted by polymodus at 10:23 AM on July 17, 2020 [2 favorites]
when your AWS spend at a large multinational company is measured in hundreds of millions of dollars a year 3 million dollars in a week is a small bump on the cost graph.
If you're spending a few hundred million a year $3 million in a week is a pretty noticeable increase in that week's burn rate.
But yeah, I see where you're going, so three points as to why this is still relevant:
1) Not everyone who wants to improve things is at a large multinational company; if the prediction is correct expect more and more people to be frozen out;
2) even at a large multinational company the argument that you should get $3 million for your specific project because it is "a small bump" is extraordinarily unpersuasive; and,
3) $3 million is the opening bid, and they are predicting costs continuing to rise.
I think the Wired article would have been stronger with better examples; "OMG machine translation cost $3 million!" does in fact deserve the response "that's not a bad deal" but that's not the only problem people throw AI at.
In my field AI is generally unproven in terms of having practical benefits. Most things fail, in the sense of not being better than humans. But there are obvious possible applications that people have been trying at. We'll simply stop trying if the cost goes up and up.
posted by mark k at 12:17 PM on July 17, 2020 [2 favorites]
If you're spending a few hundred million a year $3 million in a week is a pretty noticeable increase in that week's burn rate.
But yeah, I see where you're going, so three points as to why this is still relevant:
1) Not everyone who wants to improve things is at a large multinational company; if the prediction is correct expect more and more people to be frozen out;
2) even at a large multinational company the argument that you should get $3 million for your specific project because it is "a small bump" is extraordinarily unpersuasive; and,
3) $3 million is the opening bid, and they are predicting costs continuing to rise.
I think the Wired article would have been stronger with better examples; "OMG machine translation cost $3 million!" does in fact deserve the response "that's not a bad deal" but that's not the only problem people throw AI at.
In my field AI is generally unproven in terms of having practical benefits. Most things fail, in the sense of not being better than humans. But there are obvious possible applications that people have been trying at. We'll simply stop trying if the cost goes up and up.
posted by mark k at 12:17 PM on July 17, 2020 [2 favorites]
I guess in my specific use case if 3 million dollars in one week trains a security model that would otherwise take 3 years of 60 developers fucking around with ELK to achieve then it's pretty cheap over the lifecycle of the program I'm building.
posted by noiseanoise at 2:16 PM on July 17, 2020
posted by noiseanoise at 2:16 PM on July 17, 2020
Please note the $3 million dollar price tag and one week time is not what that project cost you. You are also paying the developers and collaborators on the deep learning team, and for all their computing resources that produce models that don't help.
They, too, might spend 3 years on it, and the main difference might be that they are also burning an extra $3 million/week in cloud services.
posted by mark k at 3:23 PM on July 17, 2020 [3 favorites]
They, too, might spend 3 years on it, and the main difference might be that they are also burning an extra $3 million/week in cloud services.
posted by mark k at 3:23 PM on July 17, 2020 [3 favorites]
Yeah I’ve been there! Fortunately I’m not there anymore...honestly the issue in my case was the data science organization not being able to understand that security use cases require a different kind of learning model. And on that note I’ve probably said too much...
(It’s astounding what kind of organizational chaos emerges from a company spending 550 million dollars a year on AWS lol!!!)
posted by noiseanoise at 5:42 PM on July 17, 2020
(It’s astounding what kind of organizational chaos emerges from a company spending 550 million dollars a year on AWS lol!!!)
posted by noiseanoise at 5:42 PM on July 17, 2020
Nobody technically competent is remotely impressed by this "AI is gonna run out of steam any day now" malarky, because we all know we're brute forcing because it's easy, not because it's necessary.
And we also know there's generalized tech fear, because to a real extent it's gotten much less accessible. Tech feels like something that will take your job and not give you one managing the tech, to a lot of people.
Meanwhile, this code still works, and gets wilder every day. Hopefully we can fix the inaccessibility part.
posted by effugas at 9:43 AM on July 18, 2020
And we also know there's generalized tech fear, because to a real extent it's gotten much less accessible. Tech feels like something that will take your job and not give you one managing the tech, to a lot of people.
Meanwhile, this code still works, and gets wilder every day. Hopefully we can fix the inaccessibility part.
posted by effugas at 9:43 AM on July 18, 2020
Obviously what needs to happen is you get all the deep-learning algorithms to look at all the other deep-learning algorithms, and determine which one is the most deep-learningy. I'm pretty confident that whatever happens after that will definitely be good.
Google Research just published a paper on (basically) that very topic [pdf].
posted by jedicus at 1:35 PM on July 23, 2020
Google Research just published a paper on (basically) that very topic [pdf].
posted by jedicus at 1:35 PM on July 23, 2020
« Older Can't stop thinkin bout it | Your Porn is About to Look a Lot Different Newer »
This thread has been archived and is closed to new comments
posted by Dr. Twist at 2:28 PM on July 16, 2020 [3 favorites]