How I cut GTA Online loading times by 70%
March 8, 2021 11:08 PM Subscribe

"GTA Online. Infamous for its slow loading times. Having picked up the game again to finish some of the newer heists I was shocked (/s) to discover that it still loads just as slow as the day it was released 7 years ago. It was time. Time to get to the bottom of this."

A non-technical explanation.

posted by Pyrogenesis (45 comments total) 18 users marked this as a favorite

My favourite genre of programmer story is "This bug in your closed source software is so annoying that I hacked it to fix it for you; look, I've even drawn you a map; now could you please fix it properly?".
posted by confluency at 11:15 PM on March 8, 2021 [32 favorites]

Accidentally quadratic runtime is a common trap for software developers — Bruce Dawson describes it as "the sweet spot of badly scaling algorithms: fast enough to make it into production, but slow enough to make things fall down once it gets there".
posted by cyanistes at 12:41 AM on March 9, 2021 [14 favorites]

I remember it having some showstopper save game loading bugs in the single player around release too. So glad the new crop of consoles are SSD.
posted by GallonOfAlan at 12:52 AM on March 9, 2021

i love this for the altruism - the amount of time it took them to figure this out and fix it probably exceeds the amount of time they saved playing the game over the lifetime of playing the game but now many could benefit and maybe the balance of time comes out ahead

also, it's some nice sleuthing
posted by kokaku at 1:13 AM on March 9, 2021 [1 favorite]

8 years on and GTAO still makes it into the steam top 10 occasionally. All they have to do is release a flying motorcycle or something every month and the cash rolls in. They opened a casino in 2019 where you can play with real-world money (there are no real-world payouts though). That's where the profit is, not debugging. They should hire this guy but they won't. Where's the motive? There's no motive to make single-player DLC, or GTA6.

That's the tragedy. We should have GTA7 by now and it'd look like a working version of Cyberpunk. But they're rolling in flying motorcycle money.

All that said, this fix is really cool. It doesn't surprise me much that Rockstar North prioritizes microtransactions over debugging.
posted by adept256 at 2:11 AM on March 9, 2021 [7 favorites]

One for the Accidentally Quadratic tumblr (except that's not been updated in over a year).
posted by pw201 at 2:20 AM on March 9, 2021 [6 favorites]

I’m wondering if the developers aren’t connecting internally to a server that sends a much smaller list and thus aren’t experiencing it. Because minutes on load might annoy players but when you’re a developper you start that executable 100 times per day because you’re debugging it it’s really really annoying. Also it’s the easiest kind of issue to profile, it takes minutes your 3-4 time you notice you’re still parsing the json you go have a look...
posted by WaterAndPixels at 5:03 AM on March 9, 2021 [7 favorites]

Clawsoon, if there are only 4 items, you either know the index because you’ve cached it or you scan. Anything fancier is slower. Put a limit on the size if you’re worried
posted by WaterAndPixels at 5:05 AM on March 9, 2021 [2 favorites]

i love this for the altruism - the amount of time it took them to figure this out and fix it probably exceeds the amount of time they saved playing the game over the lifetime of playing the game but now many could benefit and maybe the balance of time comes out ahead

For me, this is just another level of the game. and once he started playing it, he couldn't put it down.
posted by mikelieman at 5:26 AM on March 9, 2021 [5 favorites]

For me, this is just another level of the game. and once he started playing it, he couldn't put it down.

Interesting idea for a game there. Make a game that requires players to hack-in and fix specific parts so the game can remain playable and players can advance. Kind of the ultimate in co-op gaming.
posted by Thorzdad at 5:30 AM on March 9, 2021 [2 favorites]

Hummm phone replies... I meant to say you *break* 3 or 4 times....
posted by WaterAndPixels at 6:02 AM on March 9, 2021

I was a business application programmer for thirty years, so not much under-the-hood stuff beyond getting something of a handle on the way I could get the OS to help me be more productive, but that was a fascinating read. I have had the pleasure on a couple of occasions of figuring out weird anomalies, and I could appreciate t0st's elation at tracking those two pieces of "logic" down.
posted by gp_guy at 6:14 AM on March 9, 2021 [1 favorite]

Make a game that requires players to hack-in and fix specific parts so the game can remain playable and players can advance.

Government: The Game
posted by thatwhichfalls at 6:18 AM on March 9, 2021 [3 favorites]

@adept256 It's been said that the Playstation 2 had three Grand Theft Auto games, and Grand Theft Auto V has had four Playstations (PS3, PS4, PS4 Pro, PS5)

Having said that looking at the gap between GTA V and Red Dead Redemption 2, I was never expecting to see a GTA6 before 2023 at the earliest.
posted by GallonOfAlan at 6:46 AM on March 9, 2021 [2 favorites]

Augh, anecdote time, inspired by the Accidentally Quadratic stuff.

A colleague of mine is working on a codebase originally produced by an outside company. It's the backend for an application used by a few million users, backed by a database. There's a company-to-end user broadcast message wall feature in it that's currently used with a subset of a few hundred thousand users. Upon examining (unrelated) high IOPS numbers for the database, a colleague came across an implementation detail in it which, if not caught, would have meant disaster down the line.

The backend stores message read state per message per user. The way this is implemented is perhaps the most absurdly wasteful one I've encountered, and I cannot believe how the original devs did not see this coming.

For each new user, a database row tracking read state is created for every past message. There's fortunately only a few dozen messages right now. Because conversely, for each new message, a database row is created for each user.

A new user registration comes in? A few dozen new rows are created. A single new message is sent? A few hundred thousand new rows are created... And don't get me started about all the "mark as read" SQL UPDATEs.

Had this functionality been expanded to cover all users and used for more frequent messaging before we took over,
a single burst of more active messaging would have brought the database to its knees; the messaging feature is extraneous to the core application functions, and the DB was set up with completely different types and frequencies of write operations in mind.

I shudder to think what other surprises lurk in the code.
posted by jklaiho at 7:12 AM on March 9, 2021 [4 favorites]

Interesting idea for a game there. Make a game that requires players to hack-in and fix specific parts so the game can remain playable and players can advance. Kind of the ultimate in co-op gaming.

There is some of this in Doki Doki Literature Club!
posted by Quonab at 7:42 AM on March 9, 2021 [3 favorites]

A long time ago we had a bottleneck in our system. It read a bunch of entries from the database, stuck them in XML and sent them across the network. As the number of entries grew, eventually it started to slow down. Everyone started to speculate wildly: the database people looked at optimizing the query, the network folks suggested compressing (!) the xml before we sent it across. We put it in the profiler and the problem was malloc. Like I said, this was done a long time ago when all this was done by hand. So the XML was built up by iterating over each database entry. For course this would append to the existing XML, requiring the whole thing to be reallocated requiring many many mallocs. Simply chunk allocating the XML memory made the problem disappear.
posted by lowtide at 7:44 AM on March 9, 2021 [3 favorites]

Where's the motive? There's no motive to make single-player DLC, or GTA6.

/me cries in Red Dead Redemption 2 story mode
posted by nathan_teske at 7:45 AM on March 9, 2021 [1 favorite]

At my first game dev job, my boss was an old-school programmer who worked on arcade games in the 90s before transitioning to modern games. He didn't trust C++ at all so the codebase was almost all pure C.

Anyway one of his favorite coder anecdotes was about a coworker of his who was told to write a hash table, but that coworker didn't know what that was and was too proud to go get an algorithm book and look it up. So he instead wrote a giant table of hashes to search linearly, and very proudly showed it off to the other programmers as a new innovation. Who all called him an idiot.

I guess that guy later got a job a Rockstar
posted by JZig at 7:48 AM on March 9, 2021 [4 favorites]

Make a game that requires players to hack-in and fix specific parts so the game can remain playable and players can advance.

Steam has several games that are exactly this! Check out else Heart.Break() or Hack 'n' Slash
posted by JZig at 7:55 AM on March 9, 2021 [1 favorite]

Lego Island 2 is the game I always remember for its obscenely long load times. I am told the reason for them is that the load screen displays a rotating pizza, and will only condescend to load any data if doing so won't interfere with the speed of the pizza animation.
posted by one for the books at 8:07 AM on March 9, 2021 [9 favorites]

My favorite observation so far:

“A week from now, someone will prove P=NP because all the problems we thought were NP were just running strlen() on the whole input.”

posted by JoeZydeco at 8:15 AM on March 9, 2021 [7 favorites]

That's where the profit is, not debugging. They should hire this guy but they won't. Where's the motive?

When a game is taking 10 minutes to load? That's enough time that it'll make someone pause to think whether or not they want to play it or load up something else.

Fewer players means less DLC/in-game sales, fewer streams to give your game free advertising, etc.
posted by explosion at 8:49 AM on March 9, 2021

We put it in the profiler and the problem was malloc.

Malloc! Solitude! Filth! Ugliness! Ashcans and unobtainable dollars! Children screaming under the stairways! Boys sobbing in armies! Old men weeping in the parks!
posted by thelonius at 8:55 AM on March 9, 2021 [7 favorites]

Load time stories is it? Well, in the early days of the Commodore 64 for most European owners the Commodore C2N tape deck was the storage device used, as the 1541 disk drive was expensive. Before tape fastloaders were invented, loading a game could easily take half an hour due to Commodore's overly cautious approach. And it would fail a good percentage of the time.
posted by GallonOfAlan at 9:10 AM on March 9, 2021 [3 favorites]

It always amazes me that fairly simple and glaringly obvious to anyone actually trying to figure it out stuff like this doesn't get fixed. It is almost entirely a business process fail IMO.

The way I have most often seen it happen is that the developers are overextended enough and constantly bombarded with requests for new features that they can't get around to dealing with something that sort of works, and then the stupidity persists for enough time that the users learn to accept it as normal.
posted by each day we work at 9:14 AM on March 9, 2021 [4 favorites]

Interesting article. My experience in pointing out bugs to devs is mixed...some take it seriously, others will absolutely not fix their bugs even though they are obviously there. Then again, my most regular gig these days is fixing the code done cheaply elsewhere, where my fee to track down and fix is more than it would have been to hire me to do it "right" in the first place...
posted by maxwelton at 9:15 AM on March 9, 2021 [4 favorites]

It always amazes me that fairly simple and glaringly obvious to anyone actually trying to figure it out stuff like this doesn't get fixed. It is almost entirely a business process fail IMO. It would have taken an experienced developer with access to unobfuscated course less than a couple weeks to figure this out and ship at least something like the authors patches.

At my place of business on, things I've built and maintained, your "less than a couple weeks to figure this out and ship" is a seven figure financial commitment once all the costs of all the people (the developer is just one of many involved in getting this from his/her dev environment to the end user) and other resources are added up. I'm guessing the cost-per-hour at Rockstar for work at is less than mine (if they wanted to make more money, they wouldn't be in gaming), but it still costs a lot of money to roll out features and fixes.

Anyway, the there is almost certainly a list of features/fixes, dozens/hundreds of entries long, similar in effort to this one, and each of them requires a Project Manager to decide "is this worth spending a million dollars on?", and the answer for most of them is: "not now, if ever". For the rest, it's "is it worth spending a million dollars on Fix A, or is it better spend it on Feature B?", and it's possible (even likely) this exact problem was on that list. It just was never prioritized above other stuff that got done.

I say "likely", because for the good part of a decade I've been employed at everyone's favorite fruit company, I've seen dozens of feature requests/complaints/potential fixes/etc. for stuff I've been personally responsible for on Twitter, Facebook, Hacker News, and even many on MetaFilter.com. In all these years, I've literally never heard about them for the first time via those channels. 100% of them were known to me beforehand, but resources are finite, and whatever you all were complaining about at that point in time was prioritized lower than something else.
posted by sideshow at 9:59 AM on March 9, 2021 [7 favorites]

In this case the trade off would be "okay we can get the flying motorcycle in which will make us money or we can fix that thingie that nobody has cared about for a decade and won't make us any money at all," I expect.
posted by Ghostride The Whip at 11:14 AM on March 9, 2021 [2 favorites]

It is almost entirely a business process fail IMO.
I agree with this. I work in software, and we have business cases that pay back in 12 months too, hundreds of millions of dollars in some cases, and the inertia of business is really slow to get them approved.

I'd also say that in our business, bug fixes are not reviewed by project managers, who only handle new development requests funded by business dollars (where they really do ask if such and such is worth the $$ tradeoff) but rather they are generally funded internally, so you have to have a really good manager/app president willing to spend to maintain products. I'd personally slate that at slightly less than 50/50 good managers to coasters.

Even with a good manager, it would still go on a list of sustainment activities, and may get a low priority, but funding is not the primary gating factor.
posted by The_Vegetables at 11:21 AM on March 9, 2021

When I'm reviewing the code of younger programmers, I find myself torn on calling out much smaller versions of this. I've already asked them to fix half-a-dozen things during this review... and there are only four items in this list, and it'll probably never grow... I guess it would be okay to just scan the whole list for each of the items? But is that a good habit to let them get into?

I'm not a young programmer and I made this kind of mistake not so long ago. Part of a program I wrote kept a list of special case entries and every new entry was checked against all others. It wasn't good code, I knew it, the code reviewer knew it, and my boss knew it. We were pushed for time though and, as I pointed out, in the hundreds of millions of test entries, across multiple files, the longest I had ever seen the list grow to was 16 entries. Most of the time the list consisted of no more than 4 entries. So for a list that small it didn't matter how inefficient it was and anyway, I said, that entire section of code needed a rewrite and I would do it for the next release in a couple of months time.

Then the pandemic hit and we all got sidetracked by covid work for a few months, the rewrite was forgotten about.

And then, I got a query from a user asking why the program was taking over a week to run. So I took a look at their data. Instead of creating lists of 4 or 16, it was creating lists with hundreds of thousands of entries. The longest was 850,000 entries long and the program was maintaining multiple lists of that length. My program could handle lists of up to 100k long without too much trouble but anything more than 300k made it slow to a crawl.

This is the joy of scientific programming. Leave a weakness in your code and some scientist somewhere will produce a huge, pathological, dataset to break it.

I rewrote my special case handling code. The resulting algorithm was more complex but reduced the amount of total work. The run time went down from 7+ days to about an hour.

So I did the wrong thing, I knew I had done the wrong thing, I didn't think it mattered, and it came back to bite me. I really should have learnt that lesson by now.
posted by antiwiggle at 11:25 AM on March 9, 2021 [11 favorites]

JZig: So he instead wrote a giant table of hashes to search linearly, and very proudly showed it off to the other programmers as a new innovation

I present to you SortedList from Microsoft's .NET framework. When I first encountered this, coming from Java I assumed it was like TreeMap and must be using a binary tree so that inserts and removals could be done in O(log2(n)) time. But I couldn't see anything in the documentation about that. Not a single big-O to be found. Surely they aren't literally using a list and shifting half of the entries out of the way to do an insert? Oh yes they did.
posted by swr at 12:45 PM on March 9, 2021 [5 favorites]

I am 100% calling it THE TASK MANAGER from now on.
posted by turbid dahlia at 2:20 PM on March 9, 2021 [1 favorite]

Wow - what a coincidence - just watched this series about "THE TASK MANAGER" by the original author. (from Regina, SK to Microsoft)

(I loves me some Task Manager, even over the sysinternals alternative)
posted by rozcakj at 3:00 PM on March 9, 2021 [1 favorite]

When a game is taking 10 minutes to load? That's enough time that it'll make someone pause to think whether or not they want to play it or load up something else.

Fewer players means less DLC/in-game sales, fewer streams to give your game free advertising, etc.

At this point, so far into its lifecycle, there's probably not a lot of resources being directed toward bugfixes for GTA5, especially not Known Shippable bugs that have existed the whole time and not noticeably slowed down the BILLIONS of dollars the game has made. Any programmers interested in fixing these sorts of bugs are probably being put to use working on GTA6 or Red Dead Online or whatever else Rockstar is currently working on behind closed doors. It will be very interesting to see if Rockstar actually implements the fix this guy laid out.
posted by Mr.Encyclopedia at 3:30 PM on March 9, 2021

Being a developer I'm usually on the developers' side. Developers don't spend a little more effort to fix things because often it took 120% of their effort to just get to that point. It's like saying, just run faster to win the race.

Also simple fixes might involve a lot more than just changing the code; there may be many other processes: code reviews, regression tests, moving the change through a release process. The phrase I like is, "It costs 3 days to do nothing."
posted by lowtide at 3:45 PM on March 9, 2021

This was an interesting read. This is the sort of bug that probably is in an issue tracker somewhere at Rockstar marked low priority. I have committed this sort of sin myself; sometimes you can figure out how to get something to happen inefficiently very easily, and just iterate the stupid thing to finish the job rather than come up with an efficient way to do everything at once. It's not that you want to do the bad thing, it's just that something else is more important and prevents you from spending the time to craft a $50 solution to a ten-cent problem.

Almost certainly this problem was introduced very late in the development cycle, because I know how I am, and if there was one of me at R* waiting minutes to load every time through I would've fixed the problem.
posted by axiom at 5:04 PM on March 9, 2021 [3 favorites]

bombarded with requests for new features that they can't get around to dealing with something that sort of works

There is a dangerous trap with software development where, rather than fix the things you have that are partially working, somewhat broken, but just about enough to count as "a version of the feature", you instead are assigned to work on the as yet entirely nonexistent features you need to pass the upcoming milestone. Rinse and repeat this about 20 times over several years and you have the monstrosity that is a "feature complete", but disastrously inefficient and flaky modern game.
posted by Jon Mitchell at 7:47 PM on March 9, 2021 [2 favorites]

I wonder if they would even need to do any coding to fix the strlen problem, and not just upgrade their JSON parsing library and recompile. It would be ironic if they had decided to write their own JSON parser in C because they thought it would be faster.
posted by L.P. Hatecraft at 8:03 PM on March 9, 2021 [1 favorite]

At one point, I worked on a team that was responsible for invoicing on accounts for a Fortune 100 tech company that you have definitely heard of.

As a tech company, our invoicing system was built on some... optimism... about what billable accounts would look like. Billable accounts might require chargebacks, or corrections to past invoices, or whatever, so at some point the design decision was to just process everything from the start. Easier than having to deal with all the nuances about month/year close boundaries, and laws, and accounting esoterica: you just do everything from the start and see where the accounts are. Seems foolproof, right?

But (#AccidentallyQuadratic) it turns out that reprocessing the entire world is a quick fix at first but becomes much, much more expensive when you have 20+ years of transactions that you need to account for. You end up reprocessing every transaction ever, every hour, because maybe something changed 19 years ago. (Of course, by law, it's virtually impossible for anything that far in the past to change! But you do it anyway because that's how the system was designed.) That batch job to update everybody's accounts that you expected to run once every hour, taking 60s to run, starts taking 5m, then 10m, then 60m, and then you start to get behind on your SLO for providing your customers with timely data on what they're spending with you. Worse, sometimes you can't even close the books for the month, or the quarter, on time, which leads to Finance breathing down your neck, which leads to a whole host of other problems. It gets slower and slower, depending mostly on luck. The amount of compute resources we had to feed that beast to keep it going was completely insane; it started to rival our company's main product, and I wish I could tell you what that product is, because the fact that an invoicing system would take up a similar amount of resources would blow your mind.

When I left that team, we had a huge number of people working on making a state-of-the-art, incremental processing pipeline. I hope that they've done it because it's been quite a long time, and I can't imagine being on the poor team that was responsible for feeding that beast.
posted by Excelsior! at 9:24 PM on March 9, 2021 [2 favorites]

I am thankful for systems like yours Excelsior!

I lead a team of data engineers and scientists, in collaboration with DBAs, SREs, and SWEs. We charge really good money to bring down your 48 hours, unreliable, and very expensive batch processes to way cheaper incremental or real time, with pretty dashboards to look at. Most of our clients are companies you have definitely heard about.

What took me a while to accept is that although I am far from a subject matter expert, I haven't coded for years and I was starting to lag behind even when I was coding, I've seen and made so many mistakes like this that I can smell them now, I am willing to fight the battle to prevent shitty code from making it to production. My value to the team is that I was a terrible coder 15 years ago, and I am happy with it.
posted by Dr. Curare at 9:06 AM on March 10, 2021

This is surprising only because this bug must have been in the path of the game's developers. They probably hate the load time as much or more than anyone. Usually that is the kind of bug that somebody just rage-fixes.
posted by qxntpqbbbqxl at 6:19 PM on March 11, 2021 [2 favorites]

... And, Rockstar responds with thanks and promises to fix...
posted by rozcakj at 3:36 PM on March 15, 2021

« Older Government is back: The first 50 days of the... | Ahead of her time Newer »

This thread has been archived and is closed to new comments

MetaFilter

How I cut GTA Online loading times by 70%
March 8, 2021 11:08 PM Subscribe

Tags

Share

How I cut GTA Online loading times by 70% March 8, 2021 11:08 PM Subscribe

Tags

Share

How I cut GTA Online loading times by 70%
March 8, 2021 11:08 PM Subscribe