I cut GTA Online loading times by 70% (2021)

https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-times-by-70/

by colinprince • 3 years ago

215 99 3 years ago

nee.lv

el_duderino• 3 years ago

Discussion from a previous submission: https://news.ycombinator.com/item?id=26296339

xxs• 3 years ago

The link to HN was added to the blog post, itself: https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-times...

two_handfuls• 3 years ago

Amazing story!

m-a-r-c-e-l• 3 years ago

Rockstar fixed that bug because of this guy... Awesome!

rightbyte• 3 years ago

One could argue the bug is in scanf on Windows?

charcircuit• 3 years ago

No, the bug is not in Windows. scanf can not cache string length as it can't guarantee x, and x + offset are the same thing, nor can it guarantee the string was unmodified since the last call.

Windows provides _snscanf_s if you want to keep track of the string length yourself instead of having it recompute it each time.

pjc50• 3 years ago

The C standard library makes no performance guarantees. Besides, scanf is awful and should be avoided wherever possible.

flohofwoe3• 3 years ago

The problem was that the JSON parsing code was doing exceptionally stupid things in the first place (like calling scanf), which doesn't matter on small JSON files used in testing but exploded in production once that JSON file was growing to several MBytes. Just dropping in a different JSON parser library was probably enough to fix the problem on R*'s side.

Joker_vD2• 3 years ago

Most of the standard C library implementations, including FreeBSD's libc [0] and glibc [1] have sscanf implemented like that, by calling fscanf on a dummy FILE object (with its size populated by strlen() at every call, no caching).

Of course, there are implementations whose authors thought about that and decided to do the reasonable thing instead, e.g. musl [2] and Plauger's old stdlib [3].

[0] https://github.com/lattera/freebsd/blob/master/lib/libc/stdi...

[1] https://github.com/bminor/glibc/blob/dff8da6b3e89b986bb7f6b1... , https://github.com/bminor/glibc/blob/dff8da6b3e89b986bb7f6b1... , https://github.com/bminor/glibc/blob/dff8da6b3e89b986bb7f6b1... — two additional files and five times as much code compared to FreeBSD's implementation for pretty much the same functionality, wow.

[2] https://git.musl-libc.org/cgit/musl/tree/src/stdio/vsscanf.c

[3] https://github.com/wuzhouhui/c_standard_lib/blob/master/STDI...

ErneX• 3 years ago

And paid him 10k as bounty.

Panzer04• 3 years ago

One wonders how many millions (or tens of millions) they lost from people giving up over a 10 minute loading screen...

thiago_fm• 3 years ago

How do I become that good in debugging stuff like this? I'm guessing he works with gamedev and does this sort of debugging and reverse engineering regularly?

nolok• 3 years ago

Learn to use a performance profiler and then a debugger/disassembler/whatever fit you needs. The profiler tells you where the slowdown happens, the rest then let you figure out what actually happens.

By learn to use I mean really do, it's like refactoring tools, it's sort of boring and tiring to learn those, you end skipping large parts because "yeah I already know / it's obvious", then you need one day for an emergency / something right now and feel like you're fighting against it.

For those three (and some others), the key to being very good at it is learning all the small tricks and details, when you don't need it, so the day you do it lets you skip 90% of the work.

As an overall point, I'm sure many people have various different experiences, but for myself while being in high school I needed to parse "replay" files for some games to extract and process information and allow players to post their replays to get stats on web forums, and it helped me to learn a great deal about how to properly reverse engineer and figure out stuff, things that I have no idea how I would have learned otherwise.

beezlewax• 3 years ago

Excuse my ignorance... but what refactoring tools do you speak of? I'm accustomed to the basic ones in VS code but I have a feeling I'm missing something greater judging by your comment

etra0• 3 years ago

I learned some of this stuff being motivated by doing mods that interacts directly with game's memory.

I mostly build free-cameras, which allows you to detach the camera from characters, and for that I had to learn x86 assembly, some disassembler tools and how compilers generally work.

I think it's super rewarding because while you're seeing bits and bytes, and crashes, most of the day, at some point you can see you achieved something by being able to freely move in a restricted world.

I was surprised how far you can get with Cheat Engine. Its disassembler is really good, and you can do injections directly, if you then want to move to your own advanced stuff, you can build frameworks to do all of that using Windows APIs.

alecco• 3 years ago

Find reverse engineering discord servers or IRC channels. READ THE FAQ/WIKI FIRST and try to fit in.

Don't jump in with "hi please teach me" or asking questions in the FAQ/wiki or found easily on the web.

Try to give back to the community (like fixing/improving the FAQ/Wiki) as early as possible.

aronhegedus• 3 years ago

This is my favorite reverse engineering article! Feel like I've seen it on ycombinator a few times now

mywacaday• 3 years ago

Remember reading this way back, glad to see he/she got a 10k reward from R*, great piece of debug work and a bit of inspiration to improve my own skills.

bezier-curve• 3 years ago

10k seems low considering millions of people wasted millions of hours of their time looking at a loading screen.

lallysingh• 3 years ago

I agree with you, but let's play devil's advocate. People waiting already paid for the game, so the business value here is hard to justify. And $10k is a decent amount of $ for the labor involved.

bluedino• 3 years ago

Like the story about Steve Jobs wanting the Mac to boot faster

"You know, I've been thinking about it. How many people are going to be using the Macintosh? A million? No, more than that. In a few years, I bet five million people will be booting up their Macintoshes at least once a day."

"Well, let's say you can shave 10 seconds off of the boot time. Multiply that by five million users and thats 50 million seconds, every single day. Over a year, that's probably dozens of lifetimes. So if you make it boot ten seconds faster, you've saved a dozen lives. That's really worth it, don't you think?"

https://www.folklore.org/Saving_Lives.html

Phil_Latio• 3 years ago

10k reward is a joke. Reminds me of the guy who found you can submit a sitemap for a different domain in Google Webmaster Tools (if the site suffers from open redirect issues). You would then get huge search traffic for a brand new domain. He first got $1337 reward which was then "upgraded" to $5000 for a bug that's worth millions.

asadalt2• 3 years ago

are loading times any better now? i haven’t played in a while

ghusbands• 3 years ago

The article says they are.

asadalt2• 3 years ago

oh I meant these days on gta online. I had checked last year or so and it was still pretty terrible with frequent drops etc.

kramerger• 3 years ago

Great article,

I am baffled nobody at R* cared to investigate a 6min load time that was there for years. Doesn't sound very AAA-gamedev to me.

prox• 3 years ago

If no one carries the responsibility, no one will care. Maybe the team who worked on that moved on and what’s done is done. Not sure if R* has a team that does optimization only?

Hamuko• 3 years ago

I imagine the focus is on creating new things instead of fixing old things, since that will have more obvious and marketable impacts. The whales might tolerate longer load times over stale content.

spuz2• 3 years ago

I also cannot explain it. If I was a dev on a system that had a 6 minute loading time, I'd be stopping what I was supposed to be doing and debugging it after a couple of days of suffering through it.

DANmode• 3 years ago

That's the fun part: their dev systems almost certainly are not console dev kits, or consumer PCs.

They're absolutely jacked workstations that probably loaded it in a blazing 2.7 minutes!

charles_f4• 3 years ago

I started working on a codebase where the git repo is 56gb. Turns out that it's a bunch of binary files that were committed and updated a buncha times more than 2y ago. They're not even in the main branch anymore. Nobody in the main team working on that seem to have cared enough to fix it before.

Don't underestimate the power of complacency

flohofwoe3• 3 years ago

I can exactly imagine how it would happen in a large company:

1. It's a boiling frog problem which developed over several years. It started fast when the JSON file was small but then got a little bit slower each time new content is added.

2. It's also not exactly an "interesting" part of the code base. IIRC it's a JSON file which configures the ingame shop(?). This was probably delegated to a summer intern to implement in a couple of days, maybe in an entirely different department from the people who develop the actual core game.

3. New shop items are probably added by a content team without programmers in the loop who would know by instinct that something must be wrong. People on that content team also most likely don't stay on that team for very long, which together with the boiling frog problem above means that everybody thinks "it's always been like that".

4. Any specific feedback from the outside probably gets lost in the noise of other requests.

5. ...plus a general 'not my problem' attitude common in big organizations and an always-full kanban board ;)

johannes1234321• 3 years ago

Plus: Most developers probably don't use the full store bundle and skip it, thus don't notice it at all.

rozab• 3 years ago

This isn't some obscure b2b product, this is GTA5. Surely somebody at rockstar had, yknow, played the game? And noticed they were losing hours of their life on a loading screen? Hell, it's so long you could file a complete bug report in that time.

My theory is the fix existed internally but was never merged for weird political reasons. Some have guessed it was due to advertising being done on the loading screen. Or it could have been pure dysfunction, like how fixes for all the notepad.exe problems apparently existed at Microsoft for decades, but it took a full rewrite to have working undo.

sonicanatidae3• 3 years ago

Why? Clowns STILL buy GTA by the truckload.

wruza3• 3 years ago

Why clowns?

kowlo• 3 years ago

It's a shame that's their only post (https://nee.lv/)

cedws• 3 years ago

Agreed. I was rather hoping to see them post something about the GTA V source code leak.

stevage• 3 years ago

That's insane to achieve that without access to source code.

It's also completely baffling that a company would invest such resources building such a complex game and be fine with 6 minute load times, caused by such ... dumbness. Like selling a sportscar with a 5 litre petrol tank.

DanielHB• 3 years ago

the people maintaining GTA online systems are probably not the same ones who developed the main game (or even the initial GTA online systems)

They are probably just adding more content, but the original dev's algorithm for loading that content didn't expect a bazillion new cosmetics

wwalexander• 3 years ago

Yep. The entire game is like this, massive amounts of content clumsily tacked onto something that they clearly didn’t realize would become so popular. This also shows in the interface, where basically every piece of gameplay is triggered by plumbing through the massively overcrowded Quick Menu.

jvmboi• 3 years ago

My sneaking suspicion is that the real reason that everything is so slow is that noticeable loading or processing times give the software a feeling of heft. It strokes our ego as programmers because if something takes a while to complete then it must be very complicated and important.

rowanG0773• 3 years ago

Interesting. I have never thought about it like this. To me it always feels like bliss if something finishes instantly,

ionwake3• 3 years ago

I cant understand why there wasnt mass firings across managements and a post mortem, aswell as a hiring of the guy who fixed it, 6 figure reward and a new team setup to troubleshoot final build performance problems.

But I can understand thats its normal large companies are so incompetent, and at an older age I appreciate it gives a chance to the little guy or company to rise up, for now, until AI starts to catch these issues and large inefficient corporations can finally solidify their place in the hierarchy.

bena5• 3 years ago

Because, and I'll say this in the spirit of your recommendations, that's fucking stupid.

The problem was sscanf calling strlen. You're going to fire your management and development team because one dev used sscanf where it's probably appropriate?

Then hire a guy who may or may not have actual game development experience and give him $100k because he took who knows how many hours to trace through and find this thing. The guy himself the fix would probably take a day to implement.

Finding it could have taken longer.

You want to call Rockstar incompetent, but you're simply going to ignore everything else that goes right in not only GTA Online, but across all of their games.

jpeter• 3 years ago

Loading times for GTA Online are still insanely long on PC

haezee• 3 years ago

You must've never played before this was fixed.

oscarteg• 3 years ago

Are you using an HDD maybe?

slifin• 3 years ago

Yeah I remember this blog post the first time it made the rounds

Played recently and thought Rockstar must not have actioned the advice in the blog

But no it's just it is still slow but they did fix these issues kudos to them but performance still needs attention

People who play this regularly have their phones out/youtube videos on waiting for these load times

whatyesaid2• 3 years ago

I mean GTA Online is a microtranscation/grindfest experience, mostly to milk kids of their money. It's not really worth the time due to how grindy it is.

Most PC gamers play multiplayer mods like RageMP that allow connecting to servers with well-made scripted gamemodes like RP, but also stuff like death match where hackers are actually banned I'd guess.

skywal_l• 3 years ago

FYI, list of tools used:

- Luke Stackwalker: For stack sampling (perf analysis)

- ???: industry-standard disassembler

- Process Dump: Dump process memory to file (for obfuscated code analysis)

- x64dbg: For debug stepping.

- MinHook: For patching/modding a binary executable.

whackx• 3 years ago

???: industry-standard disassembler

guess that's IDA Pro

jasondoty• 3 years ago

Definitely IDA Pro.

It is an interesting case in piracy, as being the go-to software for reverse engineers, it also has the biggest target on its back for software crackers. Despite this, newer versions are not always cracked and released publicly so quickly. It seems each license owner gets their own custom watermarked build of the software. The crack of version 7.7 was leaked from Think-Cell, the excel add-ons company. 8.3 was recently seen released, with an anonymous supplier and Vietnamese cracking group behind it.

jcmontx• 3 years ago

I remember reading this article when it first came out. It still gives me the same feeling of inadequacy. There are some people walking around us who just belong to a different league.

NikolaNovak• 3 years ago

Always a fascinating read when it pops up on HN :)

One thing to note - assumption that one might make is that "Rockstar shipped obviously ridiculously suboptimal code; how did this ever get past testing??"

However, based on the author's discussion of the code, and the issue being in inefficient parsing of an online store content, it is likely that this not only passed testing, but was sufficiently efficient in production too... upon go-live, when the store content database was small.

This likely grew slowly, and was especially bad for a specific subset of users (CPUs with poor single core performance). They may even have had trouble replicating it in non-production environments depending on their refresh policy and whatnot.

This is not to excuse it, of course; but I've seen a repeated pattern where a not-perfectly-optimal algorithm works well enough in development, testing, and initial production... then blows up in production with growth a couple of years later.

(In particular, I deal with a lot of complex SQL in my day job, and they have a nasty habit of going exponential with data growth. If you're implementing a greenfield application with no objective knowledge of its future size and growth patterns, it may be littered with trapmines despite reasonably good efforts to not make it so. Especially dangerous with "Big Bang Go-Lives" - though a game may have internal Agile sprints, I'd still consider it overall a bing-bang go-live in that majority of its content will go live at the same time, and then you get to see how your 100s/1000s/10,000s of SQL/code/libraries/etc behave:)

whatyesaid2• 3 years ago

The loading times were atrocious even at launch though for Xbox 360. It wasn't a case of content piling up, they added a lot but probably a lot of their item catalogue was there from launch.

There was also server instability near launch which it may have been thought to be linked, as in we thought we were waiting 10 minutes for a server. But any actual server issue was probably more demand/load scaling based.

twelvedogs• 3 years ago

yeah it was definitely a case of not wanting to spend a comparatively small amount of money on dev time to even look into it

it was obviously not network based as connection speed didn't seem to change the time required

atom_arranger• 3 years ago

https://www.rockstargames.com/careers/openings/position/5761...

"Scrum", "Agile", "Lean".

The devs were may have been too busy "sprinting" and getting their assigned tickets done to think about higher level concerns like "is this good? can we make things faster or better?".

I think your points also stand though.

asylteltine• 3 years ago

I love reading this every time it gets posted. This guy is a genius and rockstar should feel bad.

vicktorium• 3 years ago

i read this article when it happened some time ago

this really makes me think:

if this obvious flaw exists ... imagine what other, more complicated and opaque mistakes exists and we're not aware of it ...

just food for thought

2OEH8eoCRo0• 3 years ago

Is there a safe way to do this and not get your online accounts banned? I'm interested in tinkering in this way but I'm not a cheater and I don't want to be confused for a cheater and banned.

robblbobbl• 3 years ago

Good job :)

mcint• 3 years ago

Other nice thoughts on it

I cut GTA Online loading times (2021) (June 9, 2022 — 476 points, 213 comments) https://news.ycombinator.com/item?id=31681515

How I cut GTA Online loading times by 70% (February 28, 2021 — 3883 points, 699 comments) https://news.ycombinator.com/item?id=26296339

I cut GTA Online loading times by 70% (2021)

Related Stories