Swift architecture at Uber

https://www.skilled.io/u/swiftsummit/swift-with-a-hundred-engineers

by tsycho • 9 years ago

305 135 9 years ago

xyzzy4• 9 years ago

It seems the Uber app was extremely over-engineered. Call me crazy but I don't think you need 100 engineers to recreate the front-end of Uber.

ReligiousFlames2• 9 years ago

Hire lots of people... complexity, arcaneness, process and knowledge-hoarding becomes vital to job-security.

tmh79• 9 years ago

also when you need to develop a new feature and it needs to go through A:B testing frameworks, localization (including Arabic Right to Left text/layouts), phased feature roll outs, 99.999% reliability, integration testing, planning for use cases anywhere from LTE/WI-FI in USA to EDGE network in developing countries, and you need to build the tools that do those things too. Clearly the 100 engineers are sitting there twiddling their thumbs.

htormey4• 9 years ago

Well your not taking into account all the A/B tests they are running, the fact that they have custom experiences for different locations (SFO, India, etc). I can easily see how they would get up to that number pretty quickly at the scale they are at.

venture_lol4• 9 years ago

At some point, consider the use of different apps altogether. Perhaps having a front loader to detect which local app should pop up based on information gleaned from the phone and GPS.

sudhirj2• 9 years ago

If you have the app, move your map pin to a different city / country to see how it changes. Lots of little and interesting changes depending on which part of the world you're in.

jjtheblunt• 9 years ago

He mentioned in the very beginning that there are city teams, so perhaps it's a number of people proportional (not exponential, like he says) to the growing # of cities.

panic• 9 years ago

You're coming at this backwards. They had 100 engineers already. They needed to design their software so that the engineers could all keep working on it. They explicitly say as much:

This application has served us very well for the past four years. But as we've expanded and exponentially grown our mobile engineering team, we started seeing breakages of the architecture, we started seeing problems where feature development would become pretty hard. We had to test multiple code-paths because we shared a lot of view controllers amongst different teams. We really started to hurt with the old architecture because it was written by two engineers and we had grown the team to over one hundred, at that point.

The actual quality of the software isn't as important, since Uber drivers and riders don't have a choice anyway.

perfmode3• 9 years ago

Uber riders have more than one ride-sharing app on their phone. If Uber can't match me to a driver, I'm still going to get where I want to go.

yawaworht12• 9 years ago

Any software from a big company like Uber, Facebook, Google, etc. is far more complex than what you see. There are typically dozens of features and experiments rolled out to some users and not others. There is also a lot of work necessary to serve so many different markets, such as localization and internationalization. Furthermore, in Uber's case they have a lot more iOS applications than the ones riders use. In terms of complexity, Uber iOS apps are probably comparable to Facebook's apps.

battwell2• 9 years ago

It's definitely a lot of engineers. But we cram a lot of features into the app. I expect that most people will only experience 10% of them. Consider: 1) We exist in a tonne of countries. And different counties often require different product optimizations and payment methods 2) We have our own map provider 3) We experiment with everything 4) We support countries that have terrible networking. That creates lots of challenges.

venture_lol4• 9 years ago

From an engineering perspective, that sounds uninteresting :)

Terrible networking? how many engineers does it take to implement a few flavors of retries? :)

Uber growth in term of getting drivers, that's what I want to know. It's not just ride subsidies, but the flow of investor honey helps, of course, but they do have some alpha growth managers.

unoti• 9 years ago

Call me crazy but I don't think you need 100 engineers to recreate the front-end of Uber.

Your sentiment is valid, but consider this: whenever you're working with a project with a ton of money on the table, it becomes possible to add small bits of functionality which more than pay for themselves in terms of return on investment for the engineering effort. This is why you can often write a basic version of a more established application in "a weekend". We've had extensive discussion and meditation on this topic on HN before[1]. Basically apps will grow and add features as long as the engineering effort will pay for itself (I suppose I use that term loosely in this case).

Also consider that the UX is different in different markets, and the reason for that is related to the first issue.

I sympathize with what you're saying, though, and couldn't help but think about Alan Kay's talk that we discussed the other day, about natural complexity vs artificial complication.[2]

[1] https://news.ycombinator.com/item?id=12626314 [2] https://news.ycombinator.com/item?id=14188759

sudhirj2• 9 years ago

It's basically tens / hundreds of apps rolled into one, with frontends loaded based on your currently stated or GPS location. They all share a common foundation, but features vary wildly depending on city / country.

perfmode3• 9 years ago

I would love to see an implementation of the router component.

MooMooMooney• 9 years ago

"Lastly, we started combining files, and we found out that combining all of our 200 models into one file decreased the compilation time from 1min35sec, to just 17sec. So we are like, "Hold on, this is interesting, combining everything into one makes it much faster." The reason for this is that, as much as I know, that a compiler does type checking for every single file. So if you spawn 200 processes of Swift compilers, it needs to 200x check all the other files and make sure that you're using the correct types. So combining everything into one makes it much faster. "

Good to know

flamedoge3• 9 years ago

It's probably time spent in the linker. The advice of using WPO without opt is verrry circumstantial.

slavapestov9• 9 years ago

No, it's valid advice. Non-WMO starts a frontend job for each file, WMO runs one job.

The frontend jobs do not share state, so each one parses all the files in the module.

In general the parser is very fast, and the type checker tries to only type check declarations in files other than the primary file when absolutely necessary, so it's not always O(n^2). But there are pathological cases you can construct today where the type checker ends up doing too much work.

symisc_devel• 9 years ago

The technique is known as Amalgamation and was first introduced in the SQLite2 source tree. In my company were we do embedded software[1], this shown to be a powerful technique. Not only, the compilation is extremely fast (A 1.3 megabytes of C code, took 9 seconds to compile on a core I3), but a modern compiler will be able to do additional optimizations on code when it is contained with in a single translation unit.

[1] https://unqlite.org, http://ph7.symisc.net

lanna• 9 years ago

Why don't compilers do amalgamation themselves as a pre-compilation step?

vernie• 9 years ago

How is this different from so-called unity builds?

asimpletune• 9 years ago

Awesome post!

santaclaus• 9 years ago

Why didn't they use react native?

eddieroger• 9 years ago

Why should they have? Native is a great, if not arguably, the best option.

rezashirazian2• 9 years ago

Fast forward to 9:53

twelve40• 9 years ago

...and there he talks about IDE crashes? didn't catch any mention of react.

Tanegashima3• 9 years ago

Because it's a high performance app that thousands of people depend daily for living.

Entangled• 9 years ago

Android engineers are more welcome now. Especially if they write Kotlin.

I love Swift, and I love Kotlin.

To any young programmer out there (I'm in my fifties), learn these two languages and you will be highly employable for the next decade, on top of the wave.

jacquesm4• 9 years ago

A decade is a pretty short time for an investment of that magnitude. Learn C (yes, I know), COBOL, Java, Erlang, Python or any other non-niche language and you'll hopefully be employable for decades, or at least until general AI rolls around.

WhitneyLand2• 9 years ago

The languages actually aren't really the big investment. It's really two things:

1) Learning new concepts in computer science. For example Swift uses a lot of modern concepts around type safety, closures, parallelism, synchronization, etc. These are a big learning curve but they are not specific to Swift and you'll notice other modern languages are adopting the same concepts.

2) The UI frameworks are fundamentally different. If you're an option expert at XCode constraints, storyboards, etc, none of the applies to android and vice versa.

I can tell you this, if you do bite the bullet and learn both very well you'll be surprised how much it helps you learn other systems more quickly because you've see all the concepts before.

TheAceOfHearts• 9 years ago

If you find Erlang a bit weird at first, try out Elixir. Its syntax is more familiar, so it's a bit more approachable. One nice perk is modules can be shared between both languages.

bshimmin• 9 years ago

If you're in your fifties, a single decade is about all you need to take you through to retirement age.

To be honest, I would've said going deep on JavaScript right now (React, React Native, etc) would give you a shelf life of at least five years.

Kotlin seems like a bit of a random recommendation - it's not in the TIOBE top 50, and a quick search for Kotlin jobs on indeed.com (the first job search site I found from Google) reveals several orders of magnitude fewer jobs than for anything mainstream (hell, even Haskell beats it comprehensively). Perhaps it's the next big thing though!

sandGorgon2• 9 years ago

With all due respect, I would dispute that. That would be JavaScript based - today React Native... tomorrow something similar.

Already there are large apps that are built around React Native. But more importantly, js as a language is becoming indispensable.

A few days ago, there was a discussion about Netflix - who specifically re-engineered it's back end infrastructure to let it's app engineers (who are js) write API services in nodejs. We are talking 40% of The Internet here ?

There is a large boost in hive mind productivity with one programming language . If Java 9 Truffle/Graal.js is as good as everyone says, then it is pretty much js all the way down.

And for those who think js is a shitty language (god knows I did), please try out ES2017 or Typescript - you will be pleasantly surprised.

xor1• 9 years ago

How about learn all 3

edgyswingset2• 9 years ago

Netflix is a polyglot shop which uses a number of different programming languages for different parts of their systems. Yes, Node and JS play a large part in that, but that really isn't the whole story.

unpopular11333• 9 years ago

I was hoping to find something about how the Swift architecture affected their ability to cleanly implement a tipping feature, since they've been a tad behind the curve on this very highly demanded functionality (just pinging my general group).

Humdeee• 9 years ago

The functionality of that feature is completely subjective and I'm sure top level management has had many, many meetings with that very topic as the headliner since the first weeks of development. They're very likely ahead of the curve.

valuearb2• 9 years ago

Pretty sure a tipping feature would be the single worst possible feature they could ever add.

beaconstudios3• 9 years ago

hm, so they rewrote the whole platform from scratch in <totally hip language of the month> and it didn't all crash and burn? That's kind of surprising - this is usually a really stupid idea because you often end up mostly solving the problems of the v1 architecture but introducing a whole bunch of different, equally painful problems - but with the added headache of the whole codebase being newish.

melling2• 9 years ago

Problems aside, Swift is the future of the Apple platform. It's a much less verbose language than Objective C. (no headers, type inference, no @, no ;)

https://h4labs.wordpress.com/2016/02/09/should-i-use-objecti...

beaconstudios3• 9 years ago

my point is that a full switch-over is usually a terrible idea. I don't know if an incremental change-over from ObjC to Swift is possible in the Apple ecosystem, but if it is that's almost always the better option in my experience.

anonred• 9 years ago

It may be the future, and I love the syntax, but I've run into many of the same problems mentioned in the article (long compile times, SourceKit issues, increase in binary size, etc.) with my own smaller apps. It's honestly frustrating, sometimes more so than the eyesore that is ObjC.

battwell2• 9 years ago

We were all skeptical about doing a rewrite too. A bunch of us have worked at companies that went through bad rewrites. And we've read https://www.joelonsoftware.com/2000/04/06/things-you-should-... :p

But the old Uber app had hit its age limit. It was build on top of technology chosen for a small number of features (ex: a single global DI component, lack of typing, a small MVC hierarchy). So we either needed lots of large migrations or a rewrite. The incremental migrations required to fix these issues would have been extremely disruptive, they wouldn't have gotten us an entirely refreshed UI and they wouldn't have given us a number of other benefits.

So we decided to do a rewrite. We had lots of engineers that knew the issues to watch for from the first time we wrote the app. And a handful of us spent months researching/building different architectures, static analysis and tooling that would ensure the rewrites success. We weren't going to repeat the same mistakes twice.

On the very first day the app launched it was more reliable and performant than the version of our app that we had been maintaining for years.

beaconstudios3• 9 years ago

I did assume that in your case the rewrite was valid - after all, you laid out the case. I was just surprised that it went as well as it did - I've seen so many rebuilds that went to shit that it was a genuine shock to see one go well.

jordansmithnz• 9 years ago

In regards to the tool mentioned that provides information about binary size contribution... ("If you want to see this open-source. Just scream out loud")

I am screaming out loud. Please open source this!

dankohn1• 9 years ago

I feel very appreciative to have Uber working through all these bugs in the tooling so that the rest of us can take advantage once things are more reliable.

fred_is_fred• 9 years ago

It's really irritating that Apple picked the name of an existing software product for their language. When I saw "<large Company> swift architecture" I was pretty excited to see how they were using object storage.

ktzar• 9 years ago

and iOS had been the name of Cisco hardware OS for ages... They don't really care.

mgoblu3• 9 years ago

Does anyone have any other good articles/examples of maintaining large iOS applications like this? I found this super helpful and interesting to my current project so was also interested in other cases like this

rudros• 9 years ago

https://www.objc.io/issues/22-scale/ has some nice articles.

hota_mazi• 9 years ago

The reason for this is that, as much as I know, that a compiler does type checking for every single file. So if you spawn 200 processes of Swift compilers, it needs to 200x check all the other files and make sure that you're using the correct types.

I'm a bit baffled by that. Is the Swift compiler that naive?

Surely you know how to assess the number of processors/cores on your system and you spawn threads in a way that doesn't lead to diminishing returns. You use a bounded thread pool and you stay within these bounds.

Seeing such a speed up from simply merging text files is really puzzling to me. You have to type check the code anyway, surely the overhead of opening a new file is completely negligible compared to running this type checker on the same file? Especially since all these instances of the type checkers have to share a lot of data anyway.

Swift is very cool and I'm excited to see it and Kotlin become our next generation languages for mobile, front end and back end alike, but it seems to me the Swift compiler is still very immature.

edgyswingset2• 9 years ago

The Swift compiler has some serious issues with the way it does type inference. Try compiling code with a line of basic arithmetic on a few numeric literals.

eridius3• 9 years ago

That has literally nothing to do with what the parent comment is talking about. And that's not naivety either. That's because the way type inference, operating overloading, and literals work together ends up being a combinatorial explosion.

mahyarm2• 9 years ago

There are a lot of 'obvious optimizations' like this lurking in swift. I get the impression they are somewhat engineer constrained in an interview with chris latner.

andrecl• 9 years ago

Is there a link where Latner says the Swift team is engineer constrained?

Someone• 9 years ago

"So if you spawn 200 processes of Swift compilers"

I don't think that implies those processes all were spawned at the same time (or even that Apple's compilation process did the spawning. Given the effort they spent on their build processes, I do not rule out it is their tooling that does the spawning)

The thing is: all these processes are independent of each other. Uber likely has tons of files that are imported into many, many files. Each compilation process that imports such a file has to individually parse it.

I can see two ways around this: reusing intermediate results between processes and using fewer processes.

The first is like precompiled headers in C/C++. The second could mean having a queue of to be compiled files and a fixed number of compilation processes that pick up files from it.

Risk in both cases is that unintended state may leak between compilations. For example, compilation flags may be different for files f and g. Does that mean g can't use preprocessed module m generated while compiling f? Because of that, I would go for something like precompiled headers, because it makes it easier to reason about what information flows between compilations of f and g.

I don't know whether that would be sufficient, though. The biggest concern I have w.r.t. Swift is that all its cool features, combined, make it essentially impossible to build a fast compiler (Yes, languages such as C# have most of them and aren't that slow, but adding protocols and overflow detection (which you may want to optimize away a lot, even in unoptimized builds) may just tip the balance)

nnain• 9 years ago

It's so great that they shared all these findings: I hope the XCode team takes notice and fixes some of the nagging compile time and indexing issues!

bsaul2• 9 years ago

Unfortunately the xcode team knows that for a long time, since people inside appl have to use that tool themselves. The reason they don't solve the issues crippling the tool is probably like everywhere else : the codebase has become unmanageable and nobody wants to budget a rewrite.

andymatuschak2• 9 years ago

One other big contributor to app size we've noticed at Khan Academy: extensive use of value types, particularly ones whose fields require more than a couple words of storage.

The big picture observation is that for value types with storage larger than a few words, several instructions must be emitted per call and per storage word because they cannot be passed-by-value in registers. And Swift often emits more calls than you can see (e.g. thunks, protocol witnesses, weak sentinels, etc). This is not new to Swift—the same requirement exists when passing large C values around—but we use value types a lot more in Swift for various reasons, so the issue becomes more salient.

In terms of remediation, I audited our app for all structs larger than a few words. I just did this manually; there were something like 120 structs to look at. For each, I converted it to a class then evaluated the impact on generated code size. Only four structs had meaningful impact on generated code size (to the tune of ~13MB), and happily, they were fully immutable, so they retained their value semantics even when converted to classes. If they had not already been fully immutable, I would have had to spend some time either adapting the classes to achieve value semantics, or adapting their clients to tolerate reference semantics.

Then I audited our app for all enums larger than a few words. These can be made pass-by-reference by using Swift’s `indirect` feature, which implicitly boxes associated storage. We had one enum for which this made a substantial difference, to the tune of several MB.

Then I had to make sure runtime performance hadn’t been too badly damaged by all the new dynamic allocations and indirections. In the end, I observed nothing noticeable. We don’t have formal repeatable performance tests, though—it would have been interesting to see the impact on those.

C++ has this issue, too; it largely handles it by using reference arguments when consuming large stack values. In the future, it’s possible for Swift to optimize many cases (especially intramodule) where this occurs by allowing deeper stack frames to reference values types stored in parent stack frames when it can prove it’s safe. Rust has a lot of fanciness here you might find interesting!

slavapestov9• 9 years ago

There is some work ongoing to improve this situation in Swift 4.0. Take/copy/destroy operations for enums are now outlined and shared between call sites, and a further improvement is in the works to pass large structs indirectly even when they are otherwise loadable (no address-only fields).

Another recent improvement is copy-on-write existentials. There's no code size win here, but it improves runtime performance by avoiding copying the payload when passing existentials by value.

andymatuschak2• 9 years ago

These are both wonderful pieces of news! Thanks for sharing.

steveklabnik3• 9 years ago

Servo has unit tests that check this kind of thing https://github.com/servo/servo/blob/412f4bbb6ff075de23d1bbfd...

I think they have some more infrastructure that can audit stuff automatically in the same way you did, but I'm basing that off of a conversation I had and I'm not familiar enough with the codebase to find it; maybe it's just this set of tests.

ed_balls• 9 years ago

.Net recommendation is to use value types for values that has maxium of 128 bits.

Aqua_Geek• 9 years ago

We ran into this issue in the rewrite as well. We ended up having to convert all of our models from structs to classes (similar to your case they were immutable, so it wasn't a huge difference).

bsaul2• 9 years ago

Funny he didn't mention a command line tool that gives compile time per function ( explained here http://irace.me/swift-profiling ). That proved to be the greatest help in my case to reduce compile times drastically.

He did mention running a tool to add explicit types everywhere, but it's very often a matter of just writing the most generic ones. Maybe not in uber case, but for everybody else you should try it.

dep_b• 9 years ago

Especially when having chained calls like .filter .flatMap .reduce etcetera having an explicitly declared types will take a lot of burden off the compiler.

Apart from that dictionaries and arrays should be typed.

Also some overloads are really hard to process, especially the '+' operators are pretty expensive when used on non-integers and not (previously) strictly typed values.

Never do something like "This is a " + birdName + " on it's nest" but use the "This is a \(birdName) on it's nest" method instead. But sadly it also applies to libraries like Cartography that rely on such overloads to create more elegant declarations.

discreteevent• 9 years ago

"First, you have to be aware that structs can increase your binary size. If you have structs into lists they are created on the stack and they can increase your binary size."

I don't get this. Is it saying that structs can increase your binary size and as a separate issue they are created on the stack. Or is it saying that because structs are created on the stack they can increase your binary size? How would that work if stack allocation is something that happens at runtime and affects your memory footprint rather than binary size? (I don't use swift so I might be missing something here)

_pmf_• 9 years ago

In C, static structs that are not zero-initialized or uninitialized (i.e. that are statically to something else than zero) increase the binary size (they go into the data segment, i.e. even if you only initialize one field of a struct, the binary contains a full image of the full struct); this has nothing to do with the stack in C, but I don't know about Swift.

Swift architecture at Uber

Related Stories