I built Excel for Uber and they ditched it

https://basta.substack.com/p/no-sacred-masterpieces

by robdimarco • 3 years ago

753 514 3 years ago

basta.substack.com

I get a strangely dystopian feeling from this article, like it's almost about a character in a black mirror episode

pagnol• 3 years ago

I reflexively swiped left when the inevitable newsletter modal started to appear at the bottom, so didn’t finish reading the article, but would like to know what gave you the dystopian vibe?

Tao33005• 3 years ago

You should have kept going.

Spoiler alert: It turns out they were fulfilling a Babylonian prophecy the whole time. The whole development cycle was a complicated sacrifice to Marduk.

bastawhiz27• 3 years ago

Man, what a commentary on my career

lcnPylGDnU4H9OF• 3 years ago

Alternatively, they just accidentally (I assume) called you a good storyteller.

Vt71fcAqt72• 3 years ago

Is it possible that you thought this because the words "black mirror(ed)" are mentioned in the article iteslf?

Tao33005• 3 years ago

Starring Excel as the pig.

jhaenchen• 3 years ago

That's because everyone in this comment section is giving wisdom about how to deal with the emotional reality of being abstracted from the fruits of their own labor. "Don't get attached to your work" is dystopian.

dublinben• 3 years ago

Is it common to take code written for one employer and reuse it for another?

arcbyte• 3 years ago

Some of us will strategically write generic code on our own time and machines and import it to employers machines when needed and customize it. For instance, how many times do you really need to write a spring OAuth server that integrates with LDAP? Or the guts of a simple CRUD app?

Kwpolska• 3 years ago

how many times do you really need to write a spring OAuth server that integrates with LDAP?

I would expect this to be a library somewhere.

import it to employers machines

How would you prove that you write the code long before getting hired by them?

contravariant4• 3 years ago

Of course those of us familiar with copyright law would probably not admit to doing so publicly, right?

lph2• 3 years ago

No. Most companies have employees sign paperwork that prohibits them from doing exactly this.

crazygringo2• 3 years ago

Nope, it's extremely illegal. Unless the first employer gives you explicit written permission to.

Why anyone would run the legal risk of stealing IP from their old employer, to benefit not themselves, but to benefit their new employer... is beyond me.

nvarsj• 3 years ago

And then brazenly admitting it on their blog which hits the front page of HN... that takes some cajones.

choppaface4• 3 years ago

Well this story is about Uber, and as far as Uber goes this instance of code copying was not nearly as consequential as what some of their other employees did https://en.wikipedia.org/wiki/Anthony_Levandowski#Civil_laws...

Znafon• 3 years ago

"Nothing came of it, but I took the code and shoved it into my back pocket for a rainy day.

My idea was to take this code and spruce it up for Uber’s use case."

"My first reaction was to publish the code on Github."

I’m very surprised by this, isn’t the code property of Box, or Uber? The author does not mention their authorisation before releasing it under MIT license.

sidewndr46• 3 years ago

I believe this kind of story is the kind that gives most legal counsel nightmares.

pc864• 3 years ago

Especially the brazenness with which the author basically says "if they want to sue me for this verified and admitted IP theft, they can."

Sure, they probably won't. But they might. And if they do, you'll lose immediately. Seems like a pretty high risk no reward scenario.

michael1999• 3 years ago

Uber and the people they hired never struck me as particularly concerned by things like "laws" and "property".

pavlov• 3 years ago

Oh, you’re missing a few qualifiers. They’re not concerned with laws applied to them, and other people’s property. But in all other cases they’re big believers in law and using it to protect their property.

PNWChris• 3 years ago

The Levandowski lawsuit comes to mind https://en.wikipedia.org/wiki/Anthony_Levandowski#Civil_laws...

---

Edit:

Since I enjoyed OP's story, I thought I should clarify a bit.

I'm speaking broadly of how I remember (from the outside) Uber's fast-and-loose IP attitudes in the 2010s.

I don't think OP did anything of a similar sort. From comments here it sounds like they used some code they built in their free time that a previous employer didn't want.

At Uber it sounds like they asked and were permitted to post their no-longer-needed code to GitHub. It's got its own GH org and everything.

This whole chain is legally risky (I wouldn't do it and would strongly advise others not to do it).

I feel OPs actions are not Ethically Wrong, though. I wouldn't enjoy living in a world where OP gets sued for this, since it sounds like nobody at work wanted the work and it's not giving competitors an advantage. I won't claim the world isn't like that, though.

I really wish I could share OP's attitude and sense of ownership. I built something really cool (entirely in my free time) for a previous employer's hackathon. That code lives on some server they own now, possibly deleted. I deleted my copy after submitting it to the hackathon because I didn't want to risk anything. Company lawyers make just building things for fun feel so risky! It takes the soul out of our work.

bastawhiz27• 3 years ago

Author here. The code was originally written outside of work hours. I offered the code to Box and they didn't want it.

If Uber wants a few thousand lines of JavaScript from over half a decade ago that didn't originate with them and that they used for less than a month, they can send me a letter.

tempaway857512• 3 years ago

... Nothing came of it, but I took the code and shoved it into my back pocket for a rainy day ...

You can't really do this. Depends on your employment contract but code you write for an employer is usually copyright to them

... My first reaction was to publish the code on Github ...

You can't really do that either.

continuitylimit• 3 years ago

So come on man, let’s be honest here. I got serious sacred masterpiece vibes from this story.

This reminds me of some Hindu parable about people who let go of possessions and head out to become ascetics. So there is this wealthy man and wife and the wife is all upset because her brother keeps insinuating that he’s gonna go ascetic and cut loose. The husband tells her to stop her crying and don’t worry about it, he ain’t going to do it. The wife asks him: ‘but how can you be so sure?’ Because, the husband says, this is how you do it, and then and there he rips open his shirt, tells her “you’re my mother” and heads out to the woods.

brtkdotse• 3 years ago

they can send me a letter.

Why am I reminded of this meme?

https://amp.knowyourmeme.com/memes/what-are-you-gonna-do-sta...

treesknees2• 3 years ago

“half a decade” is practically yesterday in most codebases.

dclowd99013• 3 years ago

That’s really not how it works. Ostensibly by showing it to box, it was written for box, and they would be well within a standard of legal standing to make your life a living hell for taking it elsewhere. You should really be more careful with what you say.

Why would you wontonly open yourself to legal liability? You say “they’re free to come after you” but you really _really_ don’t want that. Ive seen that happen to friends and the stress almost killed them.

slim• 3 years ago

I just want to say I am sorry you have to cope with people trying to convince you you are not free for no reason

mihaaly2• 3 years ago

Depends on the contract but the most I seen stipulate not working hour but context of the work. If it was made by instructions of the company for the company and you got remunerated for that then it is not yours to do whatever you please with it. Better check your contract in details.

Example: "All Intellectual Property Rights with regard to Developed Materials will be exclusively vested in and owned by the Company." (with additional data protection and confidentiality clause protecting company property)

gberger• 3 years ago

Sergey Aleynikov (born 1970) is a former Goldman Sachs computer programmer. Between 2009 and 2016, he was prosecuted by NY Federal and State jurisdictions for the same conduct of allegedly copying proprietary computer source code from his employer, Goldman Sachs, before joining a competing firm.

https://en.m.wikipedia.org/wiki/Sergey_Aleynikov

kgilpin• 3 years ago

Right. This is so risky. Having to use your personal resources to defend yourself against a lawsuit brought by a large corporation - really sucks.

_lvbh• 3 years ago

I find it seriously disgusting that we have given rights to corporations so that they can sue someone over work they did in their own free time.

renewiltord• 3 years ago

That's a tremendous war story. I _love_ it! Doing a thing that's considered unreasonable. Knowing that total value is integral of instantaneous value over time (so short high-impact projects can easily outvalue long-lived projects).

constantly• 3 years ago

I wouldn’t try moving code written for one employer and using it another employer. Unfortunately for me my employers have been somewhat litigious; thankfully Uber doesn’t have that reputation.

itsthecourier2• 3 years ago

"When I met the members of the Crystal Ball team, it was about roughly ... people"

how many people there were at that point, @bastawhiz?

romnon• 3 years ago

people be rough

bastawhiz27• 3 years ago

Sorry, obvious typo! It was four people.

erulabs• 3 years ago

I read this article looking forward to the complex bespoke code to be ripped out and deleted - but the author clearly grew as an engineer in a way I didn’t expect:

Sometimes that’s just how it is. The devops saying “Cattle, not pets” is apt here: code (and by proxy, the products built with that code) is cattle. It does a job for you, and when that job is no longer useful, the code is ready to be retired. If you treat the code like a pet for sentimental reasons, you’re working in direct opposition to the interests of the business.

A lot of code is fun to write. A lot of problems are fun to solve. But a business, especially a startup, needs to stay razor focused. My entire career is effectively to sit in meetings and tell young, passionate engineers not to build things. It’s a bit depressing, but it’s also vital.

A good engineer can solve any problem with clever code. A great engineer knows what problems aren’t really problems and probably an XLS download link updated daily would have been fine.

Retric2• 3 years ago

Exactly, in 2016 there was several off the shelf options for doing the exact same thing. It’s a perfect example of a young engineer feeling a huge accomplishment from reinventing the wheel, and then realizing the clever solution wasn’t actually worth anything like the effort required to create it.

I had a long conversation to convince someone not to go down that path in 2006, and I am sure someone’s going to do it in 2026.

Pausing to think: I wonder how someone else solved this exact problem is such a huge part of how you grow as a developer I wish schools would focus more on it.

yard2010• 3 years ago

I would say what you talk about is experience. To have an experience, you must go this path to realize what to not do. It feels like a catch 22 kind of thing.

emmo2• 3 years ago

Give the time and labour you're paid for, don't give emotional energy.

stocknoob• 3 years ago

Short term yes, but long term, life's too short. Pursue FI, especially if you're a craftsman, so you can work on what brings you joy, without an expiration date.

"Cattle, not pets" may be a good way to run a business, but not your life.

sublinear4• 3 years ago

(from the article) - Having Excel in the browser was a useful solution, but the problem wasn’t showing spreadsheets in the browser: the problem was getting a specific UI delivered to the right users quickly.
(from the above comment) - A good engineer can solve any problem with clever code. A great engineer knows what problems aren’t really problems and probably an XLS download link updated daily would have been fine.

I saw the bullet list further down the substack page and it's still not good enough for this level of requirements gathering. Those questions describe the scenario, but asking them would not have arrived at this simple solution. Checklist thinking is a crutch and just overcomplicates the problem. All the signals here were organizational and social, and not a matter of improving a process.

This should be obvious, but people who are not involved with implementation details can't answer questions about implementation details.

"Just make it like Excel" is a super low quality answer from someone who has a completely different set of objectives. The only way forward would have been to consult with someone closer to the actual users and counter-argue from there. What's missing here is the courage to recognize weak assumptions and deliberately avoid writing any code until enough details are pinned down to get to an agreement from all parties, not just say yes to the person "in charge".

bastawhiz27• 3 years ago

"Just make it like Excel" is a super low quality answer from someone who has a completely different set of objectives. The only way forward would have been to consult with someone closer to the actual users and counter-argue from there.

The only contact we had with "actual users" was over WeChat because they were on the other side of the planet.

What's missing here is the courage to recognize weak assumptions and deliberately avoid writing any code until enough details are pinned down to get to an agreement from all parties, not just say yes to the person "in charge".

Uber was pathologically bad in this sense. There was no time to get details pinned down. We had a product to ship in two weeks for non-technical stakeholders. If we didn't, the stated consequence was millions of dollars in losses to the business. Throwing up your hands until you get product clarity when you know you can solve the problem as-is is a great way to find yourself with a PIP.

fader• 3 years ago

My entire career is effectively to sit in meetings and tell young, passionate engineers not to build things. It’s a bit depressing, but it’s also vital.

This was the single most impactful thing I learned in my early career. I was building out monitoring systems for an in-house service we hosted on site. My boss wanted to buy some small utility to keep tabs on some minor aspect of our environment. I was a bit offended -- I could have written that myself and here he was paying someone else to do it!

He asked: "How long would it take you to write and test this?" Me: "Probably a week. Maybe a bit less, maybe a bit more if I run into something tricky." Him: "Okay. This tool will cost us $500 to buy. What's your hourly pay rate for 40 hours?"

With this I achieved enlightenment. I've never again built something at work that I could buy cheaper.

fiddlerwoaroof4• 3 years ago

This tool will cost us $500 to buy. What's your hourly pay rate for 40 hours

This argument makes sense, but I worry that it’s a bit short-sighted. There are a lot of metrics that are hard to quantify where it might end up better: for one thing, I’ve found that integration costs and maintenance of integration of third-party systems are routinely underestimated: I’ve implemented “buy” for various systems where the work to integrate was essentially the work to build. The cost of learning a proprietary toolset rather than developing experience with open tools. The cost to the industry as a whole when something like AWS or React becomes the unreflective default choice.

MattPalmer1086• 3 years ago

I was told that you should only build strategic things you needed to differentiate on. Anything else, buy and adapt yourself to it.

I like your story of enlightenment though! I guess if you could genuinely build and maintain something cheaper, then why not.

OJFord3• 3 years ago

I'm not really sure what you're saying, or what the criticism is, if it is one - but shortly before the bit you quoted OP links the code on GitHub: https://github.com/WebSheets;

And you can't really conclude they made a poor implementation choice on a report of completing it successfully (albeit too successfully even, too much of Excel implemented, and then fixed by removing that (how do you do that to your XLS download link?)) on time within a short deadline.

The takeaway is supposed to be 'don't get too attached to your code', which in some circumstances (not this one) might mean 'don't succumb to NIH syndrome, use an XLS download link', but that's not the whole.

teaearlgraycold• 3 years ago

Any time I identify a fun and novel problem I get suspicious. Generally, programming should be mundane and you should be solving problems that have been solved thousands of times before. If something looks new it's more likely you haven't correctly identified the problem you're solving.

fruit2020• 3 years ago

A great engineer should be both good and great. I do a lot of ‘great’ stuff at work, but that won’t help me to get into my next gig unless I also code a lot and don’t lose my skill.

itsthecourier2• 3 years ago

I'm glad I read this article and learned about the circ

sr228822• 3 years ago

ironically, I was backend at Uber 2014 - 2018 and used to ask a simple version of implementing an excel formula engine as my go-to coding interview question. Its got a nice mix of data-structures, algorithms, complexity, and implementation. Good candidates can get a reasonably efficient implementation handling cell references in an hour.

nice read. made me nostalgic for the wild days of Uber-China hacking.

bastawhiz27• 3 years ago

Amazing! I independently had an extremely similar interview question (because of this work) that also gave me great signal. That's really funny!

Reubend• 3 years ago

Love the part about circular references. I gotta say though - I'm having a difficult time imagining how complex these fomulas are that reimplementing Excel's formula engine is easier than just porting the formulas into JS.

bastawhiz27• 3 years ago

The good news is that formulajs had a huge number of those functions implemented in JS already. Almost all of the time was spent on the engine, which wasn't a huge amount of code.

The problem with porting the code to JS is that a.) nothing is named, b.) there's no real way to organize the code you've written because you're going from a spatial way of organizing code to imperative script, and c.) the actual design of the spreadsheet wasn't known to any engineers (it was designed by a data scientist, or perhaps an analyst). The work of translating would have meant really understanding what the thing is so that it can be turned into functions and modules. It also would have still required getting Excel function equivalents, since there's not a 1:1 equivalence between Excel and what's available in the JS standard lib.

xen0• 3 years ago

Honestly, if you're at 'thousands of formulae' across multiple sheets, I'd probably suggest writing a graph based execution engine too.

The circular reference thing would have definitely thrown me for a loop though.

kirubakaran• 3 years ago

I just want to let you know that the pun in the last line didn't go unappreciated.

Vt71fcAqt72• 3 years ago

I love this article. It begins setting the scene with Uber's grand expectations for the chinese market and then shows just a small piece of the work Uber spent on Uber China. Meanwhile Uber China itself failed spectacularly, with factories of phones claiming the free rides money.[0] I don't think I've ever experienced dramatic irony in a blog before, certainly not a technical blog. The post itself is a masterpiece.

[0]https://www.forbes.com/sites/ywang/2016/09/27/ghost-drivers-...

gobrrrmeme• 3 years ago

There's a service in SharePoint and SharePoint Online that let you program against Excel. Just gonna leave this here.

https://learn.microsoft.com/en-us/sharepoint/dev/general-dev...

acchow• 3 years ago

Over the summer of 2016, we came up against a new twist on the project. We had a model that ran overnight to generate data for anticipated ridership in China. That data wasn’t useful on its own, but if you fed it into a tab on a special Excel spreadsheet, you’d get a little interactive Excel tool for choosing driver incentives. Our job was to take that spreadsheet and make it available as the interface for this model’s data.

They eventually built a homegrown "Excel" clone as the UI for their model because "city teams only know how to use Excel".

I would have done it the other way around - connected Excel to the data output by the model so the "city teams" could continue to use real excel. I think most finance teams do something like this.

ftxbro• 3 years ago

I would have done it the other way around - connected Excel to the data output by the model so the "city teams" could continue to use real excel.

Yeah except:

“When you click in the cells of the spreadsheet you can see the formulas. You shouldn’t be able to do that.”

“You said to make it just like Excel.”

“People working for Didi apply for intern jobs at Uber China and then exfiltrate our data. We can’t let them see the formulas or they’ll just copy what we do!”

bastawhiz27• 3 years ago

Because the city teams were in China, we didn't have this luxury. Everything had to be behind Uber's beyondcorp equivalent, and there was no real way to auth folks from the Chinese mainland. Our only surface was the browser.

MichaelZuo2• 3 years ago

A dedicated satellite link cross pacific is not that much money. Then again maybe the CFO didn't know so he took the easiest known option.

hansonw• 3 years ago

I’m building a solution that works like this - we directly connect spreadsheet models to company databases (even converting pivots/formulas to SQL). Would love to chat with anyone that might find this valuable: https://arcwise.app

pc486• 3 years ago

What a fantastic perspective from the former Uber BI team. I was on the Vertica team during this time period and the amount of effort was spent on incentives mind boggling. Millions a day lost on downtime, product features, or engineering bandwidth was a common theme.

A director asking for an exact spreadsheet to be the UI would have been par for the course, especially during the Uber China days. Heck, I personally loaded FX prices into Vertica from a spreadsheet emailed every month to the team. That process remained for more than a year as there just wasn't enough bandwidth to invert the control as automated ingestion.

Thanks for digging up these memories, @bastawhiz. I'd love to see more. :)

Macha• 3 years ago

So a lot of my time as a more junior engineer was spent on a similar project that I've described as "rebuilding excel". In my case, it was in the form of a table widget. It was an inhouse widget for displaying tabular data. We were working on AngularJS 1, and had moved from a prior iteration of our application in ExtJS. Now ExtJS had a reasonably featureful table widget out of the box, and so when we were porting to AngularJS, it was considered important that we continued to have these UI features, but at the time (the Angular ecosystem was relatively immature at that time) there were not very many advanced table widgets available in open source, so we ended up building our on.

However, while the ExtJS table widget had been treated by product management as pretty immutable "this is what ExtJS gives us, we can customise the colours and wording and that's it", the idea that we could customise the table widget started something amongst one of the PMs. And so we would get a constant stream of feature requests for the table widget to add stuff and enhance it and soon we were significantly more featureful than the ExtJS widget. It's still, to this day, the most featureful table widget I've seen in a web app. Everything Excel had in terms of resizing tables, sticky columns, scrolling behaviours, sorting, filtering, searching, etc., all saved in your config so it was synced across all your devices, as well as all the performance goodies like recycled rendering etc. The constant stream of feature requests meant there were dev team years invested in this table widget.

As a more mature engineer looking back, it's clear that at some point this had stopped being about customer value and more about one PM's obsession with getting excel like functionality in a in-browser reporting tool, but at the time we just kept building those features.

Now at this time, our company had acquired a sort of competitor of ours. This competitor had what was effectively the same product, but in a different market. And so the first merger of the functionality was basically to reskin both applications so they would pretend to be tabs in a unified application, and change some terminology, etc. They actually did happen to have a pretty similar tech stack to us, so some newer components were available in both applications.

But it became clear that our users were not happy with two applications pretending to be one. They wanted to know why this other market was not accessible by, say, a dropdown in the configuration, and not an entire application which worked in different parts from subtly differently to entirely differently.

So the discussion became about building a ground up unified interface for both of them. Of course, this ignited the discussion of "which table component do we use?". On the one hand, the acquired team were looking at our table, with it's fifty billion options and single handedly accounting for half the page weight of the minified JS of our application and did not want something so bloated. On the other hand, our team were looking at their table widget which was effectively "for row in data, for column in row, print td" and dreading having to rebuild all these features for product management again.

Ultimately the conflict was resolved by choosing to use an open grid that had less features than ours but more than theirs and telling the PM in question that table features were going to be prioritised much less heavily from then on unless there was a real user need for it.

abeppu• 3 years ago

I'm confused about the circular reference thing. Like, was there a reason to do the linear regression that way? Is there a secret story in a story where next week, in a spinoff / sequel episode, the data scientist responsible will explain why they took the weird/surprising choices they did?

andy81• 3 years ago

I used it once to run a Monte-Carlo simulation in a spreadsheet.

After enabling iterative calculation and manual calculation, every press of refresh runs a loop. Fun stuff.

DwnVoteHoneyPot• 3 years ago

It's a common excel trick in finance. For example, let say you have $0. If you borrow $1,000,000 at 5% interest, by end of year you'll be short $50k. That means you actually needed to borrow $1,050,000. But the extra $50K causes more interest ($2,500)... so you needed to borrow $1,052,500, which causes more interest... and so on.

Instead of doing some Excel Goal Seek or Solver or VBA macro, it's nice to let the excel "reactivity" handle it for you.

hu32• 3 years ago

Your explanation is brilliant!

DrDroop• 3 years ago

I didn't know you could do this in excel but it reminded me of the converge adverb in k: https://xpqz.github.io/kbook/Adverbs.html#converge-f-f

itigtohft2• 3 years ago

https://en.wikipedia.org/wiki/Fixed-point_iteration

brap3• 3 years ago

He simply couldn’t believe that I’d written a full spreadsheet engine that ran in the browser.

I can't believe it either, and I don't mean this in a good way.

Apache POI lets you run headless Excel. You import and interact with sheets programmatically in Java. We used this in my old workplace for exactly the same reason (functions, cell references, the whole thing), it worked great.

You found the ‘circ’ problem with a bit of luck. What about all of the other hidden little quirks of Excel that you would ultimately run into down the road? Are you really going to build and maintain a full blown Excel clone in JS? Is this really the objective of the frontend team?

It seems to me like a bit of googling and >90% of the work here could have been avoided. As an added bonus it would have been done by the backend team instead.

rkangel• 3 years ago

To be fair, he wrote a spreadsheet engine that could run one particular spreadsheet. Admittedly a complex one, but it was a fixed set of functions that he needed to implement and not and endless tail of things that people expect Excel to do. I think I'd probably have argued more about the UI spec and down some Excel behind the scenes but it is a familiar UI if you've got lots of number inputs all over the place.

I've always enjoyed this article about building a spreadsheet in 100 lines of F#: https://tomasp.net/blog/2018/write-your-own-excel/ The expansion from that to the feature set needed here is manageable.

brap3• 3 years ago

Project requirements are never fixed, they’re always evolving.

It was only a matter of time before users would’ve complained about features being missing/broken, especially since what they’re used to is Excel and this was meant to replace it.

bastawhiz27• 3 years ago

It seems to me like a bit of googling and >90% of the work here could have been avoided.

I had a deadline and the only idea on the team for shipping a working product, and I shipped a working product on time.

Uber ran (runs?) their own data center. Getting a Windows machine/VM procured to actually run Excel would have taken an act of god. I was able to spin up a new front-end service in about thirty minutes. And I had some code that sort of kind of already worked, so I wasn't starting from scratch. Keep in mind that this system needed to be used by multiple people with different sets of data simultaneously.

Are you really going to build and maintain a full blown Excel clone in JS? Is this really the objective of the frontend team?

If they'd have kept asking for more features and Excel parity, I suppose we would have considered it. But they didn't.

Certainly I don't expect many people would have chosen to do what I did. But the thing worked (and surprisingly well). If all you took away from the post is that it was a big complicated project, I'm afraid my writing has failed to convey the message it was attempting to convey.

dihrbtk• 3 years ago

they still run data centers but there is a cloud migration ongoing.

mherdeg• 3 years ago

A little over ten years ago I worked in the Excel Services team that makes the consumer-visible Excel Web App and also the SharePoint-integrated Excel Services product (server side processing accessible via API or web UI).

I loved seeing the genuine joy our PMs had whenever they found an honest to goodness calc bug and could get it reproduced and fixed in The State Machine. It was also a delight to see the web app approach parity with the desktop client experience -- we got to listen to a wide swath of users and build out the stuff we thought would be most useful to the most folks. And I loved our group PM's insight about what the heck Excel could be good for versus purpose built BI tools, other web sheet apps, pure SQL, etc.

This is a very fun kind of product to create and it's awesome that you were able to ship it in a way people could use!

password4321• 3 years ago

For anyone else curious, Apache POI re-implements support for the Excel file formats and most formulas.

https://poi.apache.org/components/spreadsheet/formula.html

idkyall2• 3 years ago

I think one of the biggest growth areas for junior engineers to reach mid-level and senior is recognizing when you're re-inventing the wheel. E.g. If you are given a programming task to do anything related to Excel or the Microsoft Office suite, it's worth googling it first, because some engineer somewhere was probably tasked with doing the same thing a decade ago and has written a blog post or made a GitHub repo for it.

bastawhiz27• 3 years ago

some engineer somewhere was probably tasked with doing the same thing a decade ago

*Seven year ago at Uber

JohnMakin2• 3 years ago

It's not just junior engineers. Senior/management can fall into this trap as well.

At one of my former companies we had a small problem with whitelisting cloudflare IP's that don't typically change super duper often but definitely cannot be assumed to be static. My boss at that time decided the solution was this big initiative he called "whitelist maker" and assigned it to me. I don't remember what implementation details he wanted, but it was some insane rube-goldberg machine to basically pull down this list: https://www.cloudflare.com/ips-v4 and then put it into some terraform code.

I ended up quietly killing the project during a re-org and used the cloudflare provider, which conveniently provides the forementioned IPv4 list as a data source in 1 line of code. Done, 5 mins work. He had scheduled out an entire quarter and half of a team's resources for it.

jabroni_salad• 3 years ago

As a generalist sysadmin sometimes I wonder wtf is in the future for me but at least I can say I made sure our data guys have a really good compute node instead of a crashy laptop cluster and dollar store excel. Are you guys REALLY sure about the no-ops dream?

Aeolun6• 3 years ago

Apache POI lets you run headless Excel. You import and interact with sheets programmatically in Java.

How does this help you in the browser?

eitland• 3 years ago

You make a small small backend-for-frontend that translates requests from the browser to calls to the Apache POI library and return the information to the browser, probably as some kind of json representation.

kazinator• 3 years ago

You see, when formulas create a circular reference, Excel will run that computation up to a number of times.

Like almost every spreadsheet before it: Lotus 1-2-3, Borland Quattro Pro, VP Planner ...

Spreadsheets iterating on circ references goes back to the 1980s.

The first spreadsheet application, VisiCalc, didn't track dependencies: it evaluated cells left to right, top to bottom, IIRC.

Microsoft had a product called Multiplan that competed with VisiCalc. Not sure if that did iteration.

I think it used to be a setting in some programs whether circular references are flagged as errors, or iterate. Maybe it's still that way in Excel?

itigtohft2• 3 years ago

There are a couple of nice minimal spreadsheets in js:

http://web.archive.org/web/20130606222859/http://thomasstree...

Inspired by that, an even smaller one: https://jsfiddle.net/ondras/hYfN3/

The latter catches circular references rather than trying to calculate the fixpoint

ChrisMarshallNY• 3 years ago

It’s easy to treat a particularly clever or elegant piece of code as a masterpiece. It might very well be a beautiful trinket! But we engineers are not in the business of beautiful trinkets, we’re in the business of outcomes.

This spoke to me.

However, as anyone that has looked at my code can attest, I tend to also want my code (and its functionality) to be very pretty. I'm generally writing code that I will be maintaining, so it needs to be something that I can look at, in a year, and understand.

I'm currently in the final phases of a project that I will never announce here, and don't plan on taking much credit for, but it really is da schizz. It's that way, because no one is paying for it, and no one will make money from it.

Money both spoils everything, and also makes it all happen.

allanrbo• 3 years ago

Another approach would be to use the Excel APIs. Both the classic desktop Excel and the web version has APIs to read/write cells and recompute. Rebuilding is more fun of course :-)

bjornlouser• 3 years ago

"'Growing as an engineer' means becoming a better engineer, and becoming a better engineer (directly or indirectly) means getting better at using your skills to create business value."

Learning how the Excel model worked and then reimplementing it would have been a better example of 'getting better at using your skills to create business value'.

bsder• 3 years ago

Side question: Why the hell is it so stupidly difficult to display and edit tabular data from programming code?

You can't drive Excel from Python/Rust/etc (Microsoft just announced to great fanfare that Excel can call Python--which is the wrong way around). All the editable table widgets for the web seem to suck. Nobody seems to have a TUI which you can drive from an external program.

A poorly written spreadsheet with a driveable API seems like a component that has been built multiple times by lots of people yet seems to be unavailable.

Is there some solution that I'm missing?

kubanczyk2• 3 years ago

Microsoft just announced to great fanfare that Excel can call Python--which is the wrong way around

How do you cause users to even discover that Python app, or feed an input to it? Of course the spreadsheet software is an obvious candidate for the primary UI.

easton• 3 years ago

Excel circular reference doc, for the curious: https://support.microsoft.com/en-us/office/remove-or-allow-a...

Unless you're familiar with iterative calculations, you probably won't want to keep any circular references intact. If you do, you can enable iterative calculations, but you need to determine how many times the formula should recalculate. When you turn on iterative calculations without changing the values for maximum iterations or maximum change, Excel stops calculating after 100 iterations, or after all values in the circular reference change by less than 0.001 between iterations, whichever comes first. However, you can control the maximum number of iterations and the amount of acceptable change.

cush2• 3 years ago

Any PM that walks in with “Just make it like Excel” hasn’t spent a moment writing code.

bastawhiz27• 3 years ago

Even worse: he was a director

cush2• 3 years ago

Arguably the most battle-tested, feature-rich program on the planet. Yeah just make it like that.

sha16• 3 years ago

“take this [the spreadsheet] and put it in on the website [Wesley].”

This should be taught in classrooms.

DubiousPusher• 3 years ago

Just a heads up for anyone who finds themselves with a similar requirement, there's a very robust set of office APIs available in .net. I would be surprised if you couldn't open and run a whole workbook with them though I have only used them for more tangential tasks.

JohnMakin2• 3 years ago

Reading this and having China cloud experience on my resume I wonder how many chinese data laws they may have violated, you have to really strictly segregate data when it comes to chinese users.

blincoln• 3 years ago

I was wondering the same thing, especially since it arguably involves mapping/geolocation information.

dmd2• 3 years ago

So the data scientists have multiple laptops that they download the data to, then run the models overnight.

This haphazard way of running compute jobs really stuck out to me. I can't imagine doing things this way (rather than having a central compute cluster running SLURM or similar) at a company bigger than, say, a dozen people - much less the scale of Uber. What's the rationale? Even if it's just a cluster of 3 or 4 machines in a rack shoved in the corner, isn't that better than ... laptops?

bastawhiz27• 3 years ago

It was easy to go to IT and say "get us a pile of laptops" and let the data scientists do their thing. It was hard to hire engineers to solve the problem (I was one of the engineers). They particular problem was far enough down the priority list that it took until 2016 to solve.

dmd2• 3 years ago

Heh. Well, I guess... sorry I didn't come help :) (I interviewed but turned down my offer.)

lph2• 3 years ago

Excel is an ancient and complex beast. I get the appeal of building this project---it sounds fun---but trying to duplicate the Excel engine to the level of producing identical outputs is, frankly, bonkers. The author caught the one discrepancy they noticed, circular reference handling, but how many did they miss? How do they know different inputs won't cause it to deviate from Excel? I didn't get the sense from the blog post that this had extensive test coverage. Putting it into use for a business-critical financial calculation is a massive risk, but I guess that's how Uber rolls ¯\_(ツ)_/¯

It would have been less fun but way, way less risky to wire a headless Excel up to a javascript front-end.

bastawhiz27• 3 years ago

No Windows, no Excel license, hundreds of concurrent users. How confident are you that each user is getting the calculations that they triggered and not someone else's? How are you deploying that spreadsheet? How are you version controlling the mutable parts of the system? How confident are you that the solution would be robust in the face of bad inputs and high load?

I won't say there wasn't risk, but there was quite a bit of testing and a human always made the final call anyway (I never fully understood why we didn't eliminate humans from the processes altogether).

ryukoposting2• 3 years ago

I wonder if the author would view this situation differently had Uber/Box decided to claim the code as their own. It has to bring some catharsis to know that, even if the code never actually met its potential, at least the whole world can see and appreciate it.

I created a whole programming language as an intern for <defense megacorp>. It was lazily evaluated and garbage collected. Unquoted MAC addresses were valid syntax, among other application-specific oddities. No bytecode or JIT shenanigans - the interpreter just pushed and popped stuff from a stack as it traversed the parse tree, and that was fast enough for what we were doing with it. The interpreter was written in pure ANSI C, and Valgrind was very happy with it. Maybe it has been totally forgotten, or maybe it became critical to their technical infrastructure. That code never left the airgapped lab where I wrote it, so I have no way of knowing. 3 years ago, as a recent college grad, that was by far the coolest piece of "actually useful software" I had ever written. It's still high on the list. Sometimes I wonder whatever happened to it.

lowbloodsugar10• 3 years ago

The part the author is missing is that Box and Uber have already claimed the code as their own. It will be in the employment contract. The author appears to be under the impression that asking a mid-level manager (or even a senior manager) "Hey, do you want this?" and that manager answering "No", is in any way legally binding on the company.

ryukoposting2• 3 years ago

I've only ever agreed to those terms kicking and screaming. I wish there was a larger part of the community willing to advocate for a "love of the art" clause that would explicitly state that code I wrote on my own time is mine.

joshu• 3 years ago

i feel like “learn vba” would have been a much easier path here

OJFord3• 3 years ago

having answers that were very, very close is objectively worse than having numbers that are wildly wrong: very wrong numbers usually always mean a simply logic problem. Almost-correct numbers mean something more insidious.

Ha! I massively identify with this. 'The market-wide median is off by £3.07' is a much harder but to crack than the 'worse' report that 'the median is twice the max and the min is null'.

OJFord3• 3 years ago

*bug to crack

stevebmark• 3 years ago

A good and related article about the dangers and poison of domain driven design, and why spreadsheets are used at every level of every company. https://dev.to/cheetah100/domain-driven-disaster-147i

ineedasername• 3 years ago

These are the times that, if you at least enjoy the process of coding, you should try to look at your work as a videogame. You enjoyed the game, you even got paid for it, and you beat the final boss, it doesn't matter that no one will ever see your final score.

ipython3• 3 years ago

Best quote: "People working for Didi apply for intern jobs at Uber China and then exfiltrate our data. We can’t let them see the formulas or they’ll just copy what we do!”

This is so true. People in the US just don't understand the level of economic and industrial espionage that happens in China on a daily basis. I was responding to an unrelated breach at an unnamed tech company back in mid-2000s time frame and had a side bar conversation that went like the following:

Them: "Yeah, we just opened a tech center in Xinjiang and ... wow, we've had quite the rash of lost ID badges there recently"

Me: "Have you considered that they're not 'lost', but rather 'sold' for profit?"

... silence ...

I don't know if executives are aware but just don't care, or if they're simply incompetent, but China has productized industrial espionage on a massive scale. GE Aviation was a victim more recently: https://www.cincinnati.com/story/news/2022/11/16/accused-chi...

cflewis• 3 years ago

Well, not just corporate espionage, right? State espionage is going to be at every large US company too. To speak to the article, I can only imagine how excited an agency would be to get real-time updates on the Uber movements of a target.

ipython3• 3 years ago

Two things. First, this isn't the same thing; you're talking about embedding a resource in a company to read out real time telemetry from internal systems (information with a short shelf life), not stealing industrial trade secrets (information with a very long shelf life). Second, I can assure you that, even with our imperfect systems, there is actually a set of checks and balances in place to prevent rampant (note I said rampant) abuse of this kind.

Yes, agencies would be very excited for this sort of capability. Do they get it as a matter of course that easily? No. There are layers of accountability, legal authorities, and (warranted) push back from commercial entities.

nextaccountic• 3 years ago

State espionage is going to be at every large US company too.

Snowden revealed, among other things, that the NSA did state espionage on Brazil's state oil company, Petrobras

https://www.nytimes.com/2013/09/09/world/americas/nsa-spied-...

https://www.theguardian.com/world/2013/sep/09/nsa-spying-bra...

https://g1.globo.com/fantastico/noticia/2013/09/nsa-document...

International relations are based on reciprocity. I sincerely think that if the US didn't think that industrial espionage would be a legitimate activity of intelligence agencies, they wouldn't practice it themselves.

makeitdouble3• 3 years ago

They don't understand it, because what you're quoting as a behavior is pretty standard anywhere in the world. That's why companies protect their secrets from low rank employees.

Except if you're arguing that western companies are bound by stronger ethics, in which case I'd like to see some evidence.

anikan_vader• 3 years ago

Less so ethics and more so credible legal action against offending companies and individuals.

ipython3• 3 years ago

If a low ranking employee in Cincinnati divulged corporate trade secrets, they'd be tried and convicted. If a low ranking employee in Shanghai did the same, what can the US company do? Not much. So it's a low risk activity, that is also actively encouraged and in many cases financed by the Chinese government. This is in no way intended to be disparaging of folks who are of Chinese descent, but rather a reflection of the reality of contemporary Chinese government politics and policies.

envsubst3• 3 years ago

western companies are bound by stronger ethics

I wouldn't say people in the west are "better people in their hearts" but they absolutely follow more strict norms regarding honesty and theft. This is one of the main reasons why companies pay a premium for workers in the west when they could hire from other places.

One example: accounting scandals in the US are rare. I don't think anyone trusts accounting figures for public companies from India or china.

in which case I'd like to see some evidence.

The alternative to trust is enforcement. The evidence you are looking for is the prevalence of the latter principle in systems that deal with things of value.

concordDance13• 3 years ago

The workers who make up the companies tend to have higher levels of honesty and fair dealing and lower levels of corruption and theft.

Though I cannot provide evidence for this offhand beyond anecdote and the corruption perception index.

mmcwilliams• 3 years ago

It's somewhat ironic to frame that as a problem that "isn't understood" in the US considering what the CEO of this company was convicted of.

mkii3• 3 years ago

And yet, when the author takes code from one company to use for another, or releases company code on Github, nobody bats an eyelid.

stickfigure2• 3 years ago

If we're being generous, the author may have had permission to do so. It's not inconceivable; the code was abandoned. If one of my reports had asked, I would have approved.

zik• 3 years ago

My hair stood on end both times he did this. Holy crap that's some massive legal liability.

camel_gopher• 3 years ago

That’s called being a Code Gypsy

tomcam3• 3 years ago

That jumped right out at me too. Two levels of IP theft except on the remote chance that both companies allowed OP to open source the code.

concordDance13• 3 years ago

That's because that code wasn't in use, so there's no lost business.

Angostura• 3 years ago

Indeed. Having warned against "If you treat the code like a pet for sentimental reasons, you’re working in direct opposition to the interests of the business." He does exactly that.

prmoustache• 3 years ago

Actually that was the first thing I looked for in the comments, even before finishing the article and seeing he even published the finished code on his own repo (and not box or uber one), under his only name.

somsak2• 3 years ago

what are you talking about? there's a huge comment thread on here exactly about this issue.

endisneigh• 3 years ago

Well information wants to be free, copyright shouldn’t exist, piracy is ok yadda yadda

diogenes44• 3 years ago

People in the US just don't understand the level of economic and industrial espionage that happens in China on a daily basis.

Back in my day this was called "competition" and worked for the consumer, not against them. I find the espionage factor to be theatrical hogwash trotted out by the corporate types. We americans love competition, right? Stfu and compete.

oh_sigh• 3 years ago

What it would do is punish companies that do real R&D, and reward companies that have no actual talent in that area. So ultimately the field would stagnate because no one would invest in R&D.

Proziam• 3 years ago

Is it "competition" when you're up against state-sponsored actors who will use tactics the rest of the world would consider illegal?

phendrenad22• 3 years ago

You're missing a huge part of this: The "competition" you think you're referring to had, you know, actual rules. Sending people to get jobs at a rival, and steal all of their internal documents and trade secrets, is illegal in most countries.

rfrey• 3 years ago

When exactly was this "day" when industrial espionage was legal?

concordDance13• 3 years ago

I'm a bit confused what that would look like. Sending Americans to China to steal their secrets wouldn't work because the Chinese wouldn't hire them for exactly this reason.

pbj1968• 3 years ago

And I was called a racist 25 years ago when I said, “like, our engineering school is, like, crawling with Chinese nationals.” We said like a lot on the 90s.

robertlagrant7• 3 years ago

It's probably the "crawling with" that did it.

zo1• 3 years ago

We're headed for a very protectionist and isolated world because some players couldn't play nice. To me, the only thing holding the floodgates in the west back is the media which is very "rainbows and butterflies" about everything.

As soon as they start playing along and seeing this stuff in an alarmist way, they'll turn the narrative and report on it constantly.

Philpax• 3 years ago

If you used that wording, I'd have called you racist too.

chaostheory• 3 years ago

That’s because you were being a racist. Most people reserve the word “crawling” for insects and other small pests. Only racists apply it to an entire set of people.

iancmceachern11• 3 years ago

I've watched this happen.

I've watched key, core engineers and technical leaders work for US and European companies, develop their next generation products, then turn around and design and develop essentially the exact same thing for the Chinese market. They then build a company, in China, that makes essentially the same product, but for the Chinese Market, and with Chinese investors, etc.

Examples: Thoratec/Abbot Heartmate III & CH Biomedical

Auris/Verb/J&J Robotic & Digital Solutions & Renovo Surgical

The ironic thing, is that some of these companies after success in China are working to sell and be competitive in the US and Europe.

It's not even secret or under the table anymore, it's overt and largely accepted as the way it is in our industry. A brave new world.

The other factor, is that it is very very difficult for a foreign company to do business and protect their assets in China, so often the wise companies don't even try. They often just license their stuff for the Chinese market to a Chinese company. That way they at least have a chance of not having it all just stolen.

noobface• 3 years ago

Everyone should've seen what happened to Cisco and thought better, but short sighted execs focused only on next quarter gladly opened the gates and accepted the horse.

vkazanov• 3 years ago

So an engineer learns how to make something, and then goes elsewhere to make it.

To be honest, the way you put ut, the story feels OK to me.

There are many truly bad examples, e.g. the arm china story, but engineers doing their thing is not one of them.

nikau2• 3 years ago

Terrible, such actions are only tolerable when the US do it via moral operations like operation paperclip.

Lex-2008• 3 years ago

some of these companies after success in China are working to sell and be competitive in the US and Europe.

Just out of curiosity: can't these companies be sued by the IP holding company when they try to sell outside of China, and be forbidden to sell their products in US and Europe?

htss2013• 3 years ago

it's overt and largely accepted as the way it is in our industry. A brave new world.

What alternative is there? The only protection there ever was for taking business secrets was patent enforcement, civil lawsuits or prison. If a foreign government won't cooperate on any of those things, what can you do?

The only answer humanity has ever come up with is something like a government intelligence agency, where everything is obfuscated by clearance levels and need to know compartmentalization, and any violations are handled criminally, with armies of full time counter espionage people. That just wouldn't work in the corporate world.

choppaface4• 3 years ago

Given Uber's efforts on Greyballing, booking fake Lyft rides, hiring Anthony Levandowski etc etc ... it's very on-brand for them to (1) fall into an espionage war with Didi (wonder where all their Vertica data came from?) and (2) have an engineer write about using code he lifted from a prior gig and then also self-published.

https://www.nytimes.com/2017/03/03/technology/uber-greyball-...

https://www.theverge.com/2014/8/12/5994077/uber-cancellation...

willcipriano2• 3 years ago

Intellectual property is a western invention as I understand it. I don't think China agrees that you can own an idea like we assert that we do.

layer8• 3 years ago

China certainly did agree in various ways: https://en.wikipedia.org/wiki/Intellectual_property_in_China

pjmlp• 3 years ago

Not only there, everywhere.

Since I also wear a security hat, when doing code reviews, architecture and devops stuff, it is surprising how much stuff regular developers never think about in regards to security.

IG_Semmelweiss• 3 years ago

Its not only that, but when they can't beat them...they make them join: startups with valuable market intelligence get bought all the time by Chinese companies.

See for example the case of DEAR systems, which is a cloud-based WMS that is particularly useful for intelligence as to how much (US) importers are paying for worldwide goods, and what is the cost of shipping from A to B.

It was bought by a chinese company not long ago. That's 1 example.

https://www.prnewswire.com/news-releases/cin7-acquires-dear-...

iaseiadit• 3 years ago

Similar reports were widespread 20+ years ago, when I worked in aerospace. Bad faith partnerships, industrial espionage, and companies barred from competing after all useful information was extracted. It has gone on for decades and nobody will do anything about it. Executives and shareholders get dollar signs in their eyes over the thought of tapping into the Chinese market, and let themselves get burned over and over again.

fennecfoxy3• 3 years ago

Yup, the Chinese government doesn't really care about infringement unless the IP is Chinese itself and being infringed upon by non-Chinese companies.

I still remember the agency I worked for getting in an industrial designer to create some beautiful cases for some iBeacon hardware we were building. They looked great.

We organise a Chinese company to do the injection moulding and are sent samples that look pretty good, so we decide to use them. Few weeks later we see OUR cases on Alibaba/Aliexpress.

The West/other countries aren't perfect either, but that's not what we're talking about here. _Everyone_ I know who has worked with Chinese manufacturing/businesses has a "they ripped us off" "they sold our hard work to someone else" "they provided lower grade x than was agreed".

And the counterarguments always come down to: "yeah but the West does X" or "you're just being racist".

Chinese companies, especially the ones that do business on Ali-X _LOVE_ this as they can get the IP and use it for $0 and then undercut the original producer of the equipment. Plenty of makers find that their designs on Tindie etc are ripped off and appear on Ali, too.

nardi• 3 years ago

I do not like working with people like this, because I don't want to be anywhere near the maintenance burden of a pile of hacks that technically can run a single Excel spreadsheet. I don't even want to be within earshot of people complaining about this. If I even thought about it for too long I'm sure I'd have a giant headache.

ndesaulniers• 3 years ago

I worked with the author for maybe 3 years, I thought he was pretty awesome.

Not surprised he implemented a partial implementation of excel so quickly.

I'm disappointed in myself that I somehow got through most of the article without realizing who the author was (thanks Firefox reader mode); especially since the naming of Wesley and Crusher is too good. Of course it was Basta! Of course it was!

abigail95• 3 years ago

what would you have done instead?

prasoonds• 3 years ago

In a startup, you either build, ship and you survive another day, or, you die.

If I were hiring for an early stage startup, I know exactly which of you two I would want to hire.

CodeWriter23• 3 years ago

Someone will take joy in ripping it out someday.

purpleblue• 3 years ago

1) This was the type of effort and level of coding that I saw when I was Uber during its heyday. I love it and I miss it every day.

2) You stole Box code, used it at Uber, and then stole Uber code and posted it on github. I understand no one's using the code or missing that code, but you really did steal that code. It belongs to the company, not to you. I would be careful not to do that in the future, because technically that's a trade secret and people have gone to jail over that, like that programmer at Goldman Sachs (wrongfully) and ironically Levandowski who took Google code and tried to use it at Uber.

lowbloodsugar10• 3 years ago

IANAL but I've paid lawyers to answer this question in the US.

The question is, if you do some work, in your own time, on your own equipment, does your employer own it just because the employment contract says they do?

In California: if the work in any way relates to the employer's business, then yes, they own it. One way to guarantee that it relates to the employer's business is to bring it into the office and use it as part of your job. If your employer is Apple or Google or AWS or Microsoft, then probably anything you write would in some way relate to their business. Write spreadsheets by day, but games by night? All of those companies make games, or are in the games business.

I would love to hear a lawyer say, "Well, it doesn't matter what the employer does, it only matters what your job duties are, so writing games at night is fine if they don't pay you to work on games related things during they day." But I've never been told that by a lawyer, whether I paid them or otherwise.

Everywhere else in the USA: they probably own it. You could write software for washing machines, and write a video game, and if your contract says they own everything you write then they do. You signed it. There's no "but surely not!" defense.

mkii3• 3 years ago

"In 2015, I had built a prototype of a tool at Box. Box had a collaborative note-taking product called Box Notes (based on Hackpad). I had the idea to make a similar project for working with numbers: sometimes you didn’t need a full spreadsheet, you just needed a place to put together a handful of formulas, format it with some headings and text, and share it with other people. Sort of like an ipython notebook for spreadsheets. I called it Box Sums."
"Nothing came of it, but I took the code and shoved it into my back pocket for a rainy day."

...

"Apparently that was a thing. I remember being only half-surprised at the time. I hadn’t considered that our threat model might include employees leaking the computations used to produce the numbers in question."

...

"My first reaction was to publish the code on Github."

Perhaps the author feels "only half-surprised" due to their own disregard for corporate legal ownership of code. The hypocrisy is strong here.

lxe• 3 years ago

To this day, I’ve never encountered an in-house application system as well-designed as Uber’s. You could go from start to Hello World running on a *.uberinternal.com subdomain in under 30 minutes with full CI/CD.

I worked on all this at Uber back then, and this comment warmed my heart a bit. Thank you.

charles_f• 3 years ago

“Every day that we don’t have this tool as specced, we’re losing millions of dollars.” There was no budging on the spec.

If this is life or death, then the first reflex should be to find ways to make things simpler to ship. I understand that the requester in this case was a head of finance, but sh*t, if you want something, try to compromise once in a while.

This is a great story, but at some point as an engineer in a gig-selling company, I would have asked myself if my job was really to re-build excel.

alzaeem• 3 years ago

I interviewed with Uber in 2016 with the "crystal ball" team. Reading about R-crusher, I am glad that offer never came through!

StopHammoTime• 3 years ago

I find it remarkable that they used laptops for this. Surely, a Dell server for $100k with good specs could have crushed a lot of these models. Could have just given them all Remote Desktop sessions too.

schlowmo• 3 years ago

“The city teams only know how to use Excel, just make it like Excel.”
“Why can you see the formulas?”
“You said to make it just like Excel.”

I can't keep up with the espionage story that followed but I had this conversation more than once in my professional career.

The first time I sat as a junior dev on a multi-month project replacing a excel spreadsheet for financial controlling of data centers that only one person who was going to retire understood with a web-based solution.

They were quite proud that they were going to get a "modern" solution.

Then they wanted me to make it like excel. What followed was evaluating every fricking excel JavaScript library out there at this time, going for one and started duct taping all the missing pieces.

They were pleased but the look was off. It wasn't excel. I slapped some styles on it coming quite close.

I was not prepared for what happened during the next presentation: They hated it because they wanted a modern web based solution (their words, not mine) and what they got was a poor excel knockoff running in their Internet Explorers. Tables are so 90s.

I remember the pain so vividly that I regard "just make it like excel" as some kind of forming meme for my career till today.

ww520• 3 years ago

I had a somewhat similar requirement in a past work. The Excel file had all the formulas and data, and the analysts only work with Excel files. The result of the calculation in Excel needed to be put on the web.

The solution was not to recreate Excel in the browser, but ran the Excel file with its formulas and its data plus the input parameters from end users at the backend server. Apache POI was a nice Java library that could do everything on an Excel file. Once it finished the formula calculation, I just read the cells from the Excel file to extract the result data and generated the web pages to present the data and graphs.

One nice benefit was the analysts could update their work in the Excel file, uploaded it to the server, and got the new calculation reflected on the web pages right the way.

whimsicalism2• 3 years ago

Funny that this article is talking about interns exhilarating data and yet the author blatantly recycles code between employers.

ec109685• 3 years ago

This was an amazing read. One of my favorite memories in college was building a spreadsheet in Java (some of the guts were there) in third semester CS.

jengler• 3 years ago

Great to see @bastawhiz on here.

Box had a collaborative note-taking product called Box Notes (based on Hackpad).

Slight correction: Box Notes was forked off etherpad-lite. Hackpad was a parallel fork of Etherpad.

[context: I was lead engineer on Box Notes till around 2014/2015. I left before the events @bastawhiz described]

NavinF• 3 years ago

You see, when formulas create a circular reference, Excel will run that computation up to a number of times. If, in those computations, the magnitude of the difference between the most recent and previous computed values for the cell falls below some pre-defined epsilon value (usually a very small number, like 0.00001), Excel will stop recomputing the cell and pretend like it finished successfully.

hannofcart• 3 years ago

On reading this article, I realised that maybe the most noticeable change that experience brings to a developer is... laziness?

When I was 24, I was pretty much exactly this person. "Building a spreadsheet engine that runs in a browser. Sounds like so much fun!" And then I'd slog away at it for a month and get something working.

Now though, with age, I know I can't chug away coffee working late into the night. My back hurts when I have to sit too long. My wrists aren't being too kind to me these days either.

Now if someone asked me to build Excel I'd first laugh in their face. And if I don't see them smiling, I'd ask if they can afford any of these A,B,C...Z COTS products that are doing this that we can just buy. And if none of those work out, I'd look for how I can take something like ethercalc and repurpose it for our use case.

Looks like the older I get, rather than writing code, I seem to be getting more adept at how not to write it.

mathgladiator• 3 years ago

Beyond the questionable IP theft.

The lesson of what you build for a company as something that may be thrown away is a good one.

However, I believe the only way to actually have a sacred masterpiece is to own the company and make the company's mission to build that sacred masterpiece. For example, there are definitely business opportunity for converting excel sheets to online apps that are collaborative.

The core problem is alignment between the tech investment and the company's goal.

nperez• 3 years ago

I've worked on a project that was intended to replace Excel spreadsheets with a web front end. It was presented like we were moving our processes out of the stone age - as if we were replacing a pen-and-paper process that "doesn't scale", or whatever.

I will never do that again. We didn't build a whole spreadsheet engine, but we did build a web UI that simply doesn't do as much as Excel. Excel is powerful. Sometimes it's a fine tool for the job. Our team was laid off before we could roll it out, but I remember the growing sinking feeling of "I would hate using this if I already had a bunch of habits built up in Excel"

PeterStuer3• 3 years ago

Running the models on laptops overnight? They know desktops and servers exist right?

choppaface4• 3 years ago

But when you have to be your own IT / devops... and when maybe all you know is R, some stats, and how to admin a laptop...

The head of finance may have burned most of the opex budget and devops bandwidth on the Vertica and/or other DB service(s) they were using.

avidiax2• 3 years ago

Uber, at that time, was "scrappy". Every operating city was a different market with different rules, so Uber was akin to a franchiser that gave their market managers wide leeway to setup whatever business processes are needed to succeed in that market.

Excel is the easiest way for business-oriented people to make numerical decisions, run incentive campaigns, segment riders and drivers, produce reports required by cities, counties, states, etc.

The data teams providing market intelligence to the various markets, of course, knew about servers. But they would not have the time or skill to setup something like this, and the business still had daily data needs. So the laptops it was, for a time.

PeterStuer3• 3 years ago

A laptop is basically a desktop with sustained performance thrown out the window in favor of sleek portability.

quijoteuniv• 3 years ago

Is interesting what people focus on when they read, ultimately I think one needs to ask oneself why you take some jobs. Is the money, the challenge or the purpose. I find that money and challenge has little to go for me if the purpose is skew.

mcintyre1994• 3 years ago

I’m really curious how Uber China was actually supposed to work. I once worked for a company that built server software and my understanding was that they were partnered with a Chinese company to sell into that market. We built a branded, localised installer for them. I’m not sure what the business arrangement was or what equity relationships (if any) there were between the companies. But I can’t imagine Uber would ever be allowed to have a fully owned Chinese subsidiary competing in that market?

avidiax2• 3 years ago

I can’t imagine Uber would ever be allowed to have a fully owned Chinese subsidiary competing in that market

You imagine correctly, to my knowledge. Western companies in China have to be 51% Chinese owned, on paper.

In reality, they own 100% (plus or minus diplomatic relationship issues), they just haven't asserted this right yet.

This is just my opinion: Uber could never have won in China, and Uber's management likely knew this, but the story of China was needed at the time to drive company valuation and allow the "hockey stick" dashed line of future growth to be plausible.

hddqsb• 3 years ago

Tangential, but regarding footnote 7 (efficiently recomputing cells, assuming no cycles), the correct algorithm for determining the order is topological sorting (https://en.wikipedia.org/wiki/Topological_sorting). Using a breadth-first search gives a good approximation in many cases, but there are pathological cases where it can produce an inefficient order.

Uber Sheets

Hee hee, this name is funny on a number of levels :)

mihaaly2• 3 years ago

Too many eager geeks are pushing through with realization of things that are a raw (not even half cooked!) droppings of a clueless mind with authority, instead of just simply saying 'No!' - or giving alternatives at least. Driven by pride doing things that should not have been attempted the first place. Lots of waste, lots of waste, lots of half cooked thingies interacting into unreliable solutions.

MrDresden• 3 years ago

This just gave me chills. Never would I want to work on a tool like this from scratch, and alone for that matter, that was underpinning crucial financial planning for a giant operation like the one described in the article.

There are simply too many potential problems that could come up in this scenario and be the cause for catastrophic financial results in the end. And most of them wouldn't nessecarily be easily found via unit tests.

fnwbr• 3 years ago

given this website’s name it’s incredible how deeply neoliberal posts’ contents, comments and discourse have gotten…

y’all seriously believe that the only valid reason for an engineer or a programmer to exist is to “create *bUsiNess~ value”? i’m shaking my head.

this blog post is full of ideological blindness.

junon• 3 years ago

Hey I worked with Matt Basta at Uber, cool guy and great engineer. His interview question for me was about this exact thing - how one might build excel from scratch. Neat to see him here :)

roarcher• 3 years ago

The intended lesson of this article may be the "cattle, not pets" point made near the end, but I think there's another lesson hidden in it.

This is the story of literally every single large project I've done under a tight deadline in my career:

- Some non-technical department (usually sales) promises a customer something very big very soon, with no regard to how much time might actually be needed to build it. Fulfilling this promise "on time" is now your problem, Engineer.

- Months are spent frantically building said Thing, working overtime, burning out engineers, and bastardizing the previously clean codebase in the rush.

- The deadline is met, and no customer uses Thing for at least several months. Or the deadline is not met and customer waits for Thing as long as it takes, because they're not going to blow up a big contract or integrate with your competitor instead of waiting another month. They have too much invested and the deadline wasn't that important anyway. The third possibility, as it was in OP's case, is that Thing gets thrown away in its entirety due to some fateful turning of the organizational wheels.

I swear, almost as if by some karmic law of the cosmos, literally every frantic deadline turns out to be irrelevant in the end, and you should have saved your sanity and gone home at 5 every day for the last 6 months.

oq_pmg• 3 years ago

Questions at the end of the article, are the ones, every engineering manager should ask

camgunz• 3 years ago

I just finished doing this. I built an Excel formula parser and interpreter, and then a transpiler into Python. I even had the experience of having the values off by "just a little" and there's so many things that can cause it, it's just whack-a-mole honestly.

Makes me wonder how many times this has been done haha.

smrtinsert• 3 years ago

My favorite line: > On the other hand, we as engineers need to be real with ourselves. Every piece of code you write as an engineer is legacy code. Maybe not right now, but it will be.

which of course leads to what some people are saying, did this have to be built? Sometimes I think our job is to optimize ourselves away.

I built Excel for Uber and they ditched it

Related Stories