Tobias Lutke still writes code for Shopify

https://changelog.com/podcast/416#transcript-45

by jerodsanto • 6 years ago

124 64 6 years ago

I can't think of a better way to actually maintain a clear understanding of what's going on at a place like Shopify.

Good memory: Lutke's blog is responsible for a lightbulb moment of mine back in 2007, regarding cache invalidation. The post is long gone, but it was entitled "The Secret to memcached" -- the advice, which is incredibly obvious in retrospect but wasn't to me at the time, is to manage cache invalidation by adding a unique ID to the cached asset, and letting the cache fill and expire items on its own. Going into it, I had assumed I should be removing cached items in the code that was utilizing the cache. I was ignorant. I still am, but less so about memcached. Thanks, Tobias!

dmarlow2• 6 years ago

Thanks for sharing this. I've used this (I call it a cache buster) for static assets in a CDN, but never considered it for other things. I'd love to read the article if it can be dug up. It's not clear how that unique ID is managed and am curious to learn more.

craz8• 6 years ago

This technique is the way Rails Fragment Caching works, and enables Russian Doll caching

Those terms should help you find more writing about this technique that I always forget isn’t obvious outside the Rails ecosystem

mmahemoff2• 6 years ago

DHH helped to popularise it with this post: https://signalvnoise.com/posts/3113-how-key-based-cache-expi...

It's been hugely influential to the way I think about scaleable pages. It's a similar idea to the cache-busting techniques well-known with URLs (https://example.com?style.css=1234 etc), but applied to page fragments recursively, where a change in any component also "revs"/"bumps" its parent components.

embit• 6 years ago

I think he was an active contributor to RoR in those days

christocracy• 6 years ago

He’s great. I used his Money (for managing money values as cents in the db), Liquid and DelayedJob in a bunch of projects.

Now I run my own Shopify store, selling my software.

AVTizzle• 6 years ago

> I can't think of a better way to actually maintain a clear understanding of what's going on at a place like Shopify.

Talking to customers would be one better way to maintain said clear understanding.

Running your own Shopify store (which Tobias also does) could potentially be another.

throw0101a• 6 years ago

The post is long gone, but it was entitled "The Secret to memcached"

The Internet (Archive) remembers:

* https://web.archive.org/web/20120125060458/http://blog.leets...

ehnto2• 6 years ago

We call them cache keys. We use a hash of entirely arbitrary data you want a cache item to invalidate on when that data changes.

For example, a cached snippet of HTML for a related products section should invalidate if the product IDs change, so you hash the Item ID with the product IDs and bam, you got a cache key that will let you know when you need to reprocess that snippet.

Put that bad boy as a child of another snippet and you can cache the whole lot. It's cache keys all the way down!

Caveat: not worth the engineering effort for simple systems, just use an FPC and call it a day.

reidjs• 6 years ago

FPC: Full Page Cache

ashtonkem• 6 years ago

You can often go further than that with some database trickery.

We once needed to cache a list of objects that was unacceptably slow serialize the JSON for. We needed to cache them in memcache, but also needed reliable cache invalidation without having to instrument every ingestion point with the logic. Here's what we ended up doing.

1. Every record gets an int field titled "index" or similar.

2. Every time a record is inserted or updated, index is changed to the next value from a sequence

3. The cache key for memcache includes the number of records and the maximum index from all the records returned (including any intermediate objects if joins are involved)

4. Infinite TTL for all cache keys, but the entire cache is LRU guaranteeing that stale records are eventually removed.

This setup works because any conceivable operation done would change the cache key. If new records are inserted or deleted, count and index change. If a record is updated, then the max index changes. Even deleting and adding a new record will change the index, meaning that no matter what happens you'll get a new key and the system will continue to cache appropriately.

arcticbull• 6 years ago

I disagree completely.

Once you're in a management role, let alone a CEO role, your job not to write software. You defer defer implementation and architecture completely to your team. Your job is to hire and build that team and set high-level company direction.

The more power you have over your co-implementors' careers -- and the more junior they are -- the less comfortable they are standing up to your architectural or technical decisions. This stifling of collaboration can cause you to make poor decisions you wouldn't otherwise, and creates an uncomfortable dynamic.

This is why I don't believe in TLM ("tech lead manager") roles, also, fwiw.

Wanna code? Do it on your own time, Tobi! :)

ragebol• 6 years ago

As CEO or CTO, it;'s crucial you understand your business and if you're in a tech business, you better understand the tech to an appropriate level too. Coding can be a part of that

bertr4nd• 6 years ago

I’ve only once worked under a manager who also coded, and I had extremely mixed feelings. On the one hand, he was a brilliant engineer who I learned a ton from, and who set a really solid technical direction for the team.

On the other hand, I was personally unhappy a lot of the time. On one occasion he pretty much tore apart one of my designs and I felt fairly humiliated (with reason, as a few other senior engineers confirmed in private conversations).

Did my design deserve to die horribly? I don’t know. The PR had already been accepted, but maybe the accepter was wrong. In any case it felt really crappy to be criticized so heavily by someone who in theory supports me.

dmarlow2• 6 years ago

Generally true, but not always true. Shopify seem to be doing just fine with this model. If things change, maybe this model will also change.

_the_inflator• 6 years ago

You got the point provided that you are concerned about code quality. This might be an area where a CTO lacks skills over time compared to his developers.

Becoming a CTO myself within an 80 developers area, I found it invaluable to code infrequently still. It is quite simple: better decisions, better products.

To me, non-coding CTOs or most CEOs that do not understand Computer Science, it seems that they try to learn a foreign language by sending in their assistant to attend classes.

I mean this quite literally. My team can communicate to me in other terms and a different language than they would have in a simplified ELI5/KPI driven manner. And vice versa, I challenge them to think beyond day to day business.

It is not a lack of trust. It is about better understanding and better helping your team focus on the essential parts. And vice versa.

tgv• 6 years ago

I believe in dog-fooding, using your own products. That holds for programmers and UI designer, who should use the software they build, but also for architects and CTOs: they should experience the effect of their decisions, and taking part in the design and coding can help.

indysigners• 6 years ago

From inside of Shopify I’ve heard about a lot of shit stories. For their employees, it may be less ideal to have a CEO messing around with code.

From the outside though, the company does extremely well.

Nonetheless, CEOs are generally not supposed to „make“ but to manage. There has to be a tradeoff somehow.

ajharrison• 6 years ago

How is this a good thing?

All it shows is complete lack of trust and complete desire for control.

Shopify is an awful company to work for unless you’re a junior dev who likes nerding out and refactoring code over and over again.

Source: worked there

fao_2• 6 years ago

While you were downvoted for being negative about this and your style of message, I think there is a kernel of truth, and I don't think we should be so eager to dismiss the experiences of people who have actually worked at a place, versus what is essentially a form of PR for the company.

jseliger• 6 years ago

I suspect the problem is the lack of specificity: it's hard to judge from the comment whether there are likely to be legitimate grievances or not.

tlear• 6 years ago

Interesting perspective, I have talked to two very seniors engineers there who love the place(one worked there 7 years, other I am not sure). One of the things they like(and I would agree) is that Shopify does rewrite old stuff when they move to something new, so there are no leftover unmaintainable untouchable .

BTW what it shows is that, he loves to code. He already built an incredibly successful company. Whatever is that he does works. I love writing code too been doing it since I was 14. One of the reason I do not want to move to Engineering Management is the expectation of abandoning it.

drchiu• 6 years ago

I’ve never met him, but having read his blog (before he took it down around the time Shopify IPO’ed) and his work in libraries like Liquid, Rails, and others, he’s an engineer/programmer at heart. Nice to see he’s able to hold onto some of that even after the journey his company has been through these 14-15 years.

thinkingkong• 6 years ago

This is such an interesting perspective. 99% of engineering managers dont or cannot write software any more usually as a rule.

nogabebop23• 6 years ago

He's not exactly an engineering manager, but he's definitely part of that 1% who don't meet your rule, so it actually fits within the standard perspective.

tinyhouse• 6 years ago

Well, it doesn't matter if they can or cannot. It's not their job to write code. Unless it's a very small team.

ngokevin2• 6 years ago

"shouldn't" is also debatable to include, because managing itself is a full-time job

cortesoft• 6 years ago

Where do you get the 99% number from?

trustfundbaby• 6 years ago

Thin air. This is the internet after all.

rmk2• 6 years ago

For a moment I confused this gentleman's name with that of the character in Arrested Development (the "never nude" Tobias Fünke?)

But why or how on Earth would a person like Tobias Lutke find the time to do this? Are there other people who run multibillion dollar public companies who poke around in the codebase?

rock_hard• 6 years ago

Zuck shipped code all the way until 2010 I heard

mmahemoff2• 6 years ago

Bill Gates was famously coding as a public company CEO and revising a lot of code he was unsatisfied with. I recall rumours he was politely urged to stop it.

"Ultimately he had to force himself to stop revising and perfecting his peers’ work. “I had to say to myself, ‘Ok, we’re going to ship code that I didn’t edit,’” he said. “And that was hard for me, but I kinda got over that.”" https://www.cnbc.com/2018/05/10/bill-gates-quit-this-bad-hab...

One source suggests he last created production code in 1989, three years after MS went public (https://www.quora.com/When-did-Bill-Gates-last-write-code-fo....)

rmk2• 6 years ago

To be honest, creating an Operating System, the ecosystem surrounding it, etc. is technologically much more complicated than a lot of the stuff that web-based companies are doing nowadays. Besides, Microsoft was definitely a much more technical company that sold a purely technical product, so it is not entirely inconceivable that founders were deeply technical in a way that today's founders aren't.

I also imagine that there was much less 'adult supervision' of Microsoft. They probably did not take (any?) VC money, did they?

The scene today is very different, so I wonder how such a thing happens in modern-day startups. As an investor, I would be alarmed at such egregious misallocation of valuable founder/C-suite time and brainpower.

EDIT: I have read many an anecdote (from Joel Spolsky, for example), wherein Bill Gates was said to have pored over the minutest specification in documents spanning hundreds of pages and understood all the subtlety involved. Once again, how on Earth did Bill find the time?

desireco42• 6 years ago

I hope he can continue to do his craft forever tbh.

orsenthil• 6 years ago

For folks who don't know, Tobias Lutke is a CEO of Shopify. https://en.wikipedia.org/wiki/Tobias_L%C3%BCtke

BTW, I am not surprised. A technical, programmer, CEO can always write code. Nothing wrong with that. It's just like an exercise. It is amazing that is able to do that.

joeleet• 6 years ago

He’s also a self-proclaimed StarCraft 2 fan. He’s donated cash to tournament prize pools and recently hired beloved former pro TLO of Team Liquid to Shopify. https://esportsobserver.com/dario-wunsch-joins-shopify/

ignoramous• 6 years ago

He's also a lurker on news.yc: https://news.ycombinator.com/threads?id=xal

gregors• 6 years ago

He also gave a talk a long time ago I really liked. Seems the actual video is gone now. :(

https://www.infoq.com/presentations/lutke-rockstar-memcachin...

baxtr• 6 years ago

Playing the devil’s advocate here: he surely doesn’t need to do this. I assume there are enough great devs at Shopify. So why does he do it? Because it’s fun? Because he can relax? Maybe his time as CEO would be better spent not coding?

sharkweek• 6 years ago

I worked at a SaaS company where the founder CEO had a sales background. He'd hop into the inside sales (read: smile and dial) team desk once a quarter or so and practice pitching cold-to-warm-ish leads over the phone.

It was super motivating for the reps, mostly junior in that role, to see their CEO doing it and remaining empathetic to what they're facing every day.

Not sure if that's like... the best reason in the world when he could be doing other more important things, but it was definitely a morale booster for the company to see him in the pit once in a while.

ramraj07• 6 years ago

What better way to keep tabs on the state of the engineering fundamentals than actually try to write code?

BenGosub• 6 years ago

Recently I've heard this episode where the Shopify developer casually mentions the fact that their CEO Tobin Lutke has made some commits to their code. In my opinion they mention this in order to show that Shopify is not your ordinary top-down corporate environment.

It's also good PR. That's why it's also mentioned by FB employees that Mark Zuckerberg has also made some commits to the their codebase.

BTW I

codecamper• 6 years ago

imo Shopify is a company barely in control of its codebase.

rubyfan• 6 years ago

Guy Raz interviewed him on How I Built This last year, great listen

https://www.npr.org/2019/08/02/747660923/shopify-tobias-l-tk...

jes5199• 6 years ago

I don’t know about Shopify specifically, but in general I think this is a red flag. Are you going to be able to leave honest feedback on your CEO’s PR? can your CEO run a company and also continue to keep up with best practices for your language/tooling? Will your CEO go along with code style consensus, when he/she hasn’t had time to participate in any of the conversations?

naskwo• 6 years ago

Check out "No Rules Rules" by Reed Hastings and Erin Meyer. If you can't leave (constructive) feedback/criticism to your CEO, you're working at a sick company.

cocochanel• 6 years ago

Yes, a high school intern did that actually - https://news.ycombinator.com/item?id=20490964

intellirogue• 6 years ago

Have worked in a couple of jobs where the CEO or CTO still raised PRs, and they were really treated no differently. There was definitely a lot of code style changes requested, because as you say that knowledge might be out-of-date.

I suspect it might be cultural - I've definitely got the impression that in some countries CEOs see themselves as "gods" who should not be questioned by the minions.

melvinram• 6 years ago

There are numerous comments that essentially are saying "he shouldn't be doing this." That's shortsighted. You don't have insights into how this is working in practice for them. He, together with the other people at the company, have built an offering that is serving and satisfying millions of people. If there are problems with it, they are his problems to deal with. Why are you so hot and bothered about it? Focus on building your own cool shit. Be productive.

dna_polymerase• 6 years ago

Mark Zuckerberg once said that he does not write code anymore because they have a rule at Facebook, that says if you write the code, you have to maintain the code. I think that's a rational choice for a company of that size. Thus I'm not convinced Lutke's ongoing coding is that clever.

generalk• 6 years ago

I initially had that "oh no, CEOs shouldn't be coding" response too. For anyone who didn't read the transcribed segment of a podcast linked:

  > we started to see performance slowly degrading in terms 
  > of response times on the server, and eventually we kind 
  > of had to do something about it [...] Interesting story - 
  > the initial commits for that applications were Tobi 
  > himself

This doesn't sound like a CEO/founder who's unable to give up control of the software and is still inserting himself into the process unnecessarily. Many of us have been in that spot and, yeah, that doesn't work too well.

This sounds entirely like a founder who still solves problems with code, but then stands back and lets his company form a team around it if it's valuable.

No inside knowledge or anything, this just doesn't sound...bad.

_alex_• 6 years ago

post title is SUPER misleading. Here's the only paragraph in the transcript that mentions Tobias:

Interesting story - the initial commits for that applications were Tobi himself, who took it upon himself to start something, and as a prototype gets something up and running and make it as lean as possible to get started. And then eventually, that became a team, and we picked it up, and that became the project that we’re working on.

The Founder-CEO is in the commit history. Duh. So what? That doesn't mean he's writing code right now.

Tobias Lutke still writes code for Shopify

Related Stories