jroseattle

This is exactly how we arrived at our solution. We needed to satisfy the constraint; locking was one means of addressing the constraint.

Maybe we were lucky in our implementation, but a key factor for our decision was understanding how to manage the systems in our environment. We would have skilled up with Redis, but we felt our Postgres solution would be a good first step. We just haven't had a need to go to a second step yet.

up to 50000 atomic, short-lived transactions per second

50000?

You need distributed locking if the transactions can take seconds or hours, and the machines involved can fail while they hold the lock. From my experience, locks are needed to ensure synchronized access to resources. Distributed locks are a form of that isolation being held across computing processes, as opposed to the mutex example provided.

And while our implementation definitively did not use a distributed lock, we could still see those machines fail.

I fail to understand why a distributed lock is needed for anything due to it's duration.

We reviewed Redis back in 2018 as a potential solution for our use case. In the end, we opted for a less sexy solution (not Redis) that never failed us, no joke.

Our use case: handing out a ticket (something with an identifier) from a finite set of tickets from a campaign. It's something akin to Ticketmaster allocating seats in a venue for a concert. Our operation was as you might expect: provide a ticket to a request if one is available, assign some metadata from the request to the allocated ticket, and remove it from consideration for future client requests.

We had failed campaigns in the past (over-allocation, under-allocation, duplicate allocation, etc.) so our concern was accuracy. Clients would connect and request a ticket; we wanted to exclusively distribute only the set of tickets available from the pool. If the number of client requests exceeded the number of tickets, the system should protect for that.

We tried Redis, including the naive implementation of getting the lock, checking the lock, doing our thing, releasing the lock. It was ok, but administrative overhead was a lot for us at the time. I'm glad we didn't go that route, though.

We ultimately settled on...Postgres. Our "distributed lock" was just a composite UPDATE statement using some Postgres-specific features. We effectively turned requests into a SET operation, where the database would return either a record that indicated the request was successful, or something that indicated it failed. ACID transactions for the win!

With accuracy solved, we next looked at scale/performance. We didn't need to support millions of requests/sec, but we did have some spikiness thresholds. We were able to optimize read/write db instances within our cluster, and strategically load larger/higher-demand campaigns to allocated systems. We continued to improve on optimization over two years, but not once did we ever have a campaign with ticket distribution failures.

Note: I am not an expert of any kind in distributed-lock technology. I'm just someone who did their homework, focused on the problem to be solved, and found a solution after trying a few things.

Congrats on writing and completing a book! I was involved in a few myself long ago, when I had the time available to contribute to those endeavors. In a world that often measures "the juice being worth the squeeze", I'm not sure authoring technical manuals would ever meet the criteria.

One of my personal photos I keep around was taken long ago in what was the biggest bricks/mortar bookseller. I was looking at the selection of books on Java available at the time. O'Reilly was the dominant publisher, and thus had several offerings on the wall. Most of the books were at least 2 inches thick. (If you were ever involved with writing a technical book in the early 2000s, you'll understand the publisher metrics at the time were based on the width of the spine on the shelf.)

Among the many Java manuals of significant girth was a small, THIN book with the title "Java -- the Good Parts". :-{}

The vulnerability was addressed with the release of Docker Engine v18.09.1, but it was not included in subsequent major versions, causing a regression.

Without further information, this sounds like code introduced in a hotfix that wasn't merged back to feature branches.

Surely it's not that simple?

Resume fraud is rampant.

So is interview fraud. The remote-interviewee-answers-questions-while-her-face-reflects-windows-popping-up-on-her-screen is tiring at this point. So, I decided to find a way to inform me if someone was being fed answers in a tech interview.

Behold, the low-tech whiteboard. Also known as a piece of paper and a pencil. With the candidates I've run into that do not pass the "smell" test -- where I think they are being fed answers -- I ask them to draw some things, on paper. It's not a true validation, but it gives me something of a clue.

I ask for a simple diagram. Different services in a network, for example. Or a mini-architecture. For their level, I'll ask for something that should be drop-dead easy.

I ask them to show me their drawing.

The responses I've received run the gamut of "I don't know" (after 5 seconds of deliberation) to "I don't understand the purpose" (after 5 minutes of silence) to "I need to shut off my screen for a while" (while refusing to explain why) to "it depends if your cloud is AWS" (not in any way remotely related to the question.) I did have a candidate follow-up with a series of questions about the drawing, which were feasibly legitimate.

This hand-written diagram is not an absolute filter (I've only used it maybe four times), but rather it can confirm some suspicions. I think I can generally gauge honesty from questions/tasks like this. And that's really what I'm after -- are you being honest with me?

It's imperfect, but it has been helpful.

What I find more important than the timestamp format -- the timestamp source.

Centralize from where the ts is set (the db is a great place to do this.) Don't let client-code set the timestamp.

You learned some great lessons there, but I would challenge one item early in the "script":

2. Verify if it’s a problem from search volume.

It contextually depends, but correlating a problem-to-be-solved with search analytics can be really tenuous. I'd suggest a different phrasing:

  Verify it it's a problem by speaking with customers.

Do it better the next time!

I really like your on-device storage approach. I did something similar with an app for my smartwatch a few years ago to prompt/track my gym workouts. At the time and on the particular platform, other apps felt very spammy/scammy, and I just didn't care to go through the hassle of pushing/pulling my data with another application for what seemed like little to no value to me.

I haven't touched it in a while, but I've been thinking of a sync-to-another-device capability that I could then use with a locally-running AI instance to dig into. (Disclaimer: this is purely exploratory on my part; I'm sure someone will say 'why not do so-and-so'). As always, your mileage may vary. :-)

Way back in the day, I picked up a cheap ($5!!!) copy of Readings in Database Systems from Michael Stonebraker. I found it fascinating to read the original papers that proposed concepts, and then to see those concepts implemented and become the norm.

What I didn't expect was the amount of drama within the context of those papers. In Codd's original paper on relational theory in the 60s, he spends a chunk of time dedicated to talking about IBM as "the man". Hilarious.

This is generally good advice with depth, but I would add a disclaimer: organizational practices and idioms should be taken into account. A few examples where these points would need some adjustment in my org:

We're not crazy about irregular cadences for project update communications. Why? Because there are many projects of complex size and shape going on. The stakeholders of my project care about many of those other projects as well, and they plan their schedules expecting to consume your update at that time.

Another case how we operate: the unpleasant surprise. Don't sugarcoat it, and bring things to attention as soon as possible. Rip the bandaid off, as we say.

As for tone and content, we focus on the purpose and delivery on that purpose. This is established early in the project, and our stakeholders understand both before the project ever takes off. We anchor our updates on those aspects, as it gives stakeholders an understanding of progress, no matter what the project update includes.

As April Fools jokes go, this is really well done.

On one hand, this could easily be read as the musings of some overzealous, re-awakened and re-charged techbro/middle-manager who has tried and failed a few times. But this time, things are gonna be different....

But also, there's a modicum of truth in there. Only the experienced will recognize the red flags buried in the brush-away commentary.

Yes, we had some "stakeholder training" that was necessary.

The biggest complaint: "I don't understand why this takes so long."

Our stakeholders were mostly plant operations people -- users, not engineers. We re-framed the discussion to focus on WHAT they didn't understand. They didn't understand how software comes together, gets deployed or updated, etc.

I explained that it wasn't important that they understand how that occurs (or, if they do, please join the engineering team!) This literally took pressure off of them to be knowledgeable. I pulled a few of them in as "first-pass quality control", ensuring what we built would work for them and their teams.

Immediately, I had advocates for us and interested stakeholders willing to participate as partners.

I recently inherited a project that was "late". The team and project were 6 months into execution, and had nothing to show for it. The status on my Day 1 was "estimates were missed -- now what?"

On appearances, everything should have been just fine. The SWEs and product leader were running two-week sprints, with all the ceremonies scheduled on all calendars (well, everything except the end-of-sprint demo.) Every two weeks, a status powerpoint slide deck came from the product leader espousing all the fantastic capabilities that were recently implemented by the team.

I needed to understand what was going on. I chose to observe everything for two sprints (and not interrupt how everyone worked) while I focused on establishing a clear and comprehensive view of The Project.

What I discovered:

- An ambiguous and unclear definition of the project. Key facts, problem statement, overall solution ... did not exist. The product leader could give you some spin of this off the top of his head, which would invariably change from time to time.

- Work stories (we used JIRA) were garbage. I'm an engineer and even I couldn't understand what work was being completed.

- No capacity for the team to build & deploy software in a reasonable manner.

- No agreement or understanding of expectations among stakeholders.

So, what did we do?

1) Defined the project, with stakeholders in agreement.

2) Devised a solution, based on the project definition.

3) Broke down the work necessary to implement and deliver the solution.

4) Determined individual lengths of time to complete the list of work.

5) Reviewed everything, finding areas of risk and potential further problems.

It sounds like BUD (big upfront design) -- but in practice, it was Big Upfront Discovery. We structured work and components to ensure we had plenty of two-way doors (only a few one-way doors, based on requirements from stakeholders.)

And with that, we made our estimate. Another 6 months. The stakeholders did not like the timeline, but agreed "as long as we make that date." We provided weekly status updates, leaving out no details. Our stakeholder meetings went from contentious and confused to understanding and brothers-in-arms. We made the date and we didn't kill the team (morale actually went up.) The stakeholders loved the resulting solution.

****************

No magic here, just a focus on attention to details and honest assessment.

I make my own pasta when the sauce warrants it.

In my experience, homemade noodles will absorb sauce much better than the store-bought version. And, with a good sauce, the result is just _so much better_. You won't find me making my own noodles if they're going to be paired with Ragu, though.

Here's what is unsolved for me: store-bought garlic bread in foil to heat in the oven, or roll-my-own garlic butter and bread?

I'm not sure if this was intended to "flip the script" on the talented & smart, as this is sage advice for any entrepreneur. It wouldn't surprise me if the author was trying to give the "T&S" permission to do things without getting in the way of themselves.

I recently completed a project on behalf of a privately-held company in the financial sector. Big, successful, and very corporate. The owner (a single individual) was in town and spoke to a few leaders (biz, tech, marketing) in my group. His message: get in the weeds, learn the business, understand every function, ask questions, get outside your comfort zone, be persistent, act with urgency, and finish things. (Those were literally my notes from that meeting.)

He gave his team room to do the things necessary to let their talent & smartness be most effective.

let’s just consider it an expensive learning opportunity to redesign the system so it can’t happen again.

Also, Zuck: but hey everyone, keep on moving fast and breaking things!

Well-executed RCAs are just so satisfying to me. Blameless culture is absolutely critical to getting to the "right answers".

Across the teams in my company, I hit nirvana when reviewing something like a major/impactful outage and the involved members in discussion are drilling down to the the essential state and sequence of events that caused said outage. Focus on actions and outcomes, including both those central/peripheral to the outage and those with zero knowledge prior to the event. It takes a high degree of trust in yourself, your peers, and your organization to get to nirvana.

Go through a few of these in a proper way, and a simple principle tends to emerge: if your mechanism is dependent on the perfection of humans, it will eventually fail. The only real discussion beyond that is basically what to do next -- do we need a mechanism that protects against human imperfections? Is the cost of implementing a solution worth the mechanism it would be designed to protect? Can we live with the infallibility of humans in this scenario?

Organizations that can achieve this level of discourse have a distinct advantage in execution.

This book is still on my shelf. When I first ordered and received it, I sat down and read it cover to cover.

One of the things I really enjoyed was seeing how differently all of the interviewees worked and went about their business. They didn't seem to work that much differently from me, which made their work that much more relatable.

I was thinking you were asking questions and/or engaging to understand, of which I've made an incorrect assumption.

Allow me to clarify a few things:

- Format: we don't dictate format on docs. In terms of sharing important information, it's irrelevant. Dealer's choice.

- Audience: we are writing documents that are internal to our team, but also for consumption by other groups. Audience is engineering-focused.

- Consumption: we refer to documents as a way of learning all the things before asking uninformed questions to team members. RTFM is an expectation in my groups.

instead of answering their very reasonable and pertinent questions you instead accuse them of being either incompetent or lazy.

These expectations aren't random; they are well-known, part of project deliverables, and non-negotiable. Existing projects show the quality of what's been produced in the past, to help with an understanding for anyone documenting what they've built and want us to depend on in production. After that, we definitely help those who still require help, but this isn't a surprise.

I'm a documentation zealot. My eng teams understand we treat docs as products; they are project deliverables, and we allocate time to ensure we produce and maintain concise, clear and unambiguous content about the services we operate.

Nonetheless, I always have those team members who insist it's not their responsibility. A recent exchange I had with one of our engineers:

  Eng: I can create our entire service portfolio, but I just don't know how to explain it in a document.

  Me: I see. How about we start small, and just simply explain how service 'A' works. It's not too complicated.

  Eng: Well, I don't know what outline to use. I don't know how to publish it. I don't know what content I should add. I don't know who would read it. This will take too long. It feels like busy work.

  Me: So, if you're unable (seems like it) or unwilling (hopefully not) to do this work, what would you suggest we do to ensure we inform the rest of the company about service 'A'?

  Eng: I don't know.

  Me: Until we DO know, service 'A' will not be deployed.

I used to run a fairly good-sized light show, until it became too much work (job change, kids/family, etc.) xlights powered my shows and taught me a lot about the protocols and hardware and everything else that comes with it. It's amazingly good software with input from a lot of talented individuals.

It's been 20+ years but I loved this shirt for it's SQL agnostic-ity:

https://www.pinterest.com/pin/lol-code-geek-select-from-user...

(This shows the shirt you could buy once-upon-a-time from thinkgeek, which appears to be no more...)

I believe they used beefy-t brand shirts, which was like the nicest thing you could get for t-shirts at the time. (Personal opinion, don't @ me.)

That's right. It's all part of the back-and-forth of good faith.

"Good faith" doesn't mean being dishonest. Instead, it's about showing a little empathy between two people.

This situation sounds more like people trying to cover their ass so they aren't blamed for something later. Work to get things accomplished, not to avoid some kind of criticism.

So, rather than being these types of people -- be a person who helps get something accomplished and maybe improves this situation along the way.

I've had managers give stupid orders

I'd suggest that "good faith" means turning those stupid orders into sane orders. In a case like this, help your manager -- educate them.

If your relationship is tenuous, I get this can be hard. Show your good faith and explain to your manager in clear detail why something is stupid. Manage up, as they say.

Why can’t we tell management that a question they asked is non-sensical?

You should absolutely do that (I tell my team to inform me if what I'm asking doesn't make sense.) If I'm coming to someone with a request, it's because I think they're the right person to address it. Just tell me if I'm off-base.

FWIW, I encourage everyone to not care about the hierarchy, so that may not work in every place.

Having been at different levels in engineering (from junior up through C) there are a few things I've learned along the way.

(1) Everyone needs to operate in good faith. I've observed leaders ask their direct reports for something off-hand, without giving it more than 5 seconds of thought. I've watched direct reports operate as a "human command line", requiring precise syntax before they'll act. Don't be either of these type of people.

(2) If you're a manager/leader, there are no "one-off" requests. Any ask carries an implicit priority over everything else. You have to work within the overall system and environment, otherwise you are forcing your team to make choices about those things. Don't be this kind of manager/leader.

(3) If you're the one receiving an ask, help your requestor understand what's involved. Eliminate implicit assumptions; that's where the dragons lie. Communicate, inform, educate. Nail down what the requestor wants and be sure you both agree.

We do these things in the name of efficiency, of not wanting to waste time if it's not necessary. I've found if we all try to help each other in these requestor-requestee situations, it generally makes life much more tolerable.

Trying to summarize what I'm reading here.

"Enterprise-y" appears to be a catch-all term for an organizational culture built on a foundation of lack-of-trust where interactions among people need to be gamed in order to accomplish...well, anything. And the presentation covers strategies of how to be effective in that kind of environment.

I like some of the tactical suggestions, which apply in any organization trying to get things done. I dislike all the actions that are based in hidden agendas or ulterior motives, no matter the intent.

As always, your mileage may vary.