m104

Yeah but the execution still mattered. I'm a Flash / Shockwave fan as well but there's no point pretending that package was sufficient for the job it was pitched to do. Macromedia seemed to be on a really good track with Shockwave and Flash, but either didn't set up the technology for internet success, or really just sold out the goods with the Adobe acquisition.

In any case, take heart though. If we did it once, we can do it again.

Alright whippersnappers, let's chat about the history of railroads in the US.

In the early 20th century, US rail companies were beholding a very favorable situation: high demand to run loads of heavy freight all over the country, high demand to ferry passengers all over the country, and basically no serious competitors to either revenue source.

Now freight revenue was never going to be transformative to the industry, but it had the benefits of being reliable, un-fussy, and fairly easy to build a financial business around. Passengers, on the other hand, offered huge revenue potential, but had the downsides of being very fussy about things like safety and comfort and timeliness, along with wanting stations in convenient places and an ever-expanding rail network.

Students of US business management history should be unsurprised, then, that while evaluating the market that offered reliable revenue, versus the market that wanted large capital investments, the railroads overwhelmingly chose the freight market. In other words, US the railroad companies spoke and said we do not want passengers loudly and clearly.

The thinking was: passengers can do take the wagons and busses and cars and these newfangled airplane thingies, but freight is a guaranteed market for us! So the passengers slowly migrated to other form of transportation. But the kicker was, freight also wanted things like timeliness and access to an expanding transport network and, shockingly for the railroad execs, were willing to pay for it.

Add about 80 years, declining rail traffic, and tons of corporate mergers, and we have the sad state of US railways today: many residents have never seen a railway expansion or shiny new rail equipment, much less a real functioning passenger train. It's easy and comfortable to say that zoning or regulations or market forces allowed US rail to languish, but that would be ignoring the part where the industry did not want the customers in the first place.

To your first point, yes we're moving slowly towards a more general awareness that most employees are paid market (replacement) rate, not their share of value generated. As the replacement rate drops, so will wages, even if the generated value skyrockets. Unsurprisingly, business owners and upper management love this.

To the second point, the race to the bottom won't be evenly distributed across all markets or market segments. A lot of AI-economy predictions focus on the idea that nothing else will change or be affected by second and third order dynamics, which is never the case with large disruptions. When something that was rare becomes common, something else that was common becomes rare.

The anecdote about the 16-pin religion and the reluctance to use more pins is so good. It's often assumed that (later) successful companies were always making fantastic decisions in the earlier days, when in reality there were a few bizarre and harmful assumptions that were holding it back and needed to be forced out in order for rationality to prevail.

It's the "impact" style of technical write-ups: sell the problem and the scale, then present the solution, which is thus presented and understood through the lens of business and customer success.

Generously, this writing style is supposed to show the business value of teams and individuals, for promotions or other recognition. But yeah, it can be frustrating to read this style.

Oh lordy, the "two crews" bifurcation fully written down. What a fantastic way to ship until it becomes far too expensive to ship anything good.

Look, when we break the feedback loop back to the people who wrote the software in the first place, they get happier for a bit, you make some other people sadder for a bit, and then slowly your feature crew never want to be interrupted or bothered again and your customer crew can't get enough resources to fully fix anything.

Worse, your feature crews aren't learning anything beyond how to get those lines out the door, which will somehow get slower and more expensive as time goes on. Why? Because you removed the one fitness function on good software development which is to fully re-incorporate the negative feedback back into the source of development.

A real CTO leadership handbook would say clearly "it's your responsibility to help your developers improve, especially while shipping, and they're not always going to be happy about it."

Because a lot of tech workers today aren't actually job hopping and instead get very cozy in a job and a team and a career trajectory, which feels unfairly ripped away during layoffs for reasons that don't feel connected to their personal performance.

Only solution: sting operation

Answered above!

I often see rate limits framed as a way to protect system capacity, but not enough follow-through with understanding why, even with appropriate rate limits, systems can fall over and require manual intervention to restore.

The easiest way to explain this is with a simple sequence of events: the database has a temporary issue, system capacity drops, clients start timing out or getting errors, load amplification kicks in with retries and request queueing, load is now higher than normal while capacity is lower than normal, devs work hard to get the database back in order, database looks restored but now the system has 3x the load it did before the incident, other heroic efforts are needed to shed load and/or upscale capacity, whew it's all working again! In the post-mortem there are lots of questions about why rate-limiting didn't protect the system. Unfortunately, the rate limit values required to restore the saturated system are far too low for normal usage, and the values needed for normal operation are too high to prevent the system from getting saturated.

Fundamentally, there's really no way for a rate limiting (which only understands incoming load) to balance the equation load <= capacity. For that, we need a back-pressure mechanism like circuit breaking, concurrency limiting, or adaptive request queueing. Fortunately, rate limiting and back-pressure work well together and don't have to know about each other.

A few of extra considerations picked up over many years of hard lessons:

1. Rate limits don't really protect against backend capacity issues, especially if they are statically configured. Consider rate limits to be "policy" limits, meaning the policy of usage will be enforced, rather than protection against overuse of limited backend resources.

2. If the goal is to protect against bad traffic, consider additional steps besides simple rate limits. It may make sense to perform some sort of traffic prioritization based on authentication status, user/session priority, customer priority, etc. This comes in handy if you have a bad actor!

3. Be prepared for what to communicate or what action(s) to perform if and when the rate limits are hit, particularly from valuable customers or internal teams. Rate limits that will be lifted when someone complains might as well be advisory-only and not actually return a 429.

4. If you need to protect against concertina effects (all fixed windows, or many sliding windows expiring at the same time), add a deterministic offset to each user/session window so that no large group of rate limits can expire at the same time.

Hope that helps someone!

One aspect of this type of problem I missed from the article is whether the data mutations were applied evenly across transaction time. Data sets like these tend to be very active for recent transactions, while the updates fall off quickly as the data ages. If that's the case, applying a single query caching solution may not be a good fit and may always suffer from major tuning/balance issues.

If the data is in fact updated with clear hot/warm/cold sets, caching the cold sets should be extremely effective, the warm set moderately effective, and it may not even be worth caching the hot set at all, given the complexity proposed. Additionally, you should be able to offload the cold sets to persistent blob storage, away from your main database, and bulk load them as needed.

Finally, it can be faster and simpler to keep track of deltas to cold sets (late mutations that happen to "invalidate" the previously immutable data), by simply storing those updates in a separate table, loading the cold set data, and applying the delta corrections in code as an overlay when queried. Cron jobs can read those deltas, and fold them back into the cold set aggregations, making clean validated cold set data again.

Great article, BTW! There are entire database technologies and product dedicated to addressing these use cases, particularly as the data sets grow very large.

Notion has been a solid A- for shared to-dos, bookmark lists, recipes, trip planning details, etc, which covers most of our shared lives. The main issue with Notion is that mobile syncing can be very slow when we're out in the world, so using it to distribute a shopping list while we're at a grocery store, for example, has been unreliable.

For actual scheduled events, we just share our personal Google calendars with each other and invite each other to mutual events. No issues there.

The largest technical issue currently is just organizing personal media in a reasonable way. We're in the in Apple ecosystem so shared albums have generally worked for photos and videos, but it's clunky at best and doesn't cover other media or documents.

I recommend Russell Ackoff's writings as somewhat related and more to do with how systems of people and processes work (or don't). Here's a great place to start: https://thesystemsthinker.com/a-lifetime-of-systems-thinking...

They only tested on mice. Mice are nocturnal.

Data helps corporate management avoid personal responsibility, so it's not just "As a leader, I think..." but rather "Look at what the data is telling us to do." Yeah, it sucks.

Most valuable and uncomfortable part of this article:

  What I wish I knew was that if you’re an early company employee, it’s not
  likely that the skills you have on day one are the skills needed as the
  company scales to the next level. This sentence is worth reading multiple
  times as no one – not the person who hired you, the VC’s or your peers -is
  going to tell you when you’re hired that the company will likely outgrow you.

Precisely. Often times a rejection isn't saying that the candidate didn't meet the job requirements or wouldn't do well in the position, it says that the employer found someone who fit the role even better or knew that they could given past hiring experience.

Naturally, Musk lost it.

No, it's not natural for a grown-ass adult to lose it. Many people can face loss, disappointment, or criticism without losing it. Elon doesn't need to lose it and neither Tesla, Space X, The Boring Company, nor the Thai cave rescue effort were aided in any way by Elon's self-centered histrionics.

We don't need to pretend that his drive to be a visionary entrepreneur also requires him to lash out when he feels personally wounded, nor should we support attempts to rationalize this behavior.

Imagine how much more effective Elon would be in his professional life if he had a bit more self-control.

I'm guessing that, even if it wasn't more productive, the perks help keep Google employees at Google and not, say, at a competitor.

I'll second that sentiment.

The largest source of your tech debt may in fact be that brilliant one writing volumes of code destined to become the collective headache of future dev teams.

If you find yourself looking for a quick way to gauge the impact to people and property from an earthquake like this, I suggest you look for the USGS's PAGER report:

https://earthquake.usgs.gov/earthquakes/eventpage/us1000chhc...

They put this type of report together for each significant quake and it will inform you very quickly about how concerned you should be. In this case, both the Estimated Fatalities and Estimated Economic Losses are in the green bands, so nearly everyone should be ok even if they feel a bit shaken (ahem).

For sure! After years of running a personal mail server with a setup similar to what's described in the article, I moved to using a pre-built docker setup and haven't looked back: https://github.com/tomav/docker-mailserver

The easiest way to get this type of VM setup going is to start up the container on your mail host with all of the fun features (filters mostly) turned off, verify that the new mail container works as expected, then slowly start turning on features one by one so that if you happen to break something with a bad configuration you know how to roll back to a configuration that is functional.

As the article points out, relatability is the key to bridging the gaps of knowledge and understanding between people. I think, at least in the USA, there is cultural pressure against "talking down to" others, at it indicates either that the speaker is an outsider or is attempting to set themselves apart intellectually from their peers.

"I do know that over the years I've pointed out why something wouldn't work as planned and been told I was wrong and then later proved right. More than a few times it was an expensive lesson to be learned for those who ignored me. And pretty much every time I was hated for pointing it out."

The phrase "nobody knew" comes to mind, especially when used by person of authority to indicate why no preemptive action was taken before some accident/crime/disaster occurred. I mentally substitute one of the following phrases:

"significant barriers prevented the communication of obvious warning signs"

"fear of inconvenience and embarrassment prevented further consideration of the possibility"

"authorities were not able to be convinced in time"

Do you have support for logarithmic charts?

Some of those top examples would benefit greatly from a logarithmic Y-axis.

Very interesting reading. I've gotten many migraine headaches over the years that I can only correlate to having consumed major brand name orange juices. Minute Maid, Tropicana, that kind of stuff.

Since realizing this, I've stuck to plain old oranges and fresh squeezed OJ and not had a problem. The link between consumer OJ and my migraines, if one exists at all, never made sense ("it's just crushed oranges, after all") until now.

So, theoretically at least, my body could have a bad reaction to whatever is in the flavor packs that are used to make the tasteless juice taste more like real orange juice.

It is bizarre when companies operate this way, but, without defending the practice, you need to understand that few middle managers have the power to change this situation. Moving up the ladder, owners and executives will simply say "this is how we do things" and leave it at that.

Personally, I think this situation exists because many corporate leaders have simply decided to preserve their compensation/authority pyramid at the expense of keeping their best potential employees. There is less short-term risk and disruption to the pyramid, the theory goes, to hiring a new wave of employees than to augment the more experienced staff.

Remember all of this, though, for when you have the power to influence the compensation of others! This understanding can become a competitive advantage when you're building teams of your own.

It's true and unfortunately this has been my experience at a couple of previous employers as well. Given the difficulties of actually collecting good data and properly analyzing it, it's no wonder that the same analytical tools, used to great success in some companies, can be rendered useless or even harmful.

I'll add that a third reason for the obsession over analytics is that stat reports tend to take the focus off of the decision makers themselves. Managers who aren't confident in (or wishing to escape responsibility from) their decisions can simply blame the numbers and graphs and say they had no other alternative.

In some dysfunctional organizations, "judgement" and "decision" are bad words that imply bias and liability, whereas "report" and "conclusion" are pure gold.

Don't be too discouraged, though. There are plenty of employers who avoid using analytics inappropriately and know that great products and customer service are even better than stat reports in the long run.

I think there are two main sects in audiophile-land, really: perfection and experience.

The perfection side has scopes, meters, and blind testing among its tools. The goal is in fact "perfect sound reproduction". The problem here is that between the original source (microphone, usually) and the listener's brain is all of this stuff that distorts the sound. Removing it completely is really hard, and replicating that quality repeatedly is even harder.

The experience side has experts, rituals, and demos. The goal there is to reach the limit of audio experience, literally to hear what wouldn't otherwise be heard with lesser systems. The problem on this end is that "near-perfect" systems deliver less exciting and engaging experiences than what the experience experts deem to be better. They don't enjoy the systems calibrated to perfection as much as their own custom systems with tweaks and rituals applied.

These two sides really are in a bit of opposition, but it's nice to have both to drive the high-end market of audio equipment.

Don't give this page a once-over. Make sure you drag the diagram to "flip" it over. You can click on regions on side B of the diagram, too.

Impressive work!