bjacokes

I've read about this strategy and am giving it a shot this year.

If I understand it correctly, the reasoning is that until the late summer, the underground root system is putting its energy towards growing the stalks and leaves, so you _want_ it to grow as much as possible. If you trim daily, you'll never let it get into the rapid growth stage where it's really depleting its energy reserves. In other words if you trimmed daily or monthly til the end of time, eventually you'd kill the plant, but it might ironically die more quickly if you trimmed monthly.

It's a bummer knowing that it'll be hard to fully eradicate it given how widespread it is in the area (I'm also in Massachusetts), but I guess there aren't really any permanent victories in life when you think about it.

This is a bit like saying earnings going up isn't good for stockholders, because they would have been charged a higher price to buy the stock if people had known the earnings were going to go up.

Once you've taken out a fixed-rate mortgage, inflation absolutely has the effect of reducing the value of the debt you owe. It's more if you're about to take out a mortgage that you're rooting against (expectations of) inflation, as lower inflation will also serve to decrease the prevailing interest rate.

It's just like any hiring process: get referrals from people you trust, look at prior work (previous purchases/sales in this case), learn the right questions to ask when interviewing.

Sarah & Duck and Bluey are my favorite kids shows, but they're very different. S&D is more quiet and contemplative, whereas Bluey tends to be more playful and energetic. My daughter grew out of S&D and into Bluey at around age 4.

I used to be frustrated that the parents in Bluey seem to have endless energy and attention to devote to their kids – at least with S&D, I didn't feel like there was such a lofty version of parenting with which to compare myself. But I've come to realize that the playful and fun dad in Bluey is a much better role model, for me personally, than the dads who are just sort of vanilla and kind (Daniel Tiger, Doc McStuffins), not to mention the dads who are just awful (Peppa Pig). Physical comedy, imaginary play, and committing to the bit, are all great for having fun while also connecting with young kids. Sarah and Duck obviously didn't teach my any of that, so it's been a useful change of pace as my daughter has gotten older.

It sounds like a lot of it just went to unprofitable SaaS companies

U.S. GDP is north of $25 trillion and has grown 75% since 2009. Annual VC investments appear to be in the $200-300 billion range, from a quick search - presumably not all of it going to unprofitable SaaS companies. I'd go with the statistics over the gut intuition here.

There are plenty of never-profitable businesses that have gone bankrupt in the past. See e.g. pets.com from the dot com bust. And this downturn has seen "functioning" businesses like SVB and First Republic go under too. So I'm not sure this time is different, at least to the extent you're suggesting.

I do agree that this era of low interest rates has led to some companies getting absolutely massive without ever turning a profit. But hey, that playbook worked for Amazon. Most tech companies have substantially lower capital requirements than WeWork, and can weather this sort of downturn as long as they have a reasonable cash position. Sure, Uber might not make back the amount of funding it's burned through, but that's different than it having a business where the unit economics will never work out.

Protecting bike lanes is a hugely impactful problem that is also far more expensive to solve. That doesn't mean we should be dismissive of a solution to a less impactful (but still important!) problem. Think how many orders of magnitude less money it would take for a municipality to buy and loan out some bike sweepers than to fully redesign its bike lanes.

Any examples come to mind? This is true of most Internet communities but I've actually found that one to have more informed opinions than usual.

I go to https://www.reddit.com/r/bikewrench for bike maintenance Q&A. The responders there are often extremely knowledgeable, although it doesn't have the article structure of Sheldon's site.

Adjusting water chemistry is extremely prevalent in brewing. For example, if you've ever had a hazy IPA, part of the softer bitterness comes from high levels of chloride in the water. The cost of common brewing salts (gypsum, calcium chloride, etc) is a small fraction of a penny per beer.

I'm guessing that the beer brewer you spoke with was talking about the cost of buying distilled or RO water, as opposed to the cost of the water adjustment itself. It's probably a lot more economical if you're cleaning and reusing graywater, vs. trucking in distilled water, or running municipal water through an RO filter and essentially paying twice for water treatment.

They're worth $80m today, and then maybe $82m next year, $84m the year after, and so on until they're worth $100m at maturity. (Obviously these numbers depend on current and future interest rates, and you'd be earning some interest in the meantime).

As I was trying to point out to the parent commenter, conflating "$100m today" with "$100m at maturity" leads to clear contradictions, like saying that a bank could earn $20m on paper simply by buying bonds trading below par value. Or to put it another way – if bank A holds $100m face value of 10-year bonds yielding 4%, and bank B holds $100m face value of 10-year bonds yielding 2% (but worth, say, $80m at market price), how can you claim that those banks are on equally good footing?

Valuing liquid bonds at par value is pretty clearly a hack to reduce volatility and increase confidence in banks' balance sheets, even if some people in the comments seem to view it as a more logical way of accounting. (Although to be clear, I don't mind companies doing their own fuzzy math as long as they give investors enough information to do proper due diligence. It's similar to the non-GAAP earnings that a lot of tech companies report.)

You're saying that if a bank paid $100m for low-yielding bonds in 2021 which are now worth $80m, those bonds should be valued at $100m on the bank's balance sheet. What if a different bank pays $80m today for the same bonds? Should they be able to show an immediate $20m increase in their book value because those bonds are "worth $100m"?

In 2008 the Fed was decreasing interest rates, which helped support asset prices. Banks held a lot of bad loans which were worth much less than their balance sheets showed. Both of these factors could have caused the unrealized losses in 2008 to look somewhat small, but the high leverage at banks caused forced asset sales, and the uncertainty around credit losses led to asset prices tanking.

In 2023 the situation isn't necessarily worse, but it is certainly different. Asset prices seem relatively well-understood, in that their declines are a straightforward function of interest rates as opposed to an uncertain function of credit losses. Bank leverage is less than in 2008 as a result of regulation.

If the situation in 2008 was "some banks are _super_ insolvent, and it's hard to tell which ones", in 2023 it seems to be "some banks are mildly insolvent, and it's fairly clear which ones". A mildly insolvent bank can probably stay afloat as long as it continues to have access to capital, which the Fed is giving them. But if people start withdrawing their deposits from one of the mildly insolvent banks, it will become increasingly difficult for that bank to dig out of even a small solvency hole, so there's still some uncertainty as to whether the Fed lifeline is enough to save them.

Well this is one of the wilder comment threads I've been a part of. Good luck with your investing.

You keep referring to this 40% number and calling it a "significant discount", when in actuality it's only a 7-8% decrease in asset value that would be needed to wipe out shareholders ($211B assets vs $195B liabilities on their latest balance sheet). In your original comment you mention the bank selling assets at a 8.5% discount as a point _in favor_ of your thesis. You seem to not only be missing the contradiction there, but also making a vaguely optimistic case for investors getting money back in an asset firesale.

I don't mean to dunk here, I just get nervous seeing someone propose a super-high-variance trade that goes empirically wrong an hour later, and then quote Benjamin Graham. As the responder above said, given the size of the dodged bullet you should really be updating your priors on investing strategy, but you seem to barely even regard your thesis as mistaken.

You're mistaking assets and equity. A 40% discount in asset value absolutely would wipe out shareholders. To be a bit blunt, given your thesis here I would advise against trading in individual stocks, at least in the banking sector.

See: non-response bias [1]

Parent comment should've said "could" instead of "will most likely", but their point is correct.

[1] https://en.wikipedia.org/wiki/Participation_bias

Great question, although I'm not sure there's a concrete answer to it other than "it depends". You can think of that metric as representing the number of logs that haven't been garbage collected, so as it goes up, performance will get worse.

If you're seeing spikes in RollbackSegmentHistoryListLength that coincide with dips in DB performance, you've probably identified the culprit. In the scenario described in our post, that metric would have grown monotonically for the duration of the long-lived ETL query – probably a more overt problem than what you're describing with short spikes to 100,000.

The Accidental Integration Test on Purpose. Larry David would be proud.

From what I can tell, they provisioned their DB instance(s) in a single AZ, but weren't aware that Aurora automatically provisions its own storage and always uses multiple AZs. We touch on the separation of compute and storage in the post.

I think the surprise is that it's not possible to have a truly "single AZ" Aurora database, even though you might have thought you provisioned your DB instances that way.

This seems plausible given our understanding of the database internals. In general we found our AWS contacts to be knowledgeable and forthcoming about complex tradeoffs between Aurora and vanilla MySQL, even if some of that information is hard or impossible to find in the docs.

For Aurora MySQL, the default for read-only replicas is repeatable read. As we mentioned towards the end of the post, read committed support appears to have been introduced to Aurora MySQL just last year. But you're right – now that it's supported, switching to read committed is by far the easiest fix.

No idea why people would be using serializable reads for ETL jobs though! :O

You seem to be misreading the parent comment. It isn't complaining about small businesses getting loans, it's complaining about them having to pay a higher interest rate than large corporations. (Presumably the reasons for that are a combination of the corporations being "too big to fail" and their loans being collateralized.)

Your question feels pretty sarcastic, but I think it'd be great to write a post about our process for projects specs and reviews. We do have an old post on the blog about code reviews that you could read, but it's more focused on the cultural approach to making code reviews less intimidating for junior engineers, less about the details. While we don't have an "architect" title, a fair portion of our more recent hires have 10+ years of experience (I don't have exact numbers though), and we're working on the best way to coordinate efforts and improve our system-wide architecture.

In general, I think it'd be great if more companies communicated their engineering practices and war stories externally – I'd love to read such posts myself! Unfortunately, it takes quite a bit of time to write a post (for engineers who are already stretched thin), and it seems that being honest about shortcomings at an early-stage company is an invitation for people to be personally disrespectful. It is what it is, but I imagine that's one reason we see such posts from only a small handful of startups, and the subject matter is often cherrypicked and sugarcoated.

So anyway, thanks for engaging with our post all the same – and maybe we'll be back with a post on architecture reviews in 5 years when all the kinks are ironed out :)

I don't disagree with most of what you're saying. The Nth engineer at a startup rarely looks with admiration at decisions made by the (N/10)th engineer – but it was those decisions which helped the company grow to its current size. Likewise, I think most of us will be happy if the company 10x's again. Then some super-duper-senior engineer can look at the decisions we're making now – they're not perfect, but we're doing the best we can with our current knowledge and resources – and gripe about them. The circle of life goes on.

FWIW, I don't believe Node was chosen specifically for its concurrency – it was just the language chosen for the entire stack by the company's founding engineers, and lives on in just this one service.

In one case I cited elsewhere in the comments, an engineer had called ramda.uniq on an array of nested objects which was occasionally very large. When calling into external packages, I don't think we have as much control over yielding to the event loop, but I could be wrong. I know that there are some JSON/regex libraries that give you some protection on this front.

I agree that it would be nice if all developers were infallible – I'm reminded of a friend describing their company, where "we don't write tests because we all write good code". At a certain point, you have to look for processes – linters, monitoring, testing, language choices [1] – where people can't shoot themselves in the foot. (Code reviews being only moderately less fallible than a single engineer.) It's not enough to just say "be better" whenever bad code is written.

I think when the decision was made (years ago) to handle a single request per container, they couldn't find such a process to prevent event loop blockages, other than migrating an already-large codebase away from Node. As others have pointed out, maybe such a migration is necessary – after all, event loop blockages are still an inherent risk because of how Node works. It's just a lower risk than it was a year or two ago, because we've significantly improved our usage of the event loop, and also have tooling in place to catch blockages before they become an issue.

[1] https://news.ycombinator.com/item?id=18564643

Hi, Plaid engineer here (not the author, but I helped with the post).

I don't think we've tried to assert that the old system is perfect. We went into some detail in the post about why it took us this far. Certainly, the single request per container approach wouldn't scale if our unit economics were different. We didn't get into this too much in the post, but the Node service sits behind a couple of layers of Go services, so the we had more control over scaling API traffic than it might appear.

Likewise, I hope we didn't give the impression that the new system is perfect. We've explored other languages for integrations in the past (even Haskell, at one point), and are continuing to do so. A migration away from our years-old Node integrations codebase would be a massive undertaking at this point. Absent that, it doesn't seem consistent to say "you're incompetent for handling 1 request per container" and also "you're incompetent for writing this post" – if you believe the former then it makes sense to be an advocate for this project, at least until a language migration can be done.

I think the set of hoops we had to jump through in order to add concurrent requests without adding latency is a good demonstration of why we didn't do this sooner. It wasn't a massive undertaking by any means, but it wasn't trivial. At any rate, we're not really looking for a gold star here – just putting this out there and hoping this will be useful for others who are, as other commenters have put it, building their own "Frankensteins" :)

I say "possible" because our system observability was less mature even 12 months ago. Firefighting 10 different root causes of memory or event loop issues without the right tooling in place would be a nightmare. That's why we did a deep dive into the tooling that we considered to be a prerequisite for this project – hopefully it's helpful for others in our situation.

Different companies make different decisions when weighing ROI against architecture concerns. We're heavy on pragmatism and impact at Plaid, so it's quite intentional that we don't fall all the way on the latter end of the spectrum. I appreciate the discussion in the comments as to how effectively we are balancing these two concerns – certainly this is an area where reasonable people can disagree.

While we were worried about event loop blockages causing outages, another more subtle problem would have been if event loop blockages doubled our user-facing latency. (If you read the section on latency ratios, you'll see that comparing parallel vs non-parallel workers was the most useful stat in figuring out how effectively we were using the event loop.) It definitely gave us peace-of-mind to know that event loop blockages wouldn't have an effect beyond the requests they're processing.

Honestly, the accounting for which would've been higher impact – investing in parallelism earlier, or adding infrastructure and having more resources to devote to other pressing needs – is difficult to do, even in retrospect. There was surprisingly little effort required to get to 4,000 node containers in an ECS cluster, other than deploy speed issues which we talked about in a previous post [1]. But it's possible this migration process would have been easier if we had done it sooner.

[1] https://blog.plaid.com/how-we-reduced-deployment-times-by-95...

Fixed the images about half an hour ago, sorry about this!