logicfiction

I think they presented it wrong for impact. If instead of "killed" and "eliminated" they were saying that the internet made it possible to proceed without a sales team and that cloud computing made it possible to not need a bare metal IT team, it makes a little more sense. That is to say, it's not going to be the most effective way, but it went from impossible to at least viable. Similar to music recording. It used to be you needed competent instrumentalists and access to a commercial recording studio. Now you can do it all at home on a laptop and it is viable, people are successfully doing it. But in large part the most successful music is still done using the same professionals that existed before.

In a similar vein, Instagram was the first platform where I actually took a pause and realized someone had actually figured out how to give me appealing advertising over the internet. I don't use Facebook to know how they differ, it may be practically identical.

My experience is that I use my Instagram account not to interact with my social group but to engage with content I like (primarily art content like sculpture, painting, light/projection, music, etc). It's actually a really pleasant experience and I'm doing my best to protect my groove in the recommendation algorithm. I basically get an effortless feed of art events in my area without me needing to subscribe to a newsletter. And it also engages me with a lot of independent artists selling unique items that I like.

If you do it by taxing the rich, you wind up hobbling and punishing your most productive citizens

We may have different definitions of "rich" here. I don't think you can call them the most productive citizens. I think most extractive would be the more accurate label.

And the stimulus checks were short lived, not universally distributed, and came in under $1T. The majority of the money spent was on other sources, even if you want to include the boosts to unemployment in the "UBI" category. As others mentioned it's also conflated with other factors like supply chain failures and massive business loan fraud.

and the median nominal US home value is up 1728% since 1970[1].

https://fred.stlouisfed.org/series/MSPUS

The 30 days part is most likely a legal compliance bit to cover their asses if their backing data systems ever take a big dump.

They need to use an asynchronous system to be tracking when your chat becomes "finished" and then likely queue it up to a system looking to propagate deletes. They have to choose some kind of SLA on that and probably went with a common data privacy user data deletion window of 30 days.

Also a lot of things at Trader Joes are just white labeled versions of other products. As in, literally the same product, just inside packaging that says "Trader Joe's <Similar Name to Other Product>"

Was MBS (mortgage backed securities) a typo here? My understanding was SVB had a huge long term treasury position on the books from before interest rate hikes that was in the red against the market. And their second largest tier was direct loans to things like startups that weren't asset backed but instead default to warranties on the companies taking the loans (I'm not close enough to this to really say that more specifically).

Post 2/2, previous post at https://product.hubspot.com/blog/savings-logging-part1

Previous discussion: https://news.ycombinator.com/item?id=34551290

Why does high school have to go first in the morning? When I was in elementary school I would wake up at 5:30-6am naturally and watch TV before school. Come high school and I was constantly not getting enough sleep with the earlier start times and setting my own bed times. I think high school aged children have more need for a later start than elementary school aged children.

I would have to defer to one of my colleagues for most of the details on running data infrastructure on Kubernetes, I’m not that close to that domain. The major ones we run are HBase, Vitess (MySql), Kafka, Elasticsearch, Memcached, and Zookeeper.

As for the call sampling/interception, that did not factor into discovering the high cost buckets in the logging case study. It was mostly relevant to generally describing how we track costs and it ends up being useful in other scenarios. For example it could be used to assess the estimated unit economics of customers subscribed into a specific product tier.

We also have the death star microservice model, so even relatively simple attribution can be helpful when you want to run a query like “for my team which owns 30 applications, tell me the monthly attributed cost grouped by resource” and that will be able to return all the associated database and cloud costs.

Your observations are correct. I wouldn’t portray it as an ideal system, just best effort. In the end we care more about the finer details being good funnels to follow up on rather than being exact. We know our real costs of resources, which is important for finance and budget. And then we have the approximate attributions from the sampling which narrow things down enough to focus diagnosis when needed.

I would have to read more into how it intercepts some of our database calls to confirm if it tries to weight for execution time where it wires into database client code, which is probably useful and could help to a degree to approximate utilization.

I think in practice it’s a bit uncommon for the heaviest user to also be a sparse user in terms of volume. But I can also admit there are quirks to how it samples and I once personally spent a couple days tracking down a surprising cost of an application I owned only to later confirm it was a data flaw in how we were doing this sampled attribution (in this case the heaviest users were un-instrumented infra processes that can’t just wire in our java cost attribution library, making it artificially look like my app was the heaviest user).

You’re right, it’s not an either or for this as we tackle both less total data and making it smaller, although I probably failed to clarify that in this first post.

At 20% of storage costs, the should makes a lot of sense to focus on. Once it becomes 1% of storage costs it’s maybe not as problematic though. The magnitude to which, “let’s compress the logs” changes how much something like “am I logging too much” matters is important. Taking it to the absurd, if logging storage were free why not retain all logs. And if logging is cheap, why invest in complicated guardrails for what qualifies as important logs.

A specific consideration for us is organizational inertia. We have a lot of teams using infrastructure in a lot of ways both intended and unintended. One thing, for better or worse, that has been emphasized for us is developer velocity. Which includes things like abstracting “do I need to log this” from most engineers. We have some guardrails to alert if you do some egregious volume.

I think we do often opt for non-invasive infra solutions first because they have much shorter delivery times and less risk of stalling on long-tail outliers. They avoid very expensive organizational costs of buy in and team-level migration. I’m not suggesting this is the best organizational model, but it also transcends one team’s influence.

That ends up circling back around to the start of the problem. If we can transparently reduce the cost burden of some heavily-used internal infrastructure within the same relative magnitude of applying a paradigm shift on the usage of the same internal infrastructure, the former wins out.

The costs I talk about in the estimate in this post are for the remaining cost of each stored file. We have S3Inventory dumping metadata of all the files in specific buckets weekly, so I had written a job that calculated the exact remaining cost of each file, accounting for lifecycle events like moving to infrequent access storage in S3 and the eventual deletion of the file. So it’s sort of the “potential energy” version of the cost of our stores files. If we take no action they will aggregate to a certain amount of money.

I reckon you may be looking at the monthly cost of storage per gigabyte which is why the number doesn’t seem to make sense. Our retention policy started off at about 2 years, so the remaining lifetime per file amortizes out to much more than 1 month.

Also worth considering that we have a custom AWS contract, so none of our actual numbers are the publicly advertised rates and probably won’t entirely math out if you try to ballpark with those numbers.

That's a fair criticism in the edit. Part 2 will cover that a bit more. I did run analysis on the types of queries users ran against the data and what parts of the timeseries were used, which informed a bit of our solution. I don't want to give away too much, but lifecycle retention adjustment ends up being relatively lower value (but still worthwhile) compared to general space savings.

(Author here). Yes I believe you are correct with regards to tracking application utilization of say EC2 and other AWS resources.

The post fails to mention this system is also tracking internal data dimensions like customer ids, such that we can also use this sampled data to estimate the cost of customers (and joining that with tiers of customers, and so forth).

I'm also not sure if that would allow us to attribute the cost of our datastore utilizations since those are not AWS-hosted versions but ones we run ourselves. The traffic interception lets us be able to say that Application A is using 75% of database cluster XYZ, and therefore that application/product group are most likely responsible for that share of how much the database costs.

The last thing I'll mention is that CloudTrail has the potential to be expensive on its own, I believe at least moreso than us storing the raw data in S3 for something like Athena to read. I don't think I'll be writing about it, but we've also done work this last year to trim down what we track in CloudTrail due to the cost of events (for example tracking everything in S3 ends up being pretty expensive).

It's a bit more complicated though, interest rates are part of that function too.

Low prices & low interest rates: great for new entries to homeownership, but would require a surplus of housing Low prices & high interest rates: great for those who buy property with cash (wealthy people and corporations) High prices & low interest rates: great for existing owners High prices & high interest rates: generally bad for everyone except those able to offload existing housing thy own

Is it really a negative feedback and not just transient though? Sure it may cool things during the fire, but it also just destroyed some carbon sinks and worsened the overall emissions effect. Then the fire ends and it no longer has any cooling effect so its not feeding back.

Stardock released their own spiritual successor already, so give that a try if you want. And as far as I'm aware the original creators are still working on their own successor.

There's a thread feature built into slack if you need to t ake conversations "aside" but in the same channel

It seems odd that age wouldn't be a dimension of diversity, unless that was shorthand for a very specific type like ethnic.

The element that you might not be aware of is Discord. Primarily a chat and voice communication app but its divided up into individual servers run by community members for free where each is like a Slack server if you are familiar with that software.

I'm not in the child age group, but I've made friends this way as a 31 year old.

It seems to me like the intent is to be talking about the meta-concept of the Brazilian soccer organization in the statements, but that's a mouthful and people contextually understand this sentiment when Team Brazil is used even though that more literally means a transient state of the organization.

Seems like a good guise for industry leaders to move operations of out the US in a climate where political sentiment is high for keeping it here.

Land isn't the resource everyone is worried about though. Potable water, food, raw materials, survivable habits, etc.

I wonder if comparing it to home improvement type work would be a good universal analogy for most people. Would you rather hire the inexperienced person to come do your plumbing or the 30 year professional. The former doesn't give you a proper estimate but will work endlessly at a lower rate on the problem. This person eventually completely messes up and a water pipe in your ceiling leaks for days. Or you could have hired the other person who would almost certainly have completed it within the estimated time frame with professional quality and never caused the pipe leak.

I don't think Scala Native would be a good fit with Apache Spark. Starting with their work in 2.0, the Spark framework already started doing things like working with sun.misc.Unsafe for direct memory control over certain operations. In general they put a lot of effort into getting as close to metal and hyper-optimizing the framework itself. Given that, trying to do the same yourself on top of that framework seems counter-intuitive in my opinion.

I am not sure if the subscriber base is technically illiterate or users that hold a given coin have a strong incentive to dismiss any criticism.

Both of those are highly representative of the demographic that want to discuss such things on reddit. Others are often even moderated against from some of what I've seen and thus discouraged from discussion. I don't know about that specific sub-reddit, but it certainly happens in some. It's an important echo chamber for exuberant hype so people have a vested interest in protecting that in the mostly unregulated market.

Certainly getting into the conspiratorial, but telco providers have this concept of zero-rating for content they don't want applied to data rates. For example, there are zero-rating projects for stuff like calls to analytics platforms. So were a telco to have a zero rate agreement in conjunction with some eavesdropping system, it would not show up.

I think there is a growing dichotomy of companies. The ecosystem I'm exposed to, 6 figures for mid-level seems normal. I'd only imagine the very early stage startups would fall outside that. This is in Boston proper and Cambridge. I do think median ranges start to fall outside the main metro area as you get to the 128 belt though.

Maybe it was lost in translation of the quote, I believe those were references to the names of actual executives: Mark (Zuckerberg), Jeff (Bezos), Jack (Dorsey)