gdcohen

I agree. If you can reliably recover from failures by migrating instead of restoring from checkpoints, then the need to checkpoint frequently becomes far less important and reduces the overhead of taking checkpoints. Similarly, a lot of users are experimenting with asynchronous checkpoints to reduce the blocking penalty, but there are big tradeoffs there with DRAM usage. So being able to take checkpoints far less frequently gives you more flexibility on the type of checkpointing you use as well.

Take a look at Exploiting a Natural Network Effect for Scalable, Fine-grained Clock Synchronization - https://www.usenix.org/conference/nsdi18/presentation/geng (commercial version by the same authors - clockwork.io).

What's amazing to me is that without reading (watching?) too much into it, and me not being a seasoned South Park viewer, this could easily pass as a real episode.

See also: https://news.airbnb.com/an-update-about-our-community-in-new...

Essentially NY City is making short term rentals (rentals < 30 days) illegal for most renters. The only exception is a room without a lock, for up to 2 people, in a residence that has shared living space and is occupied by the owner.

Unfortunately it's more restrictive than just owner-occupied units. It's essentially an unlocked room (for up to 2 people) in someone's house/apt that includes shared living spaces.

For example, there are many multi-family houses where a family lives upstairs but can't rent out the downstairs apt.

Worth a read if you've ever used Airbnb for a short term stay in NYC (or if you're an Airbnb host). The City of NY is about to start enforcing a ban on short term rentals on Airbnb, VRBO, etc.

Does anyone have a simple explanation of how it structures the log data?

Gavin from Zebrium here. We've done some interesting things with GPT-3 - we use it to produce plain language summaries of log events that describe the root cause of a problem. See - https://www.zebrium.com/blog/using-gpt-3-with-zebrium-for-pl... and https://www.zebrium.com/blog/real-world-examples-of-gpt-3-pl...

This is really cool. You've done a tremendous job in making this very usable (I love your yellow sticky note instructions).

Gavin from Zebrium here. We've found that if only you somehow knew what you were monitoring for in logs, they can be a great source of detecting (and then describing) the long tail of unknown/unknowns (failure modes with unknown symptoms and causes). Our approach is to be able to find these patterns in near real-time using ML. This blog by our CTO explains the tech with some good examples: https://www.zebrium.com/blog/is-autonomous-monitoring-the-an....

Gavin from Zebrium here. Completely concur with #1. We are big advocates of writing good logs and not having to worry about structured vs unstructured (and even if you structure your logs, you'll still probably have to deal with unstructured logs in third party components).

Our approach to deal with logs is to use ML to structure them after the fact (and we can deal with changing log structures). You can read about it in a couple of our blogs like: https://www.zebrium.com/blog/using-ml-to-auto-learn-changing... and https://www.zebrium.com/blog/please-dont-make-me-structure-l....

We love Litmus because it gives our users a way to quickly and deterministically inject failures into their app so they can test the effectiveness of our solution.

Gavin from Zebrium here. Thanks for doing such a great job with this. I played with an earlier version of this and it's come a long way. I love how you've made it really easy to setup and run experiments with the Litmus Chaos Engine (https://litmuschaos.io/).

Thanks, this is very helpful.

But how do you detect when something is broken or know when it is good enough to deploy everywhere?

Gavin from Zebrium here. Again, we understand this concern. All data is encrypted in-flight and at rest and we have a lot of security controls in place (see our website). We also have the option of a dedicated VPC assigned to a single customer. Beyond this, we have a unique capability: since our machine learning structures and types everything, we have a feature that lets us hash (or delete) any sensitive information in logs (and we can record match to find other places this info occurs that you might not even be aware of). Not dodging the fact that we're SaaS, just pointing out what we do.

Sounds like someone is going to have to build a network of depth chargers :-)

Also that the article talked about the colon as having a longer pause than a semicolon (different to how it is used today).

I grew up with an "Eiderdown" on my bed, and until now, mistakenly thought of it as a synonym for a quilt filled with feathers!

https://www.thesaurus.com/browse/eiderdown

There is a trend towards not testing at all! Instead builds are deployed straight to production or canary (mini-subset of production) and then very carefully and closely monitored. If a problem is uncovered, a rollback is performed. If canary is done well, then the problem can be caught before it has widespread impact.

Sorry if this is pointing out the obvious, but the title is not a typo! We discuss an approach of using machine learning to automatically post-structure logs, rather than having to manually pre-structure them.

Interesting how far spreadsheets have come since 1983 and, just as interesting, how much presentation tools and techniques have changed in the same time period!

Excellent insights and a well written article. One suggestion - it would be helpful if you explained how someone can tell if they have each of the SEO features enabled.

Totally unrelated, but this reminded me of a headline I read in the '70's proclaiming that the Beatles had made a new album using the band name "Klaatu". https://en.wikipedia.org/wiki/Klaatu_(band)

I wonder if learning to program at age 40 vs age <10 would be similar?

Thanks so much. Another clever idea!

Thanks again. Very helpful. It sounds like you understand this space. If you’d be up for a chat I’d love hear from you (gavin@zebrium.com).

Thanks for the suggestion. Very clever! Do you think the word forensics works as part of a description of the product?

And this: https://nextshark.com/banana-leaves-asia-plastic-packaging/

Implementing, and practicing, a framework like this is an incredible way to build a culture that makes every person feel like his/her opinion can count and will be listened to.