throwaway290232

CrowdStrike followed up with Delta on the offer for onsite support and was told that the onsite resources were not needed

Even though Microsoft's software had not caused the CrowdStrike incident, Microsoft immediately jumped in and offered to assist Delta at no charge following the July 19 outage," the letter said. "Each day that followed from July 19 through July 23, Microsoft employees repeated their offers to help Delta. Each time, Delta turned down Microsoft's offers to help, even though Microsoft would not have charged Delta for this assistance

Two vendors offer you free on-site help, during the biggest outage in history, and you turn it down?

This isn't the CIA, it's Delta Airlines. Bring those overpriced security monkeys in, make em sign an NDA, and have them sit there and watch you type.

To not do that... even if they were useless... I can't find a justification ....... other than being able to later claim it wasn't Delta's fault it was down so long, and nobody having evidence to the contrary.

Realistically speaking AWS and GCP are not that different and anyone who has 5 years experience in one can quickly pick up the other. But teams will often go with the person who has experience in the stack they use, just because it's an easier way to weed through the stack of 300 resumes. Hiring is broken mostly because of reliance on resumes.

If anyone has a good resource that teaches makefile progressively

https://www.gnu.org/software/make/manual/make.html

Makefiles fill that great need for a high-level 'scripting DSL', where you have a lot of different programs (or scripts), with a loose set of dependencies or order of operation, and you want a very simple way to call them, with some very simple logic determining the order, arguments to pass, parallelization, etc. Their ubiquity on all platforms makes it even easier to use them.

I much prefer Make to alternatives like Just or Taskfile. Besides the fact that more people know Make, Make actually has incredibly useful functionality that alternatives remove 'for simplicity', but then later you realize you want that functionality and go back to Make. Sometimes old tricks are the best tricks.

US Law often requires power generators to operate separately from distributors. Even when it's the same company, they have to operate as two separate businesses, and it's illegal for them to even mention certain information to each other as it's considered anti-competitive. The purpose is to allow the market to provide competition. This is not specific to NYC.

An Introductory Guide to Electricity Markets regulated by the Federal Energy Regulatory Commission: https://www.ferc.gov/introductory-guide-electricity-markets-...

Deregulation and competition: https://en.wikipedia.org/wiki/Electricity_sector_of_the_Unit...

I disagree with a lot of this.

First of all, feedback is not a great way to build relationships. If it's positive feedback, and it's not perceived by them to be demeaning/dehumanizing/etc, most people will like it, regardless of relationship. But if it's negative, nobody likes to receive negative feedback, but especially not from someone they don't already trust. First build a relationship, then give negative feedback, when appropriate (which is a very qualified when).

Second, asking if you can give feedback is like asking if you can ask a question. The other person is going to feel rude if they say no, so they will instead feel pressured to say yes, and now you have a position of power over them, they feel bad, and you haven't even given the feedback yet. If the feedback is positive, just give it, don't ask them if it's okay. Nobody will legitimately complain about praise. If the feedback is negative, keep your friggin' trap shut, unless you feel the feedback is absolutely necessary, from you, right now. If it doesn't need to come from you, or it's not absolutely necessary in this moment, then wait until you have a normal conversation with them and allow it to enter conversation naturally and gracefully.

Third, don't offer feedback if the person has not already requested feedback. You can offer to them that they can give you feedback, but if they don't provide the same reciprocal offer to you, don't push it. Feedback isn't a right, it's a privilege.

Fourth, you absolutely should be afraid of giving feedback to people who might use that feedback against you. In fact, just sharing your thoughts about things in general, out in the open, can unintentionally hurt people's feelings and turn them against you. I have personally found out on several different occasions that I somehow hurt someone's feelings unintentionally, just because of my opinions on some topic (engineering-related, business-related, etc) that wasn't personally directed at anyone. Your tone, messaging, content and context can and will have unintended consequences, so think before you speak. If you're unsure of the outcome, either don't speak, or accept the consequences.

Fifth, this is general knowledge, but always complain up, never down. On the other hand, praise everyone: your boss, your direct reports, your peers. Praise costs nothing but does make people like you (and others).

If the only thing people hear you say is positive things, you will probably be one of the best-liked people at work. Giving negative feedback generally won't help you, and many won't be open to it, so choose wisely when you give it.

I think that is a complex and worthy thing to debate as it addresses many different points; agency, regret, information, intent, consent.

I think the law deals with similar concerns by not basing a judgement on how someone feels, but rather on actions and agreements. Did they lie, cheat, steal, deceive, coerce, manipulate, intimidate, threaten, abuse, etc in context of some agreement or lack thereof. But that's just the law; obviously there is an entire world of morality/ethics that exists outside the law with different considerations.

I'm just saying, he hit on a minor, and he doesn't appear to be a moron, so screw this guy.

I think what he's suggesting is that "at first, she was fine with it, but then somebody convinced her not to be fine with it" or some such thing. Which may have some debatable merit, depending on the circumstances... except that she was a minor. Any kind of defense or explanation goes out the window when you're that kind of creep.

Anybody have any tips on how to get less angry at work? I could stand to improve a lot there.

Agile is a generic umbrella term that involves a vast array of complex, subtle knowledge and skills. It's like saying "Engineer", or "Chef". Each has a shared skillset, to be sure. But each category's members can't just work the same way at all jobs, and there's no book on how to be an Engineer or Chef everywhere.

The Agile Manifesto is a failure at trying to make Agile happen because it can't tell you how to make it happen, because it varies wildly. No manager can read a book on how to "Agile-ify" their org, they have to apply their brains and figure out how their specific version of "Agile" will work. But the skill-set required to do this is not a Managerial skill, it is a lower-level-worker skill. But it's also a very advanced lower-level-worker skill.

And that's why things like "Agile", "DevOps", etc will fail. People at the higher end have no clue how to make it happen, and people at the lower end who have an idea how to make it happen don't have the power to make the organizational changes to do so. You need a way for the lower-end people to tell the higher-end people what to do, and have the higher-end people listen to them, and make the changes happen. This is very hard in a traditional organizational hierarchy, because higher-end people have big egos and bigger concerns over things like politics.

but Philly has a lot more going for it

You mean Jobs? Philly definitely has more. But I found Balto to have more going for it, despite being tiny in comparison. Culture-wise, Baltimore has loads of everything (except the "we're pretending we're NYC-lite" that DC and Philly push). There's loads of green spaces, festivals, multicultural cuisines/neighborhoods, industries. Baltimore even exceeded Philly in things like hackerspaces/makerspaces/tool libraries. And Balt has [some] free public transit! And though the history isn't pushed much, there's a ton of it all around. I'd also argue the Inner Harbor exceeds any Philly tourist district in terms of interesting ways to waste a saturday for a wide range of people. Then you've got the stadiums right near the harbor, the light rail to whisk you up into the lush suburbs, and more quirkiness and charm than practically any city in the US.

But I might be biased.

Looks like it's getting hugged to death: https://web.archive.org/web/20210716193902/https://artbma.or...

I love the BMA. It's a free museum with a great restaurant, cool exhibits, a nice park across the street. Nearby is The Book Thing, a free book store. Also another (non-free) book & record store, Normal's. And another, Urban Reads. And another bookstore/coffee shop, Bird in Hand. And a farmer's market. And a vegan restaurant. And the quirkiest diner ever, Papermoon. And a small rock venue. And a worker-owned co-op coffeeshop that President Obama visited. All in a four block radius.

Dang, I miss Baltimore.

By "Elsevier" you mean "research journal publishers". There's more than one, and many have the same policies, depending on the publication. Elsevier doesn't even publish the biggest journals.

What's weird to me is that nobody has to submit their standard or paper to ISO or a journal, they could just put them on a blog post somewhere. But people still complain that these organizations, who people voluntarily submitted their papers to, are actually charging people for what they said they would charge for, or take the rights they said they would take. It's not like somebody was tricked. This is like a manufacturer giving their jeans to Wal-Mart to sell and complaining that Wal-Mart is taking a cut. Yeah..... they're selling it. That's how that works. The manufacturer could always sell them out of the back of a truck... but they don't want to.

This reminds me that a lot of [other] papers are bullshit. Papers are basically long-form blog posts. One group of people did a thing and here are their results. From those results they often come up with generalized conclusions. A lot of the time, people just take those conclusions as truisms! But do the conclusions extrapolate to other groups, scenarios? Will bias color the readers' takeaways (like authority fallacy)? How well does this work when implemented elsewhere over 10 years? Does anyone who has implemented this paper take it seriously, or without a million caveats?

I cringe when I see a team blindly implement the design in a paper. Design to your needs, not somebody else's! Papers are great places to take ideas from, but you should never treat them like gospel, or copy them outright. It's the same trap as designing your solution around your tools rather than vice versa. When someone brags about implementing the design in a paper, I expect it to work poorly (at least until they figure out all the stuff that wasn't in the paper).

Papers are experiments, whereas Best Practices are the experiments that were reproduced by many people in many places over a long period of time. If what you're building matters (not R&D), implement the best practice first, not the experiment.

There's a particular class of bug that affects all datacenters in a globally distributed service. Almost invariably it comes down to knock-on effects due to multiple things happening "the wrong way" in tandem. Those individual failures typically happen because somebody said "well yeah we should have a better way to do this, but the other components have redundancy, so shrug". If a single domino isn't tested well and has all potential failure modes well documented and tested for, it can still take down the entire chain of dominos.

A couple people. Working groups that decide internet standards. Industries and private companies that control how the internet works. Web browsers that determine how the world wide web works. Organizations made up of people from each of these categories. And of course, every single user, developer, vendor, etc that hungrily adopts and subsequently becomes dependent on everything they put out.

There is no God, but there are working groups of angels discussing Existence Standards.

Those agents should not be falling back to unencrypted anyway! The whole ecosystem just needs to get onboard with implicit TLS and deprecate the old agents. It's not acceptable to make the whole ecosystem dependent on two completely different security mechanisms. Every client/server in the world would have to support both indefinitely, which would be a totally unnecessary cost and complexity burden.

That's why I recommend physical prints. Hopefully you would put them somewhere where you actually look at them from time to time. Or put them in a safe deposit box (assuming you trust those).

The problem of "what to keep" isn't even digital-specific. In my youth I probably bought two dozen disposable cameras, had the pictures printed, even paid for the CD-ROMs when they became available. Where are those pictures now? Who knows! That's why I like Marie's message: it's not about throwing away junk, but keeping what you love close to you.

And sometimes children feel too much responsibility for their parents’ happiness. I often hear estranged adult children request better boundaries from their parents as a condition of reconciliation. As Andrew Solomon wrote in Far From the Tree, “There is no contradiction between loving someone and feeling burdened by that person. Indeed, love tends to magnify the burden.”

Holy shit, this. I thought I was crazy for feeling emotionally blackmailed by my parents. They effectively told me that if I wasn't a constant part of their lives, they would be miserable - after things like blaming an episode of depression on my poor academic performance. At the same time being callous and judgemental towards me, and each other. I'm just surprised that they expected me to stick around?

Yeah, I totally agree it's really out of left field compared to what users are comfortable with. Like another commenter said, clients would probably laugh you out of the room for proposing it. (though that's half my point! why are we only accepting these half-baked custom solutions on janky platforms? fear of criticism? is it really saving anyone any time or money compared to the "weird solution"?)

But I'm not convinced on the latency/compute/storage comparison with Kafka or other solutions. I think a POC would need to be built and perf tested, and then tweaked for higher performance and lower cost, like most software. Considering the volume of traffic that mail software is designed for, I can't see how even a large provider like Stripe would have difficulty scaling a mail system to match Kafka. It's not like mail software is written in Java or something ;-)

I recently did a comparison of cloud storage options:

- If you want "personal storage" (no programmatic access) the options are generally around $5 per TB/month - but that's a lot of drag-and-dropping and praying the connection doesn't die during transfer.

- If you want object storage (think S3) the cheapest is $5 per TB/month, the average is $10 per TB/month, and the "high end" is $20 per TB/month, with extra costs for bandwidth ranging from $10 to $120 per TB/month.

Honestly, just store less crap. Marie Condo your digital life. Does it not spark joy? Have you not looked at it in the last 6 months? Does it not serve a useful purpose, such as tax records? Ditch it.

Even if you want to keep some pictures/video, either print a copy in original-quality, or compress/downscale it. I took a 3 minute video on my phone and it's 425MB. And it was still grainy! I used to download two-hour movies that were 725MB and looked like a DVD! errrrrrr... I mean, I heard about a guy that did that.

There are a billion lego pieces out there in the e-mail ecosystem. We can combine them any way we want, if the alternative is a totally custom solution anyway (webhooks, custom API endpoints, cron jobs, queues, etc). There are so many options; where to begin!

First, you don't have to use the rest of the internet's e-mail system. Stripe can run their own mail servers that deliver straight to clients on non-standard ports using implicit TLS, ensuring security and no middle-men. This also ensures delivery is as timely as possible (sub-second typically, as mail software has to be fast to handle its volume).

Let's say you want to poll (ex. "/events"). The client uses IMAP to poll the Stripe server with a particular username/password. Check a folder, read a message, delete it on connection close. There are of course ready-made solutions for this, but you can also write simple IMAP clients really easily using libraries.

Let's say you want pushes (ex. webhooks). The client sets up the alternative to the webhook-server they'd have to set up anyway: an SMTP server. Use a custom domain, one that has nothing to do with the customer's main business, so nobody ever gets confused. Configure it to only accept mail from a "secret mail sender" (aka webhook secret). Part of the "SMTP webhook URI" would be what mailbox to deliver the webhooks to. The client then configures an MDA on their mail server to immediately deliver new messages to some business logic code. If the MDA or business-logic code has a bug, the messages will stay in the client's mailbox until they are "delivered" successfully. If the client's SMTP server is down, Stripe keeps retrying for at least 3 days, more if Stripe wants.

Stripe could actually implement both by keeping messages in an IMAP folder on Stripe's servers, and deleting the messages once the SMTP server confirms delivery to the client. Of course all messages already have unique IDs so removing dupes is easy.

You could implement all of this in a week, write almost no code, and still handle all the weird edge cases. Virtually all of that time is just reading manual pages and editing config files. The end result is a battle-hardened fault-tolerant event-driven standards-based distributed message processing system. The maintenance cost will be "apt-get update && apt-get upgrade -y", and anyone who can configure Postfix and Maildrop can fix it.

DANE is a kludge that should be put to bed, not promoted as a solution to a problem which shouldn't exist.

STARTTLS exists for two reasons (https://www.fastmail.com/help/technical/ssltlsstarttls.html):

1. Wanting to accept mail insecurely.

2. Not wanting to use two different TCP port numbers to send and transfer mail.

To solve these problems they created STARTTLS. But obviously, STARTTLS isn't actually secure (even though that was the point of supporting TLS). So to make it secure, it's suggested to use DANE - a standard built on a different procotol, requiring a feature that is controversial, potentially dangerous, and not widely implemented. So you can use a kludge (STARTTLS) with a kludge (DANE) to send and transfer mail securely. But should you?

Since 2018, RFC8314 says that e-mail submission should use implicit TLS, not STARTTLS (https://datatracker.ietf.org/doc/html/rfc8314#section-3). Therefore the use of STARTTLS, and the use of DANE to make it secure, are deprecated. So while you shouldn't use DANE for anything seriously, you really shouldn't use it for SMTP.

Yeah, they sort of have different purposes. VPS providers are the easiest way to just start using some VMs for static workloads. But Cloud providers give you not only a ton more control, but also allow you to save money by only getting charged for the resources you use. The tradeoffs being that the latter is way more complicated, and the "hyper-scale" Cloud providers are extremely overpriced. I'm hoping more small providers pop up and make the big girls more competitively priced.

I sort of agree, but somebody already has to manage the "infrastructure" of their web apps, dns. They never mind adding more of their own home-grown services. If they used Kinesis instead that's another piece of infra to maintain. But you would never hear them say "what about Postfix instead". Regardless of infra, if it's new, they want to use it, even if something older and more boring would work better.

If I ever heard a dev at work say "No I won't use that new tech, it's too untested/I'll have to spend more time figuring out how to make it work well", I would shit my pants. Whereas if it's old tech, "it's not modern/I'll have to spend more time figuring out how to make it work well". It's practically software ageism...

There are also a bunch of OpenStack Public Cloud providers in Europe and around the world: https://www.openstack.org/marketplace/public-clouds/ Some are not listed there so you may have to do some research (I know there's one in Chicago)

These "Cloud" systems provide a number of advantages over VPS providers. They have a standard programmable interface and command-line tools, they provide many services you may or may not get with a given VPS provider (programmable networks, load balancers, object and block storage, secrets management, configurable user/role access controls, etc). They also of course allow you to scale your resources programmatically and may provide pay-as-you-go pricing. And if it's something you're into, you can typically find a Terraform provider that works with it.

Slightly more expensive than a simple VPS, but the automation, failure recovery, and security gains can be significant.

I always laugh when people end up with designs like this. They could have just used SMTP! It's designed to reliably deliver messages to distributed queues using a loosely-coupled interface while still being extensible. It scales to massive amounts of traffic. It's highly failure-resistant and will retry operations in various scenarios. And it's bi-directional. But it's not "cool" technology or "web-based" so developers won't consider it.

Watch me get downvoted like crazy by all Nodejs developers. Even though they could accomplish exactly what they want with much less code and far less complex systems to maintain.

The premise of this post is wacky. They're trying to argue for how web applications should provide consistency to operations on remote systems. You know what that's called? Distributed Computing. I don't know if you know this, but Distributed Computing Is Hard. You can't solve it with a new interface or polling really fast.

Webhooks are perfectly fine for what they're intended, which is inconsistent push-based notifications to loosely coupled web apps. If you require "consistency", you supplement with polling and queues and other junk. If you require real consistency, you must use a distributed consensus algorithm.

It is completely embarrassing how many engineers we have and still apply manually from laptops. Changes are slow and error-prone, we don't even have them hooked up to CI/CD. I think it still works because we have so many damn engineers and we don't actually need to change infrastructure multiple times a day.

That said, Terraform breaks so often that if we did it all automated, we'd have a million more Git commits from trying to fix broken apply's.