ryanjkirk

You definitely shouldn't be storing logs on giant PVCs; they should be injested into a log aggregator.

"For Basecamp, we take a somewhat different path. We are on record about our feelings about sharding. We prefer to use hardware to scale our databases as long as we can, in order to defer the complexity that is involved in partitioning them as long as possible — with any luck, indefinitely."

— Mark Imbriaco, 37Signals

These quotes are from 2009 and 2010, and yet here most of us are in 2022, having learned the lesson the hard way over the last decade that there is no refuting this simple logic. I'll add my own truism: All else being equal, designing and maintaining simpler systems will always cost less than complex ones.

quote references:

https://signalvnoise.com/posts/2479-nuts-bolts-database-serv... http://37signals.com/svn/posts/1509-mr-moore-gets-to-punt-on...

You're half right. The "bre" e is pronounced like pen or bed, not bray.

It's not.

I'd be incredibly grateful if someone could break this down for me.

Are you from 2018? Everyone running OpenShift is using CRI-O and that footprint is not small. We made the switch in our EKS and vanilla k8s clusters in 2021. Docker has now even made their API OCI-compliant in order to not be left behind. And the point is that most people don't want a docker feature-for-feature running in prod. The attack surface is simply too large. I don't need an API server running as root on all my container hosts.

Use docker on your laptop, sure. Its time in prod is over.

Docker is heavier (and more dangerous) because of dockerd, the management and api daemon that runs as root. Actual process isolation is handled by cgroup controls which are already built into the kernel and have been for years. You can apply them to any process, not just docker ones.

However, Docker is essentially dead; the future is CRI-O or something similar which has no daemon and runs as an unprivileged user. And you still get the flexibility and process isolation, but with more security.

Up until recently, I was the steward of a large, distributed, and profitable app that used no containers. The infrastructure was managed with packages and puppet, and it worked well.

Folders work perfectly in a pure hierarchical taxonomy. Many classifications defy this rigid of a structure, however. For example:

Widget 1: it is A and B but not C, so tag it with A and B.

Widget 2: it is A and B and C, so tag it A, B, and C.

Widget 3: it is B only, so tag it B.

That is pretty simple, but you couldn't represent that in a folder system without permutation folders, meaning you now have folder sprawl, making things harder to find.

This is how servers and ec2s are for almost everyone. Billing codes, environments, teams, business units, etc. A folder taxonomy to replace ec2 tags would be a nightmare.

That would be a hard link, though they can't cross filesystem boundaries.

Our current CloudFormation pipeline template is over 5,000 lines of yaml. That's probably on the high-end, but based on your statements, it sounds like you haven't seen a corporate cloud footprint.

I worked at a company that had Target and other Fortune 500 retailers as clients, and we had very strict SLAs with financial penalties if we broke them. There was absolutely the possibility that we could have ended up our clients more than they paid us.

cloud is no panacea

That is the subtitle of this article I wrote last month - https://medium.com/@rykrk/everything-is-just-build-vs-buy-d7...

I note that the grandparent's premise for a catastrophe is that everything in a datacenter must necessarily be poorly configured. To that, I ask: how many corporate cloud footprints have they looked at?

Yes, TCO can be higher, depending where you are on the curve of capex, amortization, and staffing costs. Don't forget you still need at least Developers, DevOps, and Security. If you're inefficient at cloud, spinning up ec2s left and right, using a lot of egress, storing a lot of hot/live data, your total cost is much higher, and will easily be more than the salary of that one sysadmin, or team of systems engineers, you would pay to maintain the colo space.

You have to do a lot of things right to get that Cloud Value, as the author of this blog post has shown. You have to do a lot of things right to get value out of on-prem bare metal as well, but those things are generally well-known, standardized, have less moving parts, and people with decades of experience and knowledge of best practices. The opposite of the current cloud landscape.

TCO is not a straight line.

You're correct on staffing savings, but not on Ops savings.

I delved into this pretty thoroughly last month - https://medium.com/@rykrk/everything-is-just-build-vs-buy-d7...

PassBolt comes to mind.

I guarantee you they are not.

I'm speaking from firsthand knowledge. How embarrassing for you to think that because you know some Private Equity folks (generally the worst people at managing technology), that 100% of the rest of the world is doing (or not doing) exactly what they're doing.

You wrote paragraphs about how expensive it is to hire people to do stuff in a datacenter. What is your argument, if it is not that this is more costly than outsourcing it?

This entire debate should be recast the simple "build vs buy" concept that it already is.

Nobody is arguing that a startup needs a datacenter; that is a ridiculous straw man. I'm well aware of the costs, having managed multiple colos as well as multiple cloud footprints. I also know you can lease most hardware, if you prefer not to depreciate assets.

You're casting this as a much more complicated concept than it is. It is essentially the same old "build vs buy" argument. No, it does not make sense to manage a datacenter to run a single wordpress blog, nor does it make sense for a telco to run their infra on top of a public cloud. These are all truisms.

My point stands that if you have a seven-figure AWS bill, you already have plenty of associated staffing costs. Once you hit the break-even point between the two, you can decide whether you want to continue to pay linearly, or go down the path of increasing RoI.

Again, you don't need to build an S3 replacement; you can deploy it onto k8s. This is the same skill set whether in a public or private cloud.

Agree, I don't think anyone is buying SANs that don't need them. If you're spending 7 figures on your AWS bill, you should be looking into other options.

Don't forget Minio for S3.

I took your previous comment to be about on-prem cloud technology. It exists and does not need to be "cloned". If you were taking about staffing resources, we've pretty well established you're going to need staffing to manage both cloud and on-prem tech.

You need to document whether it's on-prem or in the cloud.

Most data center machines run at a tiny fraction of their capabilities; seeing a CPU running at 25% is definitely something that stands out.

Are you familiar with VMware?

cloning

No need to clone; all of this already exists. Take a look at CNCF.

who is going to pay for the person to patch it every week, or upgrade it to the latest version, or upgrade the SAN disk, or upgrade the hardware every 3 years, etc. etc. Let's take this a step further - who is going to hire these people? And fire them? And train them? These are all transactions costs that get hidden in the "$0.20/hr cost" of spinning up an EC2 instance.

You're implicitly arguing that NoOps is not only a functional model, but also scalable and much less costly than having a {Dev|Sec}Ops staff. The last time this was publicly debated between Adrian Cockcroft and John Allspaw, it was immediately obvious to everyone that there is no such thing as NoOps. You can have developers support their own apps in production on the cloud, but that takes away time from developers and comes with its own set of downsides (most prominently, a disinterest in what they consider mundane or boring). Once your developers are spending, in aggregate, more than 40 hours a week do ops work, you'll gain efficiency by hiring a DevOps Engineer whose entire career focus is all the things developers aren't typically great at. I've seen this in-person as an employee at medium, large, and a Fortune 50 business. I haven't yet seen a single app, not even serverless, that requires zero maintenance. I have also seen highly-distributed, performant apps run on cheap hardware with no SAN or VMware to drive up the cost and complexity. That same app, made up of mostly microservices, was much more expensive to run on on the cloud.

You might want to stop throwing around terms like TCO when you also clearly don't understand how to calculate it.

I'm starting to be of the mindset that us-east-1 is already "Too Big To Fail". I guarantee you that conversations are being had in boardrooms right now about how to reduce dependency on that region, if not AWS entirely.

and XSAN <shudder/>

Having not used setcap, would that also require the setuid bit to be set with ownership as root (like sudo)? And reside on a filesystem without the nosuid mount option?

Setting this up makes much less sense than setting up a tested vpn, such as wireguard or open, or even a persistent ssh tunnel using autossh to your home rpi.

I would never allow my prod systems to be potentially exposed by an api that runs as root. (And the documentation is incorrect on that; it should run as an unprivileged user with sudo privs to only run a wrapper script that runs firewall-cmd).

This also makes little sense in the context of configuration management, which should be enforcing a static set of iptables rules.

also https://1.1.1.1/purge-cache/ for cloudfare.

Doubtful. Seeing as they maintain their own Linux distribution, they would probably be maintaining their own BSD flavor instead.

Quite the opposite. This isn't about the userland, it's about the kernel. The linux kernel is where the high-performance cluster computing advances have been. The GNU userland is just a clone of the BSD userland, nothing special there. Moreover, FreeBSD and OpenBSD's packages are diverse, well-maintained, and current. But neither OS's userland nor any of their packaging systems have anything to do with high-performance cluster computing.

This would have been a good opportunity to educate the public about APT. Mikko's argument is essentially that this was an APT attack, which is by definition unstoppable.

A more productive way to take this essay is how to detect and close breaches once they've happened. But AV companies aren't interested in pursuing this part of the market, because it's not as simple as selling a piece of automated software.