HN user

acidmath

-15 karma

section two thirty audit@mail2tor dot com ; where "two thirty" are digits rather than spelled out.

Posts0
Comments24
View on HN
No posts found.
CINC Is Not Chef 1 year ago

I haven't encountered any Chef code in almost ten years. For the stragglers not yet primarily on Kubernetes and Terraform, I see Ansible and some extra Terraform. Maybe I see some Salt here and there.

These are just my anecdotes, for sure, but (also anecdotal) I rarely hear of other "ops" type people using Chef, and most of the ones I know never got more than just their feet wet with Chef (the SaltStack beta was out early enough to avoid Chef).

Spam is far too obviously not speech is the thing. They will continue to use the moniker "Free Speech absolutist", and ignore you. But keep campaigning as the one lone voice with that, I'll still continue cheering you on while the rest of the world ignores your funny viewpoint that isn't believable for you to actually even have.

I'm a 3/4 black partially Jewish gay man, so you can imagine where I stand on "Free Speech absolutism". While I am for the most part on your "side", I want to stick to arguments that are persuasive.

All four were granted anonymity because they were not authorized to publicly discuss the private chats.

Anyone with access to NSA plus various subcontractors' toolsets can "unmask" these people in like five minutes. Musk may not be "tech genius" some of the media makes him out to be, but he knows enough about how the internet and computers work (or has advisors who do) to figure that out.

Where three different disks (or storage underneath Ceph OSDs) each in separate disk silos fail?

In some earlier Ceph clusters I was responsible for I set replication to four or even five.

There was a point where I wrote an internal memo to business leaders explaining that had I not set that replication to four or five we would have been unable to meet business goals (we would have lost the contracts keeping us afloat during critical stages), but now at scale the real dollar cost of that redundancy was showing. In that memo I explained what networking we needed vs what we had, what people we needed vs what we had, and so on. I eventually got the networking gear the company needed in place and was able to hire the kind of people we needed.

For Ceph clusters I am chiefly responsible for today we have Ceph pools in each availability zone doing erasure coding (four or more parity bits) and a small service makes sure objects are copied between the distanced datacenters (availability zones).

While I do get a bit of a boner (of the Hank Hill on propane kinda vibe) on how the erasure coding is distirbuted across cabinet rows, to be fair it is a bit of an optimization.

_

to have lower durability (say replicas are located within the same networking pod)

A network segment should never ever go down. I know in some places this is "optimistic". I can say for us it's not "optimistic" because we don't allow Ethernet protocol to "hop" anywhere. Layer-2 broadcast domains either stop at top-of-rack and the rest is layer-3 or we are using Infiniband. Stable Ethernet networks are a thing, but not using Ethernet beyond where Ethernet belongs avoids so much risk.

100% network uptime is achievable.

In some places more critical nodes are connected to three Infiniband switches with two uninterrupted power supplies dedicated to each Infiniband switch. Maybe that's "excessive", but within the last year we had to contend with a switch failure. Then a week later a datacenter provider (one that will tell you they never ever EVER lose power) for one of our availability zones was unable to provide power for over two full days.

I don't think we're ever going to have downtime or lose data. In other places this would be "optimistic", but for this deployment I have access to the resources I need to achieve the thing. The only thing that has me concerned about downtime is the current WW3 we are already in heating up.

I suggest to review the comment you replied to.

Free speech absolutism shouldn't extend to: advertising that is unsolicited and disseminated in a bad faith style.

Sorry about the state of your reading comprehension (or lack of it). Unfortunate.

object recovery is quite manual if you lose any

When I read this I think "but you should never lose an object". Do you mean like the underlying data chunks Ceph stores? Can you elaborate on this part? I know some of the teams I work with do things in unorthodox ways and we tend to operate on different assumptions than others.

so pg’s can be spread across racks/datacenters.

Some Ceph pools come to mind (this was a while ago, I'm sure they're still running though) where the erasure coding was done across cabinet rows and each cabinet row was on its own power distribution. I don't know how the power worked but I was told rather forwardly that some specific Ceph pools' failure domains aligned with the datacenter's failure domains.

We already have a loki setup

Nice. We have logs go into S3 and then anyone who prefers a particular tool is welcome to load whatever sets of logs from S3 within the resource limits set for whatever K8s namespace they work with. Originally keeping logs append-only in S3 was for compliance but we wanted to limit team members by RAM quota rather than tools in line with the "people over tools over process" DevOps maxim.

Spam can be whatever who so happens to be the site moderator, or maybe some very tiny yet obnoxiously vocal minority, wants it to be.

Beyond that practical matter, the least bad definition I have for spam (and this is just off the top of my head right now) is: advertising that is unsolicited and disseminated in a bad faith style.

Any of us who are being honest with ourselves, maybe with exception to the acutely socially challenged, know spam when we see it.

Many of the addicts are long past the mission of "trying to feel good" but are trying to stave off horrible withdrawal symptoms. Ibogaine clinics can help a lot of people. Ibogaine clinics don't work for everybody, and won't work for those who don't want to change their lifestyle, but can help bring back something resembling an okay life for a lot of people.

Just off the top of my head:

Run Ceph on https://rook.io/ ; don't bother with Cephadm. Running Rook provides very helpful guard rails. Put the logs for Ceph Rook into Elasticsearch+Kibana on its own small (three or four node) dedicated Ceph Rook cluster. Which Kubernetes distro this runs on matters more than anything.

Recently we are looking at using https://www.parseable.com/ instead of Elasticsearch+Kibana. And we had somewhat recently started moving things from Elaticsearch+Kibana to OpenSearch+OpenSearchDashboards due to the license change.

The requirement outlined by Ceph documentation to dedicate layer-1 paths (can be same switches, but must be different ports) to Ceph replication is not about "performance" but about normal functionality.

If you have any pointed questions feel free to email "section two thirty audit@mail2tor dot com" (where "two thirty" are the three digits rather than spelled out).

In the servers we have only 16gb to 64gb of NVDIMM, depending on density of NVDIMM and how many slots are populated with NVDIMM. Whatever raw NVDIMM is, usable is half because we mirror the contents for physical redundancy (if we lose a transaction it is fatal to our business). NVMe is amazing, but not everything should be NVMe, like petabyte scale object storage for example does not need to be on all NVMe (which is super pricey).

In newer DDR5 servers where we can't get NVDIMM, the alternative battery backed RAM options leave us with even less to work with.

Where we have counts of HDDs or SATA/SAS SSDs in the hundreds, we still want the performance improvements provided by WAL (or functional equivalent such as ZiL/SLOG) on NVDIMM and some layer-2 (where layer-1 is RAM) caching with NVMe.

Ceph OSDs want a dedicated WAL device. Some places use OpenCAS to make "hybrid" devices out of HDDs by pairing them with SSDs where the SSDs can accelerate reads for that HDD and the Ceph OSD goes on a logical OpenCAS device. OpenCAS is really great, but the devices acting as "caching layer" often end up underutilized.

By placing "big" Ceph OSDs on ZVOLs, we don't have individual disk slices for WAL (or equivalent) or individual disks for layer-2 read caching, but a consolidated layer in the form of ZFS Intent Log on "Separate Log" (NVDIMM) and another consolidated layer in the ZFS disk pool's L2ARC (layer-2 adaptive readback cache).

The ZVOLs are striped across multiple relatively large RAIDz3 arrays. Yeah, it's "less efficient" in some ways, but the tradeoff is worth it for us.

  https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/#devices
  https://open-cas.com/

"The people", who originally were (and are today) the lawful citizens of the several states are party to the first eight amendments in the Bill of Rights. Foreigners enjoy 1st Amendment protections at the discretion of the federal government, at best.

A libertarian will argue that fraud is an act that is not the same as free speech and hence is not covered by free speech. I don't agree with the libertarian viewpoints overall, however I will acknowledge they have a point about the distinction between those two things.

For books depicting sex acts, banning such books from public elementary and middle school libraries is reasonable. Is it not?

Does there exist any example of an administration (federal or state) successfully prohibiting the private sale of an LGBTQ book?

Before Bluestore, we ran Ceph on ZFS with the ZFS Intent Log on NVDIMM (basically non-volatile RAM backed by a battery). The performance was extremely good. Today, we run Bluestore on ZVOLs on the same setup and if the zpool is a "hybrid" pool we put the Ceph OSD databases on an all-NVMe zpool. Ceph WAL wants a disk slice for each OSD, so we don't do Ceph WAL and consolidate incoming writes on the ZiL/SLOG on NVDIMM.

Pretty much everyone I personally know, this in part to coming from a family of relative affluence in highly liberal areas, has religiously filtered their water to keep additives (such as fluoride) from going into their bodies. This same crowd has savvy towards dental hygiene, they're all into oil pulling, and brushing with Fennel oil and all that. Eliminating yeast (which is fungus, not bacteria) from the mouth prevents the biofilm that creates the necessary conditions for dental cavities.

Even the poorest Americans can afford coconut oil and Fennel oil.

There's no reason to drink that shit (fluoride), there's no reason to brush with it either.