anonacct37

What I think is funny is that circa 2008 I had a manager who used to work at Amazon who told me that "a surprising amount of Amazon artificial intelligence is artificial artificial intelligence, low paid workers".

I heard this was behind mechanical turk. Sounds like the playbook remained the same.

I really think that they bury the lede:

As a setuid program meant for elevating privileges, all code that is compiled into sudo-rs has the potential to accidentally (or intentionally) give access to system resources to people who should not have that access. The setuid context additionally puts some constraints on how code is executed, and dependencies might not have accounted for that context. We could not expect any of our dependencies to take into account such a context either.

This is the real problem. I've come to the conclusion that setuid programs basically shouldn't be using most libraries. The setuid environment is just fundamentally different. A normal library can have a debug output file who's location is controlled by an environment variable without that being a security risk. But the instant that program becomes setuid, that's an arbitrary file overwrite security bug. Most libraries aren't built with that in mind. They shouldn't have to be. Setuid is poorly designed.

I used it and even took a class taught by salt stack employees.

I really was not a fan. I think one of the core problems I ran into was that the tool attempted to layer on rpc behavior to what was fundamentally a broadcast.

So you'd apply something to everything with a web role. The tool would attempt to make it appear as if you executed apply on n nodes and return after n responses. But since this was all just a broadcast what really happened is the tool dropped a message on the bus and then guessed how many responses it would get. If it guessed wrong the cli invocation would just hang.

I'm not sure if I follow but it does occur to me that being thrifty and changing from a aerospace engineering company to a financial engineering company can be two different things that might end up categorized as "accountants are in control".

This is a very sharp tool and I find it's really rare to need it.

I do alot of profiling and performance optimization. Especially with go. Allocation and gc is often a bottleneck.

Usually when that happens you look for ways to avoid allocation such as reusing an object or pre allocating one large chunk or slice to amortize the cost of smaller objects. The easiest way to reuse a resource is something like creating one instance at the start of a loop and clearing it out between iterations.

The compiler and gc can be smart enough to do alot of this work on their own, but they don't always see (a common example is that when you pass a byte slice to a io.Reader go has to heap allocate it because it doesn't know if a reader implementation will store a reference and if it's stack allocated that's bad.

If you can't have a clean "process requests in a loop and reuse this value" lifecycle, it's common to use explicit pools.

I've never really had to do more than that. But one observation people make is that alot of allocations are request scoped and it's easier to bulk clean them up. Except that requires nothing else stores pointers to them and go doesn't help you enforce that.

Also this implementation in particular might not actually work because there are alignment restrictions on values.

Going on 5+ years using a single giant org file. It's the only system I've ever been able to stick with for more than a couple days.

I think of it as my labbook.

I love and hate this. On a visceral level I don't like dealing with issues via malicious compliance.

OTOH I absolutely agree this is a good call if your goal is for more users to be able to run more software on your platform, even if printing is broken.

DNS without CGO works perfectly

It does not. I know this because it impacts my daily work and the work of others. Honestly if you could make my day and go figure out exactly what's going wrong with the pure go DNS implementation it would make my life alot simpler and I wouldn't have to maintain shell scripts that update etc/hosts to hard code in ipv4 addresses for the APIs I access with terraform.

https://github.com/hashicorp/terraform-provider-google/issue...

I do like this about go. And on Linux it arguably makes sense (I say arguable because DNS without CGO is still a common cause of issues and incompatibility).

But Linux has, to the best of my understanding said "yes, we are ok with users using syscalls". Linux doesn't think that glibc is the only project allowed to interface with the kernel.

But for other platforms like OpenBSD and windows they are quite simply relying on implementation details that the vendors consider to be a private and unsupported interface.

This whole thing is also separate from "is making libc the only caller of the syscalls instruction" a good and meaningful security improvement.

There's a lot of subtleties. It's really easy to accidentally load test the wrong part of your web application due to differences in compression, cache hit ratios, http settings, etc.

Shameless self promotion but I wrote up a bunch of these issues in a post describing all the mistakes I have made so you can learn from them: https://shane.ai/posts/load-testing-tips/

My initial requirement was to send requests with unique parameters. To the best of my knowledge, no tool could do this.

wrk does this with lua. https://github.com/wg/wrk/blob/master/src/wrk.lua

Also even things like the venerable jmeter supported pulling parameters from a csv file.

Android has a BLE scanner app that can passively read info. I don't know enough about the protocol to know if that is the same thing you are referring to.

No memory leaks (unless you use shared memory). No weird connection pooling and caching issues or delays in picking up a config change. Every request is a fresh start, enforced by the OS. Buggy code (like image processing can only crash the current request).

The actual damage is that it's pretty common (my last team has this happen) for a team to setup a cert, verify it works, and then when they deploy the cert it works some of the time or "works on my machine" and so the failures seem really random and by definition hard to reproduce because you have to restart chrome to reproduce.

Probably the tl;Dr is that validating against a persistent cache like Firefox is fine. Validating against an ephemeral cache with chrome is likely to cause a lot of breaking.

cryptography. Things that really need to be constant time and not be "optimized" by the compiler.

See the .s files in: https://cs.opensource.google/go/go/+/refs/tags/go1.21.5:src/...

I occasionally see it in compression as well.

text/template is probably ok... For some version of text. ditto with jinja and most templating languages. The cardinal sin of DevOps is using text macros to produce structured data. It only exists because unfortunately there is no other lowest common denominator for every config file syntax.

I also ran into spooky negative dentry issues. On a fleet of seemingly identical physical machines one machine would constantly check for files that didn't exist (iirc it was somehow part of nss and a quick Google search seems to confirm https://lwn.net/Articles/894098/ ).

I'm okayish at systems level debugging but I never figured that one out. It caused the kernel to use alot of memory. Arguable whether or not it impacted performance since it was a cache.

It's the old "you are not the customer" but applied to management. I really really want to be told what the high level goals are and come up with a solution. So of course when I started managing people I worked really hard to communicate those high level goals and let them decide the how. They told me no thanks please just let me know what tickets to work on.

People are different and motivated by different things and that's ok.

The dictator who listens really seems to be what the majority prefer because they don't actually want to be accountable for those decisions they just want a seat at the table.

This is an underrated capability (patching up requests) that many devops folks often take advantage of in platforms like haproxy, nginx, etc.

I once worked at a company where someone pushed a release that added timestamp based cache busters to every js/css file. We used a similar approach to just drop that query parameter at the edge, which allowed us to get cache hits and stop serving every single asset from object storage.

Like laws? Or rules? Sure. They are methods of control.

Sorry, but that's not true.

I wish it was. But running anything that bridges google3 and GCP is a nightmare. Outside of acquisitions and OSSish stuff like chrome, it's really rare to see GCP used. Oddly enough their corp eng team does quite a bit with making GCP accessible to the rest of the company.

Source: former Google SRE who actually worked on one of those teams.

I think this is a great idea. In the old days before fuse was so widely used I saw this same idea used in the jvm since it supports custom protocol handlers that you can use for vfs operations.

It's basically still the only game in town for a high performance shared posix filesystem with multiple writers and builtin support in basically all operating systems.

As an application developer, I probably wouldn't choose to design a system that needed it.

However there are lots of good reasons why a company in 2023 might decide to use this NFS based product: https://cloud.google.com/filestore?hl=en

I have a simple guess. We are in a post zirp (zero interest rate policy) world. I've seen very large companies embracing degrowth.

Quite possibly Microsoft told it's subsidiary GitHub that they needed to spend less money and this is how they save resources while impacting the users they care about the least.

I've noticed that this week. After spending a little time just reading on how to correctly write posix compliant code that was also guaranteed to do what I want, it's hard to come to any other conclusion than "posix is a last ditch attempt to slap a bandaid on Unix fragmentation by attempting to retcon some lowest common denominator behavior from a couple popular unixish operating systems and calling that a spec".

It's not what I would actually call designed. The closest analogy I can think of for non-c programmers is that posix is like if we decided that people should only make websites using javascript that was mutually interpretable by IE6 and Netscape navigator and we occasionally made updates every decade or two.

There's really nothing particularly noble or good or correct or elegant about all the API calls with implementation defined semantics. At best it's a necessary evil for compatibility. You can admire the cleverness required to get a single simple c codebase that works correctly on multiple operating systems and future operating systems that conform to posix in creative ways, but only in the way I admire those old school zines that are simultaneously a PDF and a jpeg and a shell script. Clever and ingenious but not actually good engineering design.

I don't think it's literally impossible to avoid the same mistakes as our predecessors. But I do think that the default position that "oh we modern innovative companies won't end up like those stodgy old companies" is a recipe for repeating history. As they say in AA: the first step to solving a problem is admitting you have it.

Because yes by default you will absolutely repeat history unless you acknowledge that those old timey crazy people were fundamentally no different than you.

Can't answer your question directly but I've touched a few of those here and there.

Clickhouse was deployed to replace a home grown distributed storage system that at one point in time a long time ago was much cheaper and faster than the results the team was getting with BQ.

We evaluated clickhouse and druid for a few data stores for doing interactive queries on a fairly high throughput clickstream data pipeline.

Clickhouse won in terms of performance. We did some chaos testing on it in the form of simulating node outages and network partitions and we were happy with the results. My only complaint is that there are some other database options which don't require me to run a database and that's nice if I can get away with it.

One of the things you might try is dumping your data into parquet files on gcs. There are quite a few databases you can query that with and get good results depending on your indexing and partitioning needs. It's tough to get lower operational burden than "stick it in cloud storage and sometimes spin up some stateless compute to query it".

I think duckdb is super cool but for me at the moment it's a solution that I'm still in search of a problem for.

I wish Sentry would stop free riding on Free Software movement by pretending their product is part of it.

Their product isn't Free Soft anymore. They feel that they can increase revenue with a more restrictive license, a business move that doesn't require this cathedral and bazaar white washing.

That makes sense. I hope my first comment came across as I intended, which wasn't as criticism of the product but a suggestion about how to talk about the value of the product.

I do like the approach of static code analysis.

I found it a little funny that their big "win" for the nilness checker was some code logging nil panics thousands of time a day. Literally an example where their checker wasn't needed because it was being logged at runtime.

It's a good idea but they need some examples where their product beats running "grep panic".