HN user
Zac Bentley
he/him
github.com/zbentley
blog.zacbentley.com
That’s pretty silly whataboutism. The two aren’t mutually exclusive, don’t trade off, have both risks and benefits, and can be used or prescribed responsibly or irresponsibly. What are you getting at?
Firmly agree. A lot of the lessons you mentioned have been discussed previously on HN threads for hardware startups. This is one of my favorites:
https://www.simonberens.com/p/lessons-learned-shipping-500-u...
I guess, but like ... long context/context rot/contextual memory retrieval are still very unsolved problems. If an AI's getting input from multiple family members for years, I have a lot of trouble believing that current LLM technology will ever be able to synthesize ancient information into current conversations.
I think that's a structural defect in AI, not one of those "will be improved by the next batch of models/hardware" things. The distant-memory systems that do exist are fundamentally modeled on topic-based/feature-based retrieval. Assistant-type family AIs will probably work OK with that ("what foods do I hate" is probably reliably retrievable), but interpersonal relationships seem to be very different: rather than retrieving based on feature/fact proximity, a conversation in a long relationship has a very gestalt-ish behavior in that it's informed by lots and lots of distant memories at once. That sounds more like actual training, rather than context/memory/post-training, to me, and current AIs are not set up for that.
Row ID preloading is an extremely effective technique—and not just for SQLite. I’ve also used it to great effect on massive Aurora MySQL or Postgres clusters since I could send the SELECT to a replica, and the whole point of deletions was that index memory pressure from the row filtering was putting tons of CPU and buffer cache pressure on the db.
If you’re in a situation where partition pruning or other strategies for getting useless data out of the hot path don’t make sense, this is a killer strategy.
The White House electrician and kennel manager, Traphes Bryant, trained Pushinka with peanuts to climb up a ladder to Caroline Kennedy's playhouse and slide down the other side.
"White House electrician and kennel manager" is a hell of a job title.
Doing generics in a new Rust project’s types by “dyn”-ing everything you can and then falling back to monomorphized “true” generics after benchmarking is (or will be, when dyn compatibility for more layouts is added) a pretty okay way to go, I’ve found. Not without struggle and syntactic noise, but superior to the alternative.
Thanks for high quality reply.
Likewise; thanks for engaging constructively as well.
I traditionally have treated this as a state in a database.
Which is a super appropriate tool many (most) times! I've been using "MQ" in this context refers to the conceptual capability to externalize/distribute/persist work with a push/pop API. That can be provided by a database, a UNIX pipe, or a more traditional message broker--the reasons-to-adopt and costs/benefits are largely the same.
I want to try the email idea sometime.
Email (and the print spool) are some of the oldest message-queue primitives on UNIX systems, I think? I'm not sure if POSIX/shmem MQs predate them or not, but they're all quite venerable proofs that the pattern has its uses. I'm sure you can (maybe others already have?) use SMTP and the mailq to manage application-internal queue communication. I'm not sure how low-overhead or performant it'll be given that it's highly optimized for one/very few consumers and delivery attempts >1 being attempted on a pretty large time scale compared to application traffic, but it'd be an interesting experiment to be sure!
The answer is usually a performance problem for which this is a bandaid.
I agree that if that's the problem you're trying to solve, MQs are a poor stop-gap and not a general-purpose solution.
But many (most?) MQs aren't deployed for that reason; rather, they're used to either a) defer work whose latency characteristics are incompatible with the producer's runtime (e.g. sending email which might take minutes/hours to be accepted upstream can't be done synchronously from an HTTP request handler), b) handle work which doesn't need to be done transactionally and which has a high probability of needing to connect to resources whose uptime you don't control (queues make retries easy), or c) work that needs to be batched or otherwise completed on a schedule or dimensionality that's not available in the producer.
A while loop blocked on a socket read with a producer with a write does not generate any extra network “chatter”.
True, and some MQs support that pattern, but it's rare for the same reason that most network server applications' core loop doesn't wrap a blocking read. While-blocking-read plays poorly with timeouts, restarts, and multiplexing/wait-any. This isn't unique to MQs.
Anytime you are using callbacks you are choosing not to have a thread with imperative logic.
I mean, it's pretty easy in most languages/runtimes to turn a callback into an imperative wait. When it comes to network servers specifically, I think exposing the lowest-runtime-overhead model as the primary API (selectors and callbacks) and letting users add control flow primitives on top of that is preferable, but reasonable minds can differ here. This also isn't unique to MQs.
if you want to send data between two systems and you have producers and consumers at different rates, you don’t have to do anything special. The UNIX kernel is designed to solve this problem.
For ephemeral/can-be-lost local traffic, you're partly right (pipes/fifos are a simple and widely available tool that can help with this, though you have to layer some protocol-ish logic on top to make them work reliably between consumers that expect particular structures and don't speak ASCII/newlines alone). For durable local traffic, or remote traffic, UNIX doesn't have a prebuilt primitive. TCP tx/rx window sizes aren't surfaced such that buffers and backpressure can be reliably interpreted by userland to make delivery decisions.
the pipe is generally unidirectional
So are most message queues.
If the producer needs to know when the consumer has finished work, that's not a queue; that's an RPC.
Bad take. Most of the different behaviors you criticize about message queue systems arise from them needing to a) distribute work b) durably, c) over the network.
Most message queues want to be able to distribute work to multiple parallel consumers, either for performance, redundancy, re-deployment of the consumers, and so on. Like, sure, you can do that with named pipes, but they're not durable; their state can be lost in the event of a crash or reboot.
Other than like ... in-process, thread-to-thread message queues, almost all message queues are used because they provide some form of durability (either by persisting messages to disk, or just by virtue of running on a separate server/process than much more crash-prone producers/consumers).
imperatively, rather than one end being imperative and the other being a declarative graph of callbacks (all those “reactive” libraries).
MQ client libraries are heavily reactive because they need to work with MQs that are network services. If every consumer that ever wanted to fetch a message had to issue a network RPC poll/timeout cycle, and they were all doing that constantly, that'd be both a lot of traffic for the MQ to handle and a lot of network chatter. Some MQs do use (or support) the RPC poll model, but especially among the more performant ones, most of them are push-based so clients can just sit in an epoll/select statement waiting for the message queue to send them traffic.
That said, you're totally right that plenty of MQ client frameworks/libraries are way over complicated and want to own your whole program's entry point and design. Callbacks are probably a necessary outgrowth of the underlying network behavior, but tortured whole-program-event-loop-ownership architecture is not.
Even using a term like “back pressure” is a tell to me that someone is confused snd doing something architecturally wrong.
I don't think your other points support this; you said yourself that pipes' bounded nature is good, so what's the problem here?
We call that “Larry Wall nomenclature”.
s/some/most/
That's not meant to be snide, just true, I think.
rewrites are bad engineering, in most cases.
I think this situation is a referendum on whether that is still true—or, if it is true, whether the proportions on “most” have changed significantly due to AI.
Off topic, but watching the output of this thing renews my disappointment that so many programming languages/frameworks fail to provide signal sender information to signal handlers. It’s there! Crusty ABI and historical reasons are, as far as I can tell, the only reason why languages like Python (and many others) don’t surface sigaction to handlers without unnecessary acrobatics.
Sigh. We shouldn’t have modeled signal handling in a cross-platform, lowest-common-denominator way in HLLs. Signals have way less in common across OSes than files/sockets and the like.
Anyway, rant over. Not much to be done about it now.
I’m not sure that suffices. If a site has a very “good” (at keeping people glued to the screen) content ranking algorithm, they can still make money, albeit less, serving non-targeted ads. Longer engagement time by viewers = more ad impressions, targeted or not.
Fair, though I think a large part of why this has always been more difficult than it should be is the fundamental design of TCP and IP. The former encodes notions of “roaming client” and “durable server” in ways incompatible with devices that periodically want to assume either role, and the latter (at least IPv4) encourages network segmentation in such a way that reachability and discoverability are harder than people might think.
The centralized powers that be definitely deserve a share of the blame to be sure, but some of it is architectural.
My friend, you have reinvented something called a “purchase”.
They really, really don’t. I’m sympathetic to the goal of moving manufacturing stateside, but fixing regulatory hurdles needs to happen first.
Just because a foreign glove manufacturer has to comply with the same safety/quality standards for their product only scratches the surface. Domestic companies also have to operate compliant with regulations regarding hiring, real estate, safety, finance, and many more.
To be clear, a lot of those regulations (most of them!) achieve good things and should be preserved! Working/environmental/low-corruption conditions in the US (though they could be better) are often superior to those same conditions in the places we import from precisely because of those regulations. But not all regulations (even those with good goals) are effective, others are not worth the burden, and all of them in aggregate do directly contribute to domestic manufacturing loss. It’s a hard problem that others (e.g. Pahlka) have written about much more wisely than I can.
Sure, but “shell scripts” as an (admittedly imprecise) term usually imply separate executable files, not shell libraries/functions that are sourced.
Presumably you can call chdir(2) from the C code?
Or did you mean change the directory of the calling shell (in which case, executable shell scripts written in Bash and friends can’t do that either).
That’s a good clarification, thanks for writing it. I think you and the uncle post by bluefirebrand are both right: the social expectations are different enough now that there really might not be a practical alternative for most people to the corporate surveillance pocket computer—and you’re not wrong that the shift in values and priorities that indicates is pretty scary and grim.
Eval differs very slightly from, say, an interpreter with a very non-syntactic AST, due as you say to homoiconicity. Neither read, print, nor loop differ in any Lisp-specific or qualitative way. And the differences in eval’s behavior don’t change anything significant about the REPL user’s experience (other than that they’re writing Lisp). So what do you mean?
Specifically, cached state and behaviors still need to be reconciled with new inputs/overrides during “eval”, and no Lisp has an easy answer to those (neither, as far as I know, does any other language). Avoidance of closure state via late binding and a convention of reliance on very simple data structures help Lisp’s REPL/hot-patch story, and that is indeed nice, but it’s far from “guaranteed ease/safety of REPL patching” or a totally different paradigm.
Slightly off topic, but it is worth being familiar with AWSCLI’s built in jmespath JSON transformation system. That saves on the need for jq in a lot of cases. Given that awscli bundles that, I generally forgive it for defaulting to JSON (especially given how deeply nested and interlinked many AWS API results are), but reasonable minds can differ. AWSCLI’s schizophrenic use (or not) of pagers for tiny results, on the other hand, is less defensible.
I wrote GGP as a Lisp user and enjoyer—not pretending to deep experience, but definitely am well acquainted. And neither the REPL nor live reloading are important differentiators. Neither is the ability to attach a REPL to a running program: plenty of languages have that built in (Erlang, Ruby), and many others support it through popular third party tooling (e.g. Pyrasite for Python, and—ironically—the class executor for the Clojure REPL for arbitrary non-Clojure JVM programs). Many Lisps’ REPL tooling is very nice, and the language lends itself well to REPL-oriented development. But those aren’t “uniquely Lisp” features as you and many others claim.
There are some truly powerful and unique things about Lisps, but I wish articles like this would stop including REPLs and hot-reloading. The former have been table stakes for interpreted languages (and some compiled ones!) for years, and the latter is neither unique nor particularly widely used (hot reloads have to tangle with state and patching, so resetting the world for ease of reasoning is considered a best practice for a reason).
I interpret that the opposite way: if a huge spam campaign can be run by a guy in his bedroom, there’s no way that larger spam operators can be effectively killed by legal action. They’ll just employ different guys in different bedrooms. The same thing is true with phone phishing scams—they’re not individually hard to eradicate, but the combination of lucrativeness and rapid-rebootability means that legal crackdowns in the past have not been effective at scaring the rest out of business.
Can you send mail from something that doesn't have a DNS entry?
I hope not. Just like SSL, I think requiring a registrar+DNS server to vouch for a durable identity is an important barrier to abuse (and an important intervention point for violation reports).
I don’t think they can. Spam, like speeding on highways and drug sales, is such an asymmetric enforcement area that I have very limited confidence that legal enforcement would make a significant dent in the volume. It’s far too technically easy to anonymously, repeatedly break anti-spam laws. This is an area where consortium enforcement (like the big inbox providers pushing solutions like DKIM2) is probably the most effective.
Don’t get me wrong, there are tons of areas where countries’ legal systems have no excuse for not enforcing the law more stringently (e.g. flagrant corruption in multiple regulatory bodies’ failure to enforce investment/wire fraud). But spam is part of the other category—technically difficult enough to crack down on that legal action is a waste of time. There are ways to change that, but they’re all either more centralization-prone, worse for privacy/liberty, or extremely expensive.
OpenBSD’s equivalent is the hw.uuid sysctl: https://man.openbsd.org/sysctl.2#HW_UUID~2
Other BSDs don’t have that, but have equivalent PCI tree identifiers. “hostid”, too, is found on many systems but is much less unique as it’s often a function of local network address.
why don't authors skip the middleman themselves and just put the .pdf/.epub on their website directly in exchange for donations?
Some do!