trelliscoded

Wasn't this the plot of Terminator 3?

I thought this was totally fascinating. Having a compact static object that can be used to cook without any other fuel inputs seems like a convenient thing to have around if you're in the latitude range where it would work. I've used solar cookers before, but they're simply not portable in the way this one is.

Yes. Firefox has its own password manager and profile system. Once I copied the chrome settings to firefox, I closed chrome and rarely open it these days.

I’ve had to budget a line of business app that runs on both z/OS and AWS. Reliability for both hit an asymptote somewhere around 2019. The opex price crossover happened somewhere around the same time, depending on the customer.

Nearly every installation site and their internal support director agrees that the z/OS installations are far less expensive to support, simply because the minimum level of experience for a z/OS engineer is about a magnitude more than a cloud engineer.

One client of mine has a couple repositories for non-mission critical things like their fork of an open source project, decommissioned microservices, a SVG generator for their web front-end, etc.

They also take this approach of "whatever tool works," but their coding test is "here's some symptoms of the SVG generator misbehaving, figure out what happened and fix it," which requires digging into the commit history, issues, actually looking at the SVG output, etc.

Once you've figured out how the system architecture works, and the most likely component to be causing the problem, you have to convert part of the code to use a newer, undocumented API exposed by a RPC server that speaks a serialization format that no LLM has ever seen before. Doing this is actually way faster and accurate using an AI, if you know how to centaur with it and make sure the output is tested to be correct.

This is a much more representative test of how someone's going to handle doing actual work knocking issues out.

That’s interesting, I was looking at the ICE ERO statistics and not only do they show up as not having a published data source, only metadata, the corresponding dashboards at ice.gov/statistics seem to not be working. I don’t know what the state of either of those was before I looked yesterday, so I don’t know if this is a recent change or not.

I was extremely surprised to find out that based on ICE’s annual reports, which are still up, Biden’s administration deported far more people than Trump’s did, especially in 2024 due to rule changes which roughly doubled the number of normal deportations. I was further surprised to find out that there were roughly 3.5 times more deportations under Biden than under the first Trump administration.

I suspect that what’s happening is that leadership under Biden did the sensible thing and very quietly snuck up on priority illegal aliens, whereas with Trump, all the media attention is causing similar people to circle the wagons and proactively take steps to avoid enforcement actions.

The equal sign is a required character for anything base64 encoded, which includes some things you’d expect to be in a config file, like ssh public keys and x509 certs.

The “CC” in the TI part numbers stands for “ChipCon”, who TI acquired for these wireless products. The CC1100, one of their earliest parts, has a documented but unsupported debug mode which dumps raw quadrature samples out some of the I/O pins. Later parts do not mention such a feature in their documentation, but it’s not proven it was removed.

TI has had wifi+bluetooth microcontrollers for almost a decade now, and you get a cortex m4.

That’s interesting, it’s like the difference between code used at runtime (protein coding DNA) and initialization code (lncRNAs). Both have to be there for the program to work, but the initialization code is only used at startup to look at the environment and set up flags and data structures for the rest of the code. There’s probably signaling pathways that interact with the lncRNA genes which are part of cell differentiation.

If your staging doesn’t do capacity checks in excess of what production sees, yes.

OpenVMS and all its predecessors are still useful and, I think, ideal for real-time control applications and clustered high-reliability setups. The OS is structured much differently than pretty much everything else, so you can treat an OpenVMS machine as a completely dedicated embedded system where direct control of the hardware is possible without needing to deal with writing a complex kernel driver. It's a bit like if you crossed the low-level control guarantees of DOS with all the modern stuff you want like a network stack, Java, web servers, etc.

Despite having all the conveniences of a modern OS, the overall complexity is way lower than a typical Linux distribution. It was all designed by a single firm under tight resource constraints, so the amount of shared common code is way higher, the number of parts is way lower, and it's much more feasible to keep the whole design in your head all at once.

The clustering technology is also state-of-the-art, and has been for decades, even if you treat the cluster members as real-time control machines. Everything you need to build what most people think requires a whole k8s cluster and several helm charts is already built in to the OS. This includes stuff that most people don't think of until later, or try to reinvent on top of the DB. Distributed transactions, propagating the OS security model into the application, industrial-strength audit logs, autodiscovery, it's all built in and works out of the box. Pretty much everyone from a k8s or borg background that I've had to onboard to legacy apps running on OpenVMS clusters is blown away by how simple everything is. Even though there's fine-grained resource quotas and support for bin packing workloads throughout the cluster, it's rarely needed because of how efficient everything is since applications all use a common runtime designed around async paradigms instead of six zillion third party libraries and multiple C runtime libraries per container that can't share memory pages.

Because it has fewer moving parts, almost never needs to be patched, and they kind of got everything right the first time, it's much more deterministic than any other non-embedded operating system you can practically use on modern hardware today. It would absolutely be my first choice, today, if I had to build some kind of highly reliable control system cluster that had to stay up beyond my lifetime. To this day, I see absolutely ancient installs on a regular basis that simply will not die because they keep swapping out cluster members when they fail. Itaniums and Alpha Galaxies with RPS are extremely reliable by themselves too, and their biggest reliability threat is availability of replacement PSUs and memory. The OS was designed in a pre-Internet era where putting a control host out in the boonies somewhere to control some satellite ground station, feeder dam for a hydroelectric system, or military radar had to work, had to stay up, and had to be as reliable as the hardware of the time would allow. There really isn't anything else like it that will give you those kinds of guarantees in a disconnected, IoT-type edge environment, yet also run on modern gear.

One of the major downsides is that learning how to work with the OS to do modern things isn't well documented anywhere (e.g. how to use logical names to emulate Linux namespaces/container technology). DECNet in particular is a real sticking point for everyone, even though Linux has supported it since forever. I think I might be the only person I know with the enterprise networking, Linux, and OpenVMS background to do all the network plumbing required to integrate DECNet with a modern hybrid cloud or hyperconverged stack. It's not hard, but you have to make sure you check off a bunch of stuff from a checklist to make sure everything works. It's getting harder to find Cisco gear that natively supports it too, but using VMs to NFV some vyos routers or whatever to create brouters works just as well as it does for managing all the usual IP north-south and east-west flows in a big production system.

Despite its age, DECNet is a big feature of OpenVMS because of how it supports the clustering technology, MOP, and things like LAT. LAT is a standardized remote serial protocol that lets you do stuff like add a serial port somewhere that the OpenVMS host can see on the network, and it shows up as a native comm device in the operating system. You can open it, toggle the control lines, all that stuff just like a normal serial port connected to the host. This is unbelievably convenient for distributed control systems because everything was designed by a single firm instead of e.g. trying to integrate digi's remote serial drivers into your product or trying to reinvent autodiscovery of RFC2217 devices. There's a lot more serial ports around than you might think, like the QXDM port on your phone, BMCs, deaf TTYs, the console port on your RPis, etc. Having an infinitely expandable host capable of doing real-time control of hundreds of serial ports all at once isn't something I'd consider any other operating system for.

You kind of have had to work at a shop where they built a lot of stuff on top of OpenVMS within the Internet era to get that knowledge, because I'm not aware of any books or other reading material which explains how modern things map onto the corresponding OpenVMS concepts. I should probably write a book or something, because it would be a shame if the underlying concepts faded from our collective memory and the open source community ended up reinventing all this stuff in a much less efficient way spread over a dozen open source projects that don't have a central architectural authority beating everyone up about centralizing common code.

Cutler and his original team really did deliver a complete OS that got everything right the first time, and I can't think of any modern requirements that OpenVMS can't handle from an architectural perspective. I guess the filesystem isn't capable of doing really big work like a huge z/OS data set or a petabyte sized ZFS pool, but then you just use NFS with EFS or a netapp if you have a fortune 500 IT budget. It sounds wrong, but OpenVMS I/O still sets speed records compared to everything else because of the first-class async I/O support in the OS, and how simple the file I/O layer in the OS is. You can explain the entire write() to disk driver sequence of events to someone in 5 minutes for VMS, but I can't imagine trying to explain just the linux dcache in less than an hour. Just walking through all the filter drivers and everything else touching an IRP in a typical Windows machine is like a semester-long course.

I use a Helium Mobile eSIM, which I pay with crypto. I set it up in something like 10 minutes when I became eligible for an account. No problems so far.

I've also had eSIMs from FirstNet and AT&T, no problems there either. Running dual-sim hasn't been a problem either on any of my Galaxy or iPhones.

Disabling script signing on dev machines and requiring signatures on production scripts sounds like perfectly reasonable behavior to me. I know a lot of people are scared of pki but it’s way easier than people think. Signing things is a one liner, I keep certs on a portable HSM and it’s really low friction.

It’s not an admin vulnerability, so there’s no hatchway. The real issue here is blindly passing user-provided input to a batch script, possibly from the Internet, and if you’re doing that then you’ve got much bigger problems. If you’re doing it using an account with any kind of privileges, you’re kinda asking to get broken into.

I can’t seem to find cost information for any of Pragmatic Semi’s parts, even their high volume RFID tags which are supposedly shipping.

The 6502 seems to be a technology demonstrator that has no commercial application, as there’s no corresponding low cost power supply for their parts. Because polymer semiconductors are only about 1% efficient, it takes way more power than an energy harvester could supply.

The LLM based explanations were the key thing about this app for me. It’s hard for me to fit foreign vocabularies into context for long-term recall without etymologies and comparisons to common roots in Latin, and I’ve already had several ah-ha moments due to those explanations. Thanks.

SEEKING WORK | REMOTE or HYBRID (Cupertino, Singapore) | DevOps Engineer | Rate: SEA Digital Nomad rates for ongoing customers

  Location: San Jose, CA
  Remote: Yes.  I travel around the Singapore area for an ongoing project, but I'm based in Silicon Valley and can do on-site or hybrid work in either region.

  Willing to relocate: For the right price.

  Technologies: {AWS, GCP, Azure} plumber. IaC {Windows Server, Linux, NFV} administration. Bash, zsh, C, modern C++, JavaScript/node, Linux, Darwin, Win32/64, Python, Ruby, a bunch of other scripting and frontend languages.  Ansible, Terraform, CloudFormation, Jenkins.  SQL and Mongo, onprem, hybrid, and cloud.
  Résumé/CV: email if needed.
  Email: mackeycam25@gmail.com.

I’ve had this happen when I called in a support request for some iOS issue I had. Their interface has all the devices on your Apple ID, and they can enable screen sharing on any of them after you accept it via a notification. I have to admit, it must be a lot better for the support experience as opposed to trying to verbally describe what’s going on.

The factors that contribute the most are

1) Bluetooth devices are usually battery powered and have their microwave RF section and DSPs implemented in power-sensitive silicon. Layer 1 and 2 performance is not good.

2) The Bluetooth specification is very complex and requires IP licensing for more desirable features such as low-latency audio, so inexpensive Bluetooth devices often badly test unpopular features and forgo licensing for nicer CODECs.

3) Bluetooth is based on frequency-hopping (FHSS), and the wavelength difference between the lowest and highest permitted frequencies is about 4mm. In some cases, this means that an intermediate RF-opaque object which is around that size near either transceiver may cause portions of the Bluetooth spectrum to be blocked, resulting in some percentage of the transmitted frames to be below the receiver's ability to get the bits out of the frame. Those dropped frames are not retransmitted, so the audio presented to the user sounds like swiss cheese, with evenly interleaved gaps of a few dozen milliseconds each.

4) Many legacy Bluetooth devices for Windows have driver packages that install an entire parallel Bluetooth stack (CSR, Toshiba...) next to the Windows one, because the in-box stack was pretty bad for a long time and didn't support many newer features. The situation is much better now, but machines which were upgraded from such a state, or ones which had whatever drivers were in the device's box installed are often using Bluetooth stacks which are totally ancient.

5) Embedded Bluetooth firmware is often not tested against many hardware failure cases, or omits more sophisticated portions of the stack due to CPU or memory constraints, and relies on cost-sensitive NVRAM technologies which have high bit error rates compared to e.g. a NVMe device with a 1 GHz controller to do lots of fancy error correction. Sometimes, the device just doesn't have reliable areas of NVRAM to store certain persistent state, and has no way to tell you that the checksums aren't working out (if they're even implemented), so pairing just fails for what seems like mysterious reasons. Keep in mind that many, many Bluetooth chips are driven by extremely low-end CPUs (I've seen 8051s), so there just isn't space for handling errors. The spec is complicated enough that many procedures have lots of failure modes, and in many of those cases the firmware just gives up and drops the frame on the floor.

A product I worked on got real popular, so scaling up the support department became a real pain since the devs were running all kinds of different distros and so we had CI and package support for Arch, Debian, Ubuntu, CentOS... except that all of those come in a bunch of different variants and customers expected the product to run on them. At one point more than 90% of the tickets were dealing with distribution-specific weirdness. There was an epic internal wiki page that must have weighted in at a couple megabytes of raw text that was an encyclopedia of run-time linker caveats for hundreds of distributions, complete with troubleshooting flowcharts and links to our internal test farm to get one-click access to a VM reproducing the scenario described. I still wish I had that, it would have been so handy for some of the Rust and MIPS porting I'm doing now.

We couldn't containerize it (it's a client for a security monitor), we couldn't ship it as a VM, so eventually we all agreed to ditch everything except RHEL and everyone else would be unsupported (we grandfathered all the existing ones in, and it took a year and a half before everything worked.).

Part of the reason CoPilot is so good is that they reportedly ingested a huge chunk of GitHub. Now that everyone's onto this scam where they're not getting paid to help train some model, there's legal challenges to the legality of this approach, and a move back towards self-hosted infrastructure. If there's an injunction and a ruling on that legality, it may be that GPT-4 will be the only model that will ever be trained on a codebase as large as GitHub.

I wish this was upvoted more. This is the correct way to troubleshoot the bug, full stop. You can get symbolic stack traces with full arguments and source code on Windows in about 5-10 minutes for any Chrome crash by following these instructions. I always have a last change exception handler that fires up a WinDBG script on Windows for our chromium-based test runners, which reduces troubleshooting time to just a couple minutes in order to find the symbol in their bug database. Playing blackbox what-if games like the Gusto team is a waste of time and doesn't contribute any situation-specific knowledge to bugs.chromium.org.

The canals are just the edges of the entire path, the part in between is enormous compared to them. Because they flooded a forest, there’s a lot going on in the intermediate lake.

SEEKING WORK | REMOTE or HYBRID (Cupertino, Singapore) | DevOps Engineer | Rate: SEA Digital Nomad rates for ongoing customers

  Location: San Jose, CA
  Remote: Yes.  I travel around the Singapore area for an ongoing project, but I'm based in Silicon Valley and can do on-site or hybrid work in either region.

  Willing to relocate: For the right price.

  Technologies: {AWS, GCP, Azure} plumber. IaaC {Windows Server, Linux, NFV} administration. Bash, zsh, C, modern C++, JavaScript/node, Linux, Darwin, Win32/64, Python, Ruby, a bunch of other scripting and frontend languages.  Ansible, Terraform, CloudFormation, Jenkins.  SQL and Mongo, onprem, hybrid, and cloud.
  Résumé/CV: email if needed.
  Email: mackeycam25@gmail.com.

SEEKING WORK | REMOTE or HYBRID (Cupertino, Singapore) | DevOps Engineer | Rate: SEA Digital Nomad rates for ongoing customers Location: San Jose, CA Remote: Yes. I travel around the Singapore area for an ongoing project, but I'm based in Silicon Valley and can do on-site or hybrid work in either region. Willing to relocate: For the right price. Technologies: {AWS, GCP, Azure} plumber. IaaC {Windows Server, Linux, NFV} administration. Bash, zsh, C, modern C++, JavaScript/node, Linux, Darwin, Win32/64, Python, Ruby, a bunch of other scripting and frontend languages. Ansible, Terraform, CloudFormation, Jenkins. SQL and Mongo, onprem, hybrid, and cloud. Résumé/CV: email if needed. Email: mackeycam25@gmail.com.

It's an emulated s390x on x64, but yes, you can spin up emulated z/OS images in their own LPARs just fine in IBM cloud. You can get a real Linux LPAR on mainframe hardware, too. For a while, it was even part of the free tier.

IBM's provided a free developer course, including a free z/OS account, for at least a decade now: https://www.ibm.com/z/resources/zxplore

Going through the challenges is worth it just to see how the mainframe way of doing things works, even if you never plan to use a mainframe in your career. There's a lot of core concepts and cultural touchpoints that help you build more reliable systems everywhere else.

Big tech products are decoupled from the physical world and ultimately not critical for society to function, so the workloads are highly variable. Mainframes are a good fit when you have workloads that don't vary by several orders of magnitude on a daily basis, and you need an ecosystem that treats reliability like a religion.

If you haven't used z/OS or worked in a z/OS shop, you will never understand how fundamentally different mainframe environments are. Sure, JCL is weird and ISPF doesn't make it clear how much raw power and how many OS features are available under the hood, but once you've worked on a real production application maintained by a good team, none of the syntax or UX stuff matters anymore. The abstract representation of the entire application and everything that supports it starts to live in your head and you start to realize how incredibly backwards a lot of modern cloud scale stuff is.

The dirty little secret of a lot of z/OS shops is that they are already using modern cloud environments on their existing hardware. All the productivity gains of Github Copilot + Visual Studio Code, Python, Rails, are all available on the same high speed, high reliability, high capacity frame where you already keep all your data sets. You can even spin up a Linux LPAR if you don't want to run on top of z/OS directly. The node.js people can whip out Next prototypes and slurp in 500 megs of questionable NPM modules on top of either OS, too.

While "modern" environments focus on bolting APIs together using web technologies, a typical mainframe environment simply hosts everything in the same place and uses the data sets on disk or the database as the interface between applications. This allows you to do all kinds of stuff that's fundamentally impossible in a cloud environment. You can instantly give an application a 100% consistent view of a checkpoint of a 200 terabyte data set, while it's being used by 3 other applications pounding out a million IOPS, and remaining failover ready to the other hosts in your metro sysplex. Many sites are sized well enough that when the LTO robot backs it up while all this other stuff is going on, there's not even a blip in your transaction latency numbers.

There is no cloud vendor, and probably will never be a cloud vendor capable of providing support for all of these features, even if they had them in the first place. I've worked on a service that's white box resold by all three major cloud vendors, and believe me when I tell you that there is no way you can get everyone involved in every component that might be contributing to, say, a weird block storage problem on a conference call. You just can't. You'll get escalated to someone who might have a chance at maybe doing some initial isolation to narrow down the scope of what's causing it, but that's ridiculous if you're a bank or a nationwide grocery store. Your EBS call is returning a HTTP 302 today? There's no documentation for that, not at the level you're going to want when it's breaking production at a factory that employs thousands of people. None of those industries use any of these fad technologies for anything important for exactly that reason.

Mainframes, on the other hand, have these kinds of support considerations built into nearly every component of the environment. The error message culture alone eliminates all the frantic googling that most non-mainframe engineers are used to doing when they get error messages. The OS and applications are expected to emit error and warning codes, which translate to actual English text that tell you what happened, and what to do about it. If necessary, IBM support can remotely JTAG any component in the frame after it's been automatically taken out of service.

Your SAN vendor can't point fingers at your server vendor, because they're the same vendor. You can't even call the network vendor for your ToR switch, because it doesn't exist; the frame has multiple internal PCIe, Infiniband, and IBM coupling adapter backbones, and you don't need a switch because you can simply add 96 WAN-capable 10 gig ports to each frame. There's no separate cluster interconnect transceiver vendor for the optics, that's IBM too.

If any of this detailed documentation refers to some operating system data structure you've never heard of, well, those are all documented in the MVS data area manuals, volumes 1-4, with convenient "eyecatcher" human-readable four character strings so you can quickly identify them in a hexdump of the operating system's memory. Nearly every single thing the OS outputs is also documented in the system message manuals, volumes 1-10. Don't even need to bookmark it, everything's immediately available via the master documentation index at ibm.com/docs/zos.

There is absolutely no comparable equivalent to any of these supportability-oriented resources in any other supported software ecosystem that exists today. Just go look at the manual for DFSORT, which is the z/OS equivalent to the Unix sort utility. Seriously, go look at it:

https://www.ibm.com/docs/en/SSLTBW_2.5.0/pdf/icem100_v2r5.pd...

There's over a hundred pages explaining what to do for any of the runtime messages the utility can output. There's no handwavy "undefined behavior" like you get with C and Unix. IBM has thought through every possible thing that can go wrong with a sort utility, with paragraphs of supplemental information and advice for many of them. If you use the online facilities to look it up on the mainframe itself, it will conveniently ask you if you want to print out the relevant documentation on the printer closest to you (yes, it knows where it is.) Most of the 3rd party application software is the same way; this level of documentation and programmer attention to detail is simply part of the culture.

People say IBM's mainframe support is really good, and in my experience it usually has been, especially when you're hard down and they get their best people on the phone immediately, but no one ever mentions that the best thing about their support is that they do all this behind the scenes work so you can support yourself using the documentation, without having to call them. Meanwhile, things which are extremely basic, fundamental, excruciatingly well-documented operations in z/OS continue to be totally impossible in every single other operating system ecosystem, like "is this file in the page cache, and if not, why?" Imagine trying to answer that question without having to break out bpftrace or WinDBG in kernel debugging mode, and even then you'll have to go read the source code or load ntoskrnl into Ghidra to figure out where the data structures are. The worst thing about all this is that a lot of people in the industry think that it's normal and they're patting themselves on the back for being oh so clever at knowing how to do that.

Once you've been exposed to the mainframe way of doing application development, it ruins you for how most of the rest of the industry prioritizes feature velocity and how their focus is growing DAUs for people looking at cat pictures, or have PR campaigns about how it wasn't actually lying, technically speaking, when their other PR campaigns said the car can drive itself, and none of that had anything to do with killing those drivers when they took it at face value. Oh, by the way, the car can fart now, did you know?

All of this narcissistic Silicon Valley tech industry bullshit starts to feel, quite frankly, kind of disrespectful and like you're in a business relationship with juvenile clowns if you're trying to run a business that people are depending on for something that actually matters. Building things for support and for uptime isn't sexy, is bad for growth, and doesn't require a team of rockstar SREs who dress and act like fighter pilots. There is no "module of the week" for CICS. There will never be an "oh-my-TSO" package with emoji and themeable color schemes. If you go watch "For All Mankind," the 1970's era TSO prompts in the TV show look exactly the same as they do when you fire up your 3270 emulator on your iPhone today and remote into your frame. Everything about the mainframe ecosystem is exactly the kind of boring you want when your applications are supporting real people, spending real hard-earned money, on real products and services. That's why the parts of the tech industry that are constantly in the news don't use any of this technology: it works.

CS61A in particular is important for understanding the nuts and bolts of how programming works. The people who tended to do well in the course had little prior exposure to specific programming languages outside the prerequisite class. I think that all the abstraction that other programming languages do get in the way of understanding why the languages made some of those design choices, which is why throwing all the abstraction out the window is important to understand what's really going on.

I'm reminded of an experiment I was told to do when I was learning how to be a better cook, and I complained that I didn't understand what bay leaf was adding to a dish. Someone suggested cooking some plain white rice with some bay leaf, and I could finally taste the peppery notes it added. Sometimes you need a neutral substrate to see the details of the thing you're trying to learn.