jfrisby

Since this is HN, I'm gonna pick a nit.

A clever and witty bash script running on a unix server somewhere is also not utilitarian coding, no human ever directly benefited from it.

Back around 2010, my friend Mat was doing cloud consulting. He wrote some code to screen-scrape the AWS billing and usage page for an account to determine how much had been spent day-over-day. This was, of course, all orchestrated via a bash script that iterated through clients and emailed the results to them (triggered by cron, of course).

He realized he had startup on his hands when something broke and clients started emailing him asking where their email was. Cloudability was born out of that.

I'd say that both the Ruby and bash code involved count as pretty utilitarian despite running on a server and not having a direct user interface.

If you are good enough at the task of "finding a cofounder" that you can offer useful advice on it, you are... bad at some very critical things.

This is not a task anyone should be experienced enough to be good at.

I saw a headline about the book on Reddit and made a joke to a friend of mine: "How much you wanna bet it uses mmap?" Well...

It uses mmap.

Because employers _want_ people to be fungible. The quest to quantify and systematize hiring is a Quixotic effort at solving real challenges, but it persists because nobody has come up with an actual solution.

Almost, but not quite the audience for your question.

I was going to use Heroku, but the security requirements for my app made that a non-option. Instead, I went with simple Terraform that spins up CoreOS nodes, using cloud-init to spin up Docker containers. CI process built the Docker images, and deploying was just a matter of making an ECR tag point at the new image then cycling instances. Not quite as simple as Heroku, but pretty close.

I moved onto AWS via this: https://registry.terraform.io/namespaces/GoCarrot

No k8s. No Docker. Beautifully clean system. Blue/green deploys with automatic rollback. Continuous deployment (there's a CircleCI orb as well). Tightly buttoned-down system configuration. Debian built from scratch with an eye towards supply chain security. (There's a root image factory and base image factory to handle the layering of image build processes involved.) Log aggregation and configuration management baked in. Security that can only be described as "insanely paranoid, yet oddly pragmatic." Cascaded image builds, so I can update the entirety of my infrastructure by pushing one button to kick things off then clicking a few buttons to approve deployment of the various services.

1. About a week, although I had the assistance of the author. I'm now rebuilding my personal infra on Omat and it's going much more quickly. Probably 3 days total, with no assistance.

2. More experience with DevOps stuff than most, but not a DevOps person by any stretch.

3. Very, very, very instructive.

4. Given what I was coming from, cost-neutral. Compared to Heroku? Notably cheaper.

5. At present, we're in alpha so traffic is negligible and back-end workload is fairly minimal (tens of thousands of jobs per day). The author of the tools, however, is CTO of a mid-tier SaaS that handles a quite significant (millions of transactions/day, IIRC) amount of traffic, and he is super aggressive about not being hobbled on performance needs -- but also being cost-efficient.

6. Avoiding the k8s iceberg, while having all the modern amenities in a system I actually have a hope of understanding top-to-bottom (modulo my hesitation around reading the systemd source) is nice. This system is an object lesson in "loosely coupled, highly cohesive" design. I haven't felt at any point that I may be stuck in a You Can't Get There From Here situation. Avoiding layering a second layer of software-defined networking (Docker/k8s) on top of the software-defined networking of AWS means I neatly avoid the single biggest source of chronic issues (and, via the workarounds needed, system complexity) that I've experienced with "modern" (Docker/k8s) DevOps approaches.

My home folder is a git repo (with a very extensive .gitignore, so I'm only versioning dotfiles, ~/bin, etc), with a branch per machine. Copying over the .git folder to a new machine is step 1. From there, I selectively restore

I have a Brewfile/Brewfile.lock, so I can go from setting up Homebrew to having most of the core tools I need very easily.

I have some scripts in ~/bin for helping with setup using the defaults command.

Other handy tools include chflags, scutil, possibly pmset, etc.

My config setup script looks approximately as follows -- hope you find some of this useful!

https://gist.github.com/MrJoy/20accc3b463e75ce5eecbbd0cff841...

Rails + graphql-ruby + ActiveAdmin + Devise + SideKiq/Faktory + Postgres remains a ridiculously productive combination for me.

I'm leaning more towards Faktory these days because it makes it possible to move individual jobs from Ruby to Go if/when performance becomes an issue and the much simpler client-side logic makes making performance-related adjustments much simpler and less risky.

For the front-end, I've been dabbling with Next.js, and using Tailwind heavily. Deployed to S3/CloudFront. I'm not completely sold on Next.js yet, but I've liked it better than the other options I've tried so far.

For deployment of the backend / admin tooling, Docker + ECR + EC2 + CoreOS, although I'm looking at changing out CoreOS for Debian now that it's effectively DOA. CI does a Docker image build and pushes to ECR, tagging with the git hash. Deployment consists of changing an environment-specific tag to the desired image, and replacing servers. For blue/green deployment you just use 2 different tags (e.g. prod-blue / prod-green).

I keep a docker-compose config for developers who only work on one piece of the stack at a time and don't want to deal with the complexities of managing the local development environment for the rest but don't personally use it in development.

For context: I'm a serial technical founder, so time to market is usually the biggest priority for me. Of course, different requirements / career paths are often going to lead to other options being more suitable.

The workaround I've found: Select the relevant line(s), but before releasing the mouse button, cmd-c (or the equivalent shortcut in your local computing environment).

But yes, it's incredibly annoying. Especially for those of us with ADHD who highlight parts of text as an anchor so we can get back on-track faster if we get distracted.

As I touched on, the cost can vary a lot depending on the particulars of the situation.

For example, one company I came into was using Redshift and fighting a lot of problems with it. The problems stemmed from a combination of not knowing how to use it effectively (e.g. batching writes), it being the wrong tool for the job (they were using it for a combination of OLAP and OLTP(!!) workloads), and so forth. The long and short is that for both workloads, a basic RDS Postgres instance -- or a pair, one for OLAP, one for OLTP -- would've been considerably cheaper (they'd had to scale up a couple notches from the minimum cluster size/type because of performance), avoided correctness problems (e.g. UNIQUE constraints actually doing the thing), been several orders of magnitude more performant at the scale they were at, etc. They simply didn't understand the tool and tried to apply it anyway. They basically thought Redshift was like any RDBMS, but faster "because it stores data in a columnar format."

Had they understood it, designing a data pipeline that could make use of such a tool would have required considerably more engineering person-hours than the pipeline they actually built.

Obviously this is a terribly extreme example, but the learning curve for tools -- including the time/effort needed to discover unknown unknowns -- is a cost that must be factored in.

And, even if your org has the expertise and experience already, more-scalable solutions often have a lot more moving parts than simpler solutions. Another organization I was at wanted to set up a data pipeline. They decided to go with Kafka (self-managed because of quirks/limitations of AWS' offering at the time), and Avro. After 2 months that _team_ (4 sr + 1 jr engineers, IIRC) had accomplished depressingly little. Both in terms of functionality, and performance. Even considering only the _devops_ workload of Terraforming the setup and management of the various pieces (Avro server, Zookeeper cluster, Kafka cluster, IAM objects to control access between things...), it was a vastly more complicated project than the pipeline it was meant to replace (SQS-based). Yes, that's a bit of an apples-to-oranges comparison but the project's goal was to replace SQS with Kafka for both future throughput needs and desired capabilities (replaying old data into other targets / playing incoming data into multiple targets without coupling the receiving code).

By the time I left, that project had: 1. Not shipped. 2. Was not feature-complete. 3. Was still experiencing correctness issues. 4. Had Terraform code the CTO considered to be of unacceptably poor quality. 4. Monitoring and observability was, let's say, a "work in progress." Some of that is for sure the Second System Effect, but it is not at all clear to me that they would have been better off if they'd gone with Kafka from day 1.

Given that we could pretty easily extract another 2 orders of magnitude throughput out of SQS, there's a real discussion to be had about whether or not a better approach might've been to make a Go consumer that consumed data more efficiency, and shunted data to multiple destinations -- including S3 to allow for replay. That would've been a 1-2 week project for a single engineer. Kafka is 100% the right tool for the job _beyond a certain scale_ (both of throughput and DAG complexity), but the company was something like 4 years in when I got there, and had been using SQS productively for quite some time.

And no, 26k/sec/server isn't especially huge. I was referring to the fact that the downvoted commenter was making sweeping generalizations. Sweeping generalizations tend to shut discussion down, not prompt more nuanced discussion. Other threads on this post have seen very interesting and productive discussions emerge, but note that the downvoted commenter's post hasn't really drawn anything other than people being sucked into the very discussion we're having now. It's counter-productive.

Well, for starters, it's simply untrue that every company (likely) will (barring bankruptcy of course) eventually need extreme scale, and the implicit assumption that the up-front cost of implementing such scalability is necessarily worthwhile. That may be true in Staruplandia, but the industry is a _lot_ bigger than the world of Silicon Valley style startups.

For example: My current company is a small lifestyle biz. Sure, it's a "tech company", has an event-processing pipeline and the like, but if we were 100x more successful than our wildest ambitions, we still wouldn't need anywhere near 25.6k IDs/sec/server. And, given the objectives of the company, it would make more sense to turn customers away than to grow the team to accommodate that sort of demand.

The simple fact is that in a great many situations extreme scale isn't needed. And, either way, the cost of implementing extreme scalability can inflict its own harm. Highly scalable systems usually come with more operational complexity, and steeper learning curves. When you have a small team, this can impair -- or even cripple -- product development. If your company hasn't found product/market fit yet, this attitude of sacrificing the present for an imagined future can materially reduce the likelihood of that future coming to pass. Of course, the opposite is sometimes true as well. Companies have, in fact, failed because they found product market fit but took so many shortcuts they couldn't adapt and grow into their success. But the point is that determining how much to invest in future-proofing is a complex and nuanced problem not amenable to sweeping generalizations.

Now, this clearly isn't such an extreme case, but frankly Sonyflake (the original, not the Rust implementation) seems to be operationally simpler than Snowflake while offering perfectly reasonable tradeoffs. The Rust implementation might prove entirely useful to any number of organizations _based on their needs_. If they have a Rust codebase with a single process per machine this could easily be a simpler and more robust option than Sonyflake.

The kind of arrogant dismissiveness based on one's own personal (and highly specialized) experience that's shown in the downvoted comment tends to leave a bad taste in peoples' mouths. Thus the downvoting.

I think you're crossing the streams a bit here.

Twitter designed _Snow_flake. _Sony_flake is a reimplementation that changes the allocation of bits, and the author acknowledges that the maximum ID generation throughput ceiling is lower than that of Snowflake.

Snowflake uses a millisecond-precision timestamp, and had a 12-bit sequence number. So 4,096,000 IDs per node-second. At that point, the bottleneck won't be the format of the IDs but the performance of the code, and IPC mechanism. Which, in this case, is non-trivial since Snowflake uses a socket-based approach to communication. Lower-overhead, native IPC mechanisms are certainly possible via JNI but would probably take some doing to implement. For Sonyflake, I don't imagine the socket overhead is all that much of an issue given the low throughput it's capable of with its bit allocations.

Were I to design something like this again[1], I might start with something like Sonyflake (the self-assignment of host ID w/out needing to coordinate via Zookeeper is nice), shave a couple bits from the top of the timestamp, maybe a couple from the top of the host ID, and pack the remainder at the top, leaving a few zero bits at the bottom. That would essentially mean the value returned was the start of a _range_, and anything needing to generate IDs in large quantities can keep its own in-process counter. Only one API call per N IDs generated by a given thread/process. And, of course, unix-domain socket or other lower-overhead approach to communication for when the API calls are needed.

[1] - A decade prior to Snowflake's release, I wound up taking a very similar approach at one of my first startups, albeit much more crude/inelegant and without nice properties like k-ordering.

George R.R. Martin? Is that you?

I suspect the surrounding context is Apple threatening to yank the Hey app for bypassing Apple's cut by not using IAPs for managing subscriptions.

What you're doing by keeping up to date continuously is reducing the cost of a major version bump down the road -- should that become required.

For example: If you had a Rails 3 app, and you updated to Rails 4 when Rails 4 came out, then to Rails 5 when that came out, you'd find the upgrade to Rails 6 is relatively straightforward. If, however, you stayed on 3 until 6 came out, then discovered you _really really_ need to be on 6 for whatever reason... Well, that's a nightmare of a project for any non-trivial app.

Each individual upgrade takes time and effort, of course but the developers and/or community of whatever tool/library/framework are absolutely gonna chart an upgrade path from (X) to (X+1). Going from (X) to (X+N), however can be exponentially more complex -- and you'll rarely find clear guidance / documentation / etc, if only because of the large number of permutations.

Now, why might you find yourself in sudden need of a major upgrade? A common situation is that a major security vulnerability is discovered that affects you -- but the version you're on is past the end-of-life for security updates. That's not the only possible situation, but it's perhaps the most obvious and broadly applicable scenario.

The worst possible time to have to face a potentially very challenging upgrade process is when there's a zero-day in the wild, and you're vulnerable.

So really, what you're doing by keeping deps up-to-date is mitigating the risk of winding up in that situation. This comes at a non-zero cost, of course. Hence your question.

The difficulty is that it's really hard to put a meaningful probability on whether/when you might wind up in a situation like that. I've started a bunch of companies over the >20 years I've been coding professionally, so I've gotten to see this play out in a variety of ways. I've had companies where it was never an issue. I also had one where my last-ditch effort to save the company was foiled by an upgrade path that effectively amounted to "rebuild everything from scratch". That one wasn't security-related and it was a (3D) game, so the circumstances there are probably pretty atypical for most folks on HN -- but it was a serious kick in the teeth for me.

Given that, I tend to assume the probability that I'll be in a position where I _must_ do a major, multi-version upgrade goes to 1 on a timeline of about 4-5 years. From there, it's just a matter of how effectively I can amortize those incremental costs. Patch-level releases? Automate the update (e.g. DependaBot). Minor upgrades? Lean in on anything where changes are mandatory (pretty uncommon), but don't block things up on fixing every deprecation -- those can be addressed incrementally. Major updates? Deal with that on a case-by-case basis, factoring in how major/breaking the changes are, and what are the circumstances the business is facing. Often this involves doing a spike of a _super_ rough upgrade, just to see what breaks.

I can't actually think of any circumstances where I'd look at the costs of incrementally keeping up to date and conclude that it was a bad idea to do so, although I confess that may just represent a lack of imagination and/or a bit of paranoia on my part.

I've never quite gotten the hang of `git add -p`, so I use `git gui` (incl. with Homebrew version of git, among others), or other tools that let me do partial staging/unstaging via line-selection with a mouse. It's a handy way to onboard oneself to such a powerful capability, if one feels a bit overwhelmed by the relatively complex UX of the CLI approach.

It's just serialization as a workflow mechanism, and it really gets rid of a lot of workflow cliffs.

It's also nothing to be afraid of: You can absolutely do everything programmatically, using the same (mostly-public) APIs that the Unity editor (which is built using much of Unity itself). If you're ever wondering how something gets achieved in the editor, use MonoDevelop to decompile/disassemble the various .dll's that Unity includes.

I built UnityREPL precisely because I wanted to do a lot of things programmatically (especially, for example, making similar changes to a set of related prefabs).

Octopress, which is built on Jekyll. Unfortunately I bet on Octopress waaaay back in the day when it was essentially a very customized Jekyll install. This has led to a situation where updating to a newer release of Octopress, or even upgrading Jekyll itself is more or less a rewrite.

Just saw your question on a thread from a couple weeks ago about building a TD game. As I'm only now seeing it, I apparently can't respond directly anymore. Email me -- jfrisby at MrJoy dot com and I'll be happy to offer some tips!

I've now worked at two companies in that space -- a startup I cofounder in 97, and a much later-stage startup I recently joined -- and all I can say is... shrug

The company I'm currently at is certainly making things better for a subset of candidates, but what we do is largely orthogonal to these issues.

i suspect that even without the ridiculous hoops, the incompetent will always spend a disproportionate percentage of their time job hunting, and thus making the hiring process challenging.

It's a combination of two things:

1. We're bad at hiring, as an industry. We just aren't very good at identifying good candidates.

2. We're an industry where the average skill level is below the minimum threshold of competence for even low-level roles.

I.E. there are lots of people looking for jobs, but most of them either couldn't code their way out of a wet paper bag -- and identifying the good candidates is time consuming and error prone.

I had a chance to ship the first Tower Defense game for iOS. The OS X game I was porting had some crippling performance problems that were incredibly hard to track down.

The problem was two-fold:

1. The relevant tools (Unity3D) were extremely immature and the problem was quite diffuse. No profiler, poor quality of generated code, tiny caches, etc.

2. A problem in string-handling code that was quite diffuse throughout the game. As near as I can tell, it was blowing out the tiny CPU cache hundreds of times per frame.

On desktop, this code was a complete non-issue. On the puny little ARM on the iPhone? It was the difference between having dozens of towers and 50 enemies in play vs half a dozen towers and less than a dozen enemies in play. The impact on game dynamics, and need to re-balance everything by itself would add weeks to the shipping schedule.

There were plenty of other things that needed to be scaled WAY back of course: Switching from 3D to 2D to get vertex count and draw call count down. Completely rebuilding the entire UI. Revamping the pathfinding and suffix caching to not play havoc with the CPU cache. Moving from a 24x24 grid to a 12x12 grid. All of that combined helped a LOT, but not nearly enough.

The string manipulation was for a hierarchical property system that let me parameterize all sorts of attributes for enemies/spells/towers/projectiles in a set of text files. Ultimately, I had over-engineered on the assumption that I would be tweaking many more things -- with much greater frequency -- than I wound up actually tweaking.

Had I ripped most of it out and just had local properties on each prefab that I assigned manually, I might've hit that market opportunity. Finding that that was the cause was a multi-month project because of how interwoven it was with everything else. Hell, it would've been fine, had a not over-generalized it into a shared component on each prefab that the other components inquired with to get property values. But I did. And it took me vastly too long to identify it was the major problem it was.

Opportunity missed, and that was the final nail in the coffin for my fledgling game studio.

I've worked at maybe two companies with a bus factor > 1. Maybe.

https://en.wikipedia.org/wiki/Failure_rate

Except that the retrospective IS predictive.

And the difference is?

PR is PR -- and yes, the ultimate vision is to get to trustworthy un-assisted self-driving.

And yet, before you can engage this mode the Tesla gives you a very stern warning that it's NOT perfect, and you DO need to keep an eye on what decisions it's making and you MUST keep your hands on the wheel.

We're not talking pages of obtuse legalese, we're talking about a single screenful of clear, straightforward warning text.

Also, as a licensed driver, you have obligations that you are expected to be mindful of, not the least of which is maintaining control of your vehicle at all times.

Are some people going to ignore it? Unfortunately, that's likely. But despite that reality, Tesla's autopilot is apparently doing better in terms of accidents per 100m miles driven than unassisted drivers.

Nobody who works on safety-critical systems is an idiot incapable of imagining or caring about the consequences of their errors.

Your ignorant, fact-free fear-mongering is antithetical to saving lives.

1. An immense number of lives are being "carelessly sacrificed" right now, and will continue to be sacrificed every day until something comes along to fix the fundamental problem: People are terrible drivers. 2. Tesla's system did not override control of the vehicle. It was not presented as being safe or ready to operate without oversight. The law does not allow for it to be treated as such. The driver failed to hit the brakes despite being told by the car -- and by legal obligation -- that it was his responsibility to hit the brakes. 3. Tesla's system has proven to be safer than the average, unassisted driver, to date.

If the proportion of accidents involving Tesla's autopilot remained constant when scaling up to more drivers and more miles, then it would be a net win in terms of lives saved to deploy it widely right now.

Obviously though, nobody -- including Tesla -- expects that to be the case, or they'd have been touting this as something every driver should have RIGHT NOW.

Tesla is doing their public beta precisely because it's impossible to design a safety-critical system with perfect foresight. More information is needed to help them build a robust system and the public beta has proven to be an uncommonly safe and effective way to get that information.

If you think my rhetoric is inflammatory it's because you're blind to the logical consequences -- the human cost -- of what you advocate for.

I have empathy for the ~33,000 people that die every year because humans are terrible drivers.

You do not. You want to take a moral posture that condemns thousands of people to needless deaths because you're so caught up in emotion you can't see the harm your position would cause.

Wallowing in your own emotions is selfish and immoral when it costs actual lives.

Your entire premise is based on the false assumption that this system was making all the decisions. Tesla's "autopilot" does not override the driver. It does not replace the driver (yet). That's explicitly because everyone involved knows it's not ready for that responsibility yet, no matter how much they've tested it. Tesla knows they cannot possibly anticipate everything -- the whole point of this "public beta" is to gather more information and help them make a better, safer system that eventually is trustworthy enough to consistently make decisions better than people.

You posit that human decision-makers behind the wheel are an important thing we should be wary of giving up but you have ignored one simple fact: There WAS a human behind the wheel, expressly charged -- both by law and by Tesla -- with maintaining control of the vehicle.

The driver had the responsibility of hitting the brakes, and did not do so. That's an error that plays out dozens of times per day without AI drivers being involved.

But instead of recognizing that this is a case of not just a piece of software failing, but also of a human failing -- and instead of recognizing the cost of lost and shattered lives borne by society every day -- you resort to ill-conceived hand-wringing in an attempt to convince others to follow you down a path that would continue condemning untold numbers of people to needless deaths.

Except that the car wasn't autonomous here.

The human behind the wheel had been explicitly instructed to keep hands on the wheel and feet on the pedals -- and that it was his responsibility to override the machine's judgement if circumstances warranted.