michaeldwan

you're spot on. I use both Claude Code + OpenCode with many different models and friction is minimal as long as I'm deliberate about it. Hell, even symlinking AGENTS.md to CLAUDE.md is like 80% there.

It's just portability v convenience. But unlike ~15 years ago with cloud compute, it _feels_ like more people are skeptical of convenience, which is interesting.

I credit containerization, k8s, and terraform for preventing vendor lock in. Compute like EC2 or GCE are effectively interoperable. Ditto for managed services for k8s or Postgres. The new products Anthropic is shipping is more like Lambda. Vendor kool-aid lots of people will buy into.

What grinds my gears is how Anthropic is actively avoiding standards. Like being the only harness that doesn't read AGENTS.md. I work on AI infra and use different models all the time, Opus is really good, but the competition is very close. There's just enough friction to testing those out though, and that's the point.

Get out of here with this nonsense. We tell people when we’re a bad option all the time. Do you really think we have a desire (or time) to punish somebody for doing the same?

Also, here’s the long forgotten badge, still with 3 people… https://community.fly.io/badges/107/aeronaut

My point was that the app-not-working category is used in conjunction with support/our team getting involved. I assume this is what Sam meant by "flagged it internally", which was followed by investigation, then a post. I don't see how the timestamps uncover something nefarious.

That category was added after one of our support folks replied, likely for tracking. I don't know why it's private. They may not even know this category is private. Hiding negative shit wasn't a deliberate decision... we're aware of google cache and we don't need to give HN another reason to dunk on us.

I don't know why the app-not-working category effectively delists threads, but until we find out, I just removed it so this thread is public again.

As an employee that would be a catastrophic outcome.

Docker (or rather OCI) is our first class citizen. The launchers for phoenix/rails/etc just generate dockerfiles and config by inspecting source code.

This is really cool! I don't use discord much, how much effort would a Slack bot version be?

We do hire entry level people, in fact we're wrapping up our first cohort of interns right now. Exclusively hiring experienced dudes from the tech bubble is a failure mode in the long run. We're invested in hiring folks with diverse backgrounds and experience levels, and we need to talk about that more for sure.

Our jobs page is sparse right now because we’re focusing on hiring EMs to help grow a healthy eng org. We’ll have more openings before long, and you should absolutely apply when something sounds right. Feel free to hunt me down online in the mean time!

This is for employees only. We don't collect more than necessary for customers, and if we did we sure as hell wouldn't be sending it to Google.

Ah!!! Thanks for pointing that out, we'll take it down.

That’s an old api we need to sunset. How did you stumble on it?

What rest api is that? If it’s not the machines api it’s in graphql at api.fly.io/graphql.

Ah! I've had volumes on the top of my mind all week... Firecracker supports snapshots in a dev preview but we're not using it yet. We can't really do anything like that on Nomad, which is one of the many reasons we're keen to get off it.

If you mean moving (or copying) VMs to another region/host, sorta. Stateless compute is easy; our apps platform (orchestrated by Nomad) already does it. Volumes complicate things because they live on a specific host, and moving volumes between hosts is a slow and tedious process. Solving this is high on our priority list. We need super fast volume forking and host migrations like yesterday.

It already has it [1]. Native litestream that RPC's to the primary sounds interesting though!

[1] https://github.com/phoenixframework/phoenix/pull/4268

We're not shying away from "boring" stuff at all. We just have a small team with bigger priorities that's spread too thin. There's a million things like resizable volumes we need to ship and we're aggressively hiring to get them done.

there’s no obvious way to resize disks

Yes, this sucks right now. Resizable disks is on our list, we just need somebody to spend a few days on it. Luckily we're hiring platform engineers [1] to work on fun problems like that.

I actually am bothered that I basically don’t get billed.

We actually had a bug that skipped charging a bunch of accounts. :) Regardless, we're not overly concerned about making $1/mo from small accounts. Large customers more than make up for it. Turns out building something devs _choose_ to use on their free time often leads to using it at work too.

[1] https://fly.io/jobs/platform-product-engineer/

Node, Ruby, and Python come to mind. Oddly enough the alpine versions of haproxy and nginx on Docker hub had issues too.

Failing in one AZ probably means they changed something with DNS servers for that region. We had a similar issue recently when we rolled out a new DNS server that returned longer SOA records which broke python's mysql driver in only some regions during the rollout. Debugging nightmare fuel.

I really want to like Alpine, but we (Fly.io) have seen so many DNS issues with customer images that we’re now recommending Ubuntu or Debian slim. The extra ~50mb is a worthwhile trade off to avoid hard to debug musl-libc issues.

eg https://www.linkedin.com/pulse/musl-libc-alpines-greatest-we...

You control which region your app and it’s volumes are in. Metrics, logs, and volume snapshots end up on servers in the USA. We haven’t addressed data residency for those platform services yet, but we might someday if there’s enough interest.

We'll write about it when the time comes. To be fair, Nomad and Consul have served us well. Most of our troubles stem from abusing them in ways they weren't designed to handle.

As for autoscaling, our hands are tied as long as we're running on Nomad. Right now our autoscaler is nothing more than some ruby that loops over data from prometheus and changes counts in Nomad. It's slow and buggy, but worse we don't have control over where Nomad places VMs or which ones it stops when scaling down.

We're working on a replacement for Nomad (called flyd) that gives us full control over VMs. Once apps are running on that we can do a lot of cool things. Better autoscaling is one, but I'm really excited about suspending idle VMs that our proxy wakes up on demand. That'll cover most use cases without forcing customers to worry about counts or blowing through a budget.

It's unlikely we'll open source the Elixir app, most of it isn't that interesting. But extracting the machine code into an example app is a good idea! Until then, here's a prototype go proxy that launches suspended machines when requests come in: https://github.com/superfly/machine-proxy

We'll be baking that into our proxy soon so we can handle lambda-type workloads.

Image size impacts the cold startup time since it gets fetched and converted into a rootfs before launching. Caching speeds up subsequent launches. Once a machines is created, it can be stopped and resumed, which is _much_ faster. I don't remember the numbers, but it's in the several hundred ms range.

We'll have more to share soon.

The build executes untrusted code and temporarily authenticates with your fly.io and Heroku accounts, so we need a sandbox for isolation. Our Elixir app launches ephemeral VMs using our Machines API so it can offload that can of worms.

We use wireguard-go in our cli for connecting to vms over a private network. It is slower than wireguard in the kernel, but the convenience is hard to beat.

Here's an overview https://fly.io/blog/ssh-and-user-mode-ip-wireguard/

Our remote builders are just fly apps based on this image with an auth proxy in front. Works great https://github.com/superfly/rchab

Markdown + middleman. We use it for docs and landing pages in addition to the blog.