jacobwg

Hey, I'm one of Depot's founders - we built our own custom parser for the YAML syntax that translates to Depot CI's orchestration APIs. This API will actually be a public API shortly, which will unlock some interesting use-cases (support for more syntaxes, direct programmability of the orchestrator, etc)

You all should add EC2 - extra bonus if you have some way of tracking performance in addition to errors (right now we're seeing EC2 instances in us-east-1c not transition out of Pending status).

We've been observing EC2 instances launched in us-east-1c (use1-az2) remain in Pending status for a very long time / indefinitely, starting at around 16:00 UTC.

Since we launched this last year, GitHub released a v2 of their internal cache API [0], based on Twirp [1] of all things, so we adapted to that. Interestingly that Twirp service also receives Actions artifacts, though we have not intercepted those today given that you likely still want them to appear in the GitHub UI / be accessible from the GitHub API.

[0] https://github.com/actions/cache/discussions/1510

[1] https://github.com/twitchtv/twirp

Depot (W23) | Enterprise Support Engineer | Remote (UK, Europe) | Full-time | €100K - €160K | https://depot.dev

Depot is the fastest place to build software. We accelerate builds for customers using GitHub Actions, Docker, Bazel, Gradle, and more. We're seeking our first Enterprise Support Engineer to become a customer-facing expert on build optimization.

We're looking for someone with DevOps / CI consulting experience - you'll work directly with customers as the subject-matter expert on best practices, helping migrate legacy infrastructure, and working directly with the founders on product gaps.

Bonus: experience with Docker buildx, API integrations, or previous CI consulting.

For more details, and to apply: https://www.ycombinator.com/companies/depot/jobs/NdCr76D-ent...

Not quite for every container, but we operate a multi-tenant remote build execution service (container builds, GitHub Actions jobs, etc) so we launch a lot of ephemeral VMs in response to customer build requests. We use separate EC2 instances for strong workload isolation between different customers / jobs, and optimize boot time since that directly translates to queue time.

We also do GitHub Actions runners as a service, so a very high volume of differently-sized ephemeral VMs. We’ve experimented with .metal hosts, however they represent a bin-packing optimization problem, in that you will always be running some amount of spare compute / trying to fit the incoming build requests to physical hosts as tightly as possible.

Eventually you realize, IMO, that doing the bin packing yourself is just competing with AWS, that’s what they do when you launch a non-metal EC2 instance and it’s best to let them do what they’re good at. Hence why we’ve focused on optimization of that launch type, rather than trying to take over the virtualization.

There’s other security and performance reasons too: AWS is better at workload isolation than we can be, both that the isolation boundary is very strong, and that preventing noisy neighbors is difficult. Especially with things like disk, the strategies for ensuring fair access to the physical hardware (rate-limiting I/O) themselves have CPU overhead that slows everything down and prevents perfect bin-packing.

Yeah we do some similar tricks with our registry[0]: pushes and pulls from inside AWS are served directly from AWS for maximum performance and no data transfer cost. Then when the client is outside AWS, we redirect all that to Tigris[1], also for maximum performance (CDN) and minimum data transfer cost (no cost from Tigris, just the cost to move content out of AWS once).

[0] https://depot.dev/blog/introducing-depot-registry

[1] https://www.tigrisdata.com/blog/depot-registry/

No worries, entirely valid question. There may be ways to tune page cache to be more like this, but my mental model for what we've done is effectively make reads and writes transparently redirect to the equivalent of a tmpfs, up to a certain size. If you reserve 2GB of memory for the cache, and the CI job's read and written files are less than 2GB, then _everything_ stays in RAM, at RAM throughput/IOPS. When you exceed the limit of the cache, blocks are moved to the physical disk in the background. Feels like we have more direct control here than page cache (and the page cache is still helping out in this scenario too, so it's more that we're using both).

Exactly, a ramdisk-backed writeback cache for the root volume for Linux. For macOS we wrote a custom nbd filter to achieve the same thing.

Today we (Depot) are not, though some of our customers configure this. For the moment at least, the ephemeral public IP architecture makes it generally unnecessary from a rate-limit perspective.

From a performance / efficiency perspective, we generally recommend using ECR Public images[0], since AWS hosts mirrors of all the "Docker official" images, and throughput to ECR Public is great from inside AWS.

[0] https://gallery.ecr.aws/

The block level has two advantages: (1) you can accelerate access to everything on the whole disk (like even OS packages) and (2) everything appears as one device to the OS, meaning that build tools that want to do things like hardlink files in global caches still work without any issue.

A list of fun things we've done for CI runners to improve CI:

- Configured a block-level in-memory disk accelerator / cache (fs operations at the speed of RAM!)

- Benchmarked EC2 instance types (m7a is the best x86 today, m8g is the best arm64)

- "Warming" the root EBS volume by accessing a set of priority blocks before the job starts to give the job full disk performance [0]

- Launching each runner instance in a public subnet with a public IP - the runner gets full throughput from AWS to the public internet, and IP-based rate limits rarely apply (Docker Hub)

- Configuring Docker with containerd/estargz support

- Just generally turning kernel options and unit files off that aren't needed

[0] https://docs.aws.amazon.com/ebs/latest/userguide/ebs-initial...

We're having some success with doing this at the block level (e.g. in-memory writeback cache).

I'd love to experiment with that and/or flags like `noatime`, especially when CI nodes are single-use and ephemeral.

Depot | Founding Developer Marketer | Remote (US / EU) | Full-time

Depot is a build acceleration platform that makes Docker builds and GitHub Actions faster. We've already helped companies like PostHog, Wistia, Semgrep, and Secoda save thousands of hours in build time every week.

We're looking for our first marketing hire to define and execute our go-to-market strategy. If that's you, you'll own everything from content creation to demand gen, with a focus on developer audiences. We're growing rapidly with 500+ paying customers and double-digit monthly growth.

Requirements:

* 5+ years marketing experience with a focus on developer audiences

* Experience with content marketing, SEO, social, and email campaigns

* Comfortable with analytics tools (Google Analytics, ahrefs, PostHog)

* Experience with paid channels (LinkedIn, Reddit, etc.)

* Strong communication skills and ability to work asynchronously

Apply here: https://www.ycombinator.com/companies/depot/jobs/307RqGp-fou...

Tech stack: Remix, TypeScript, Go

We're a small, remote team building developer tools we wish we've had. If you're passionate about developer productivity and marketing to technical audiences, we'd love to hear from you.

https://youtu.be/E9yZ0JusME4 and the CEO on Twitter are the main sources for Arc being in maintenance mode as they work on their new product.

The dates feel like intentional deception. The community question is "how long have you been discussing trademark usage?" and the answer is "I had lunch in February 2023!"

Like somebody not trying to be deceptive would say "we started talking about trademarks and a commercial relationship in February 2023", but that's not what this post says, and that's not the answer Matt has given in interviews, it's always this strange list of dates instead.

Yeah in our case we are operating a private registry on behalf of our customers, so slightly different use-case than running your own registry for your own internal use.

If you do want to run your own registry, there's some great OSS projects including https://github.com/project-zot/zot, https://goharbor.io/, and of course https://github.com/distribution/distribution.

Data transfer costs. Our registry is for our customers, and they expect to pull their images from environments outside AWS, e.g. GitHub Actions.

If that customer has a 2GB image, they want to build the image and then pull it into 10 separate matrix jobs (think like parallel Cypress tests), and they have 1,000 commits in a month, then the AWS data transfer costs are $1,800/mo, just for that one customer to pull their images.

With R2, it acts both as a CDN and since Cloudflare does not charge for egress, reduces the cost to only storage + the one-time transfer out of AWS.

Uploading in chunks could definitely solve the issue, and the OCI Distribution Specification does actually have some language about an optional chunked push API[0].

Unfortunately very few of the registry clients actually support this, critically containerd does not[1], so this means your regular `docker push` and a whole lot of ecosystem tooling does not work.

This also means that the single PUT must be able to support very large pushes as a single request, possibly even larger than what R2 or S3 would allow without using multipart upload. This means you actually need a server to accept the PUT, then do its own chunked upload to object storage or otherwise stage the content before it's finally saved in object storage.

This rules out presigned URLs for push too, since the PUT request made to the presigned URL can be too large for the backing object storage to accept.

There's also other processing that ideally happens on push (like hash digest verification of the pushed layer) that mean a server somewhere needs to be involved.

[0] https://github.com/opencontainers/distribution-spec/blob/mai...

[1] https://github.com/containerd/containerd/blob/192679b05917b5...

The annoying thing about trying to implement a Docker registry on Workers and R2 is that it's so close to having everything you need, but the 500MB request body limit means Workers is unable to accept pushes of layers larger than 500MB. The limit is even lower at 100MB on the Pro plan[0].

We are running a registry that does store content on R2[1], and today this is implemented as the unholy chimera of Cloudflare Workers, AWS CloudFront, Lambda@edge, regular Lambda, S3, and R2.

Pushes go first to CloudFront+Lambda@edge, content is saved in S3 first, then moved to R2 in background jobs. Once it's transited to R2, then pulls are served from R2.

I would so love for Workers + R2 to actually be able to accept pushes of large layers, unfortunately I have yet to talk to anyone at Cloudflare who believes it's possible. Especially in this era of AI/ML models, some container images can have single layers in the 10-100GB range!

[0] https://developers.cloudflare.com/workers/platform/limits/#r...

[1] https://depot.dev/docs/guides/ephemeral-registry

Kinda - what we're seeing at Depot (https://status.depot.dev/clyrvud6i57402igofm6jtb7id) is that the APIs for managing Actions runners or retrieving work are receiving rate limit errors, interestingly missing the regular rate limit status headers[1]. We do have some jobs processing, but overall the GitHub API and Actions control plane seem very overloaded by the outage.

[1] https://docs.github.com/en/rest/using-the-rest-api/rate-limi...

Not a coincidence, no.

From the "self-hosted" perspectively, interestingly what we're seeing at Depot (https://status.depot.dev/clyrvud6i57402igofm6jtb7id) is API requests to provision new runners or receive runner jobs are receiving rate limit errors, but without the regular rate limit status HTTP headers[1].

I imagine the GitHub API / Actions control plane is rather overloaded at the moment with the outage.

[1] https://docs.github.com/en/rest/using-the-rest-api/rate-limi...

Not that I'm aware of, no, just the local state school.

My impression of the YC application process was that it was way more holistic than just educational background.

I mean, I didn't even graduate college :P

So we're just doing it on our own.

IMO if you can become a $100M household name without VC, that's absolutely the way to do it.

Even if you do take the VC path, YC for me was a massive boost in network, knowledge, and resources that I didn't have before, but it's also not the only way to acquire those things. You can even find that YC knowledge online, e.g. https://www.startupschool.org/.

That said, if anyone's even considering applying to YC, I'd recommend it, at a minimum it's a forcing function to think more deeply about your idea or business when assembling the application.

Correct yeah, each run starts on a brand new VM with a brand new disk - since these are EC2 instances with EBS volumes for their root disk, the whole instance and the EBS volume are deleted after the job finishes and are not reused.

Yeah, I think our goal is to be the fastest at building software, not necessarily the cheapest. Part of that involves AWS, to have access to more powerful and elastic infrastructure, but that comes at a premium.

But besides just compute, I think the bigger long-term unlock for build performance is a new distributed compute engine, to free build workloads from single machines. We've started building this for our container build product, and plan to integrate Actions jobs as an input as well, starting with the cache integration we have today.

Hey, we are building our base image from the runner-images repo! I'll send you an email!

At the moment, I think you'd want to use both products together, i.e. using `depot build` in place of `docker build` to move the container build portion to a Depot container builder.

I'd like to have a more automatic integration at some point - the challenge is that a lot of BuildKit's architecture performs best when many different build requests all arrive at a single build host, it is then able to efficiently deduplicate and cache work across all those build requests. So you really want the many different Actions jobs all communicating with the same BuildKit host.

We have some ideas for reducing the amount of change to Actions workflows to adopt ^ - longer term we're also working on our own build engine, to free those workloads from being confined to single hosts (be that single CI runners or single container builders).