Remote Code Execution as a Service

https://earthly.dev/blog/remote-code-execution/

by dijit • 3 years ago

127 42 3 years ago

Thanks for sharing this. One of the authors here.

We built a service that executes arbitrary user-submitted code. An RCE service. It's the thing you're not supposed to build, but we had to do it.

Running arbitrary code means containers weren't a good fit ( container breakouts happen), so we are spinning up and down ec2 instances. This means we have actual infrastructure as code (i.e. not just piles of terraform but go code running in a service that spins up and down VMs based on API calls).

The service spins up and down EC2 instances based on user requests and executes user-submitted build scripts inside them.

It's not the standard web service we were used to building, so we thought we'd write it up and share it with anyone interested.

One cool thing we learned was how quickly you can Hibernate and wake up x86 EC2 instances. That ended up being a game-changer for us.

Corey and Brandon did the building, I'm mainly just the person who wrote things down, but hopefully, people find this interesting.

iueotnmunto2• 3 years ago

While container breakouts do happen, they're pretty rare. I'd be more concerned about any potential injection vectors in the go code, which could lead to a cloud breach if you're not careful ;)

adamgordonbell10• 3 years ago

Oh interesting. What were you imagining as the injection vector?

The earthly backend runs on a modified buildkit so it is running the arbitrary code in a container, but it's also in its own VM. This was simpler then firecracker to get started but turned out to have pretty good performance and alright cost once we started suspending things.

raesene9• 3 years ago

There have been a bunch of Linux kernel privesc vulns that can be converted to container breakouts from standard Linux containers, just look at bounties from Google's kCTF (AFAIK they've had 10 different kernel vulns in 2 years)

It's possible to mitigate/reduce them for sure, with appropriate hardening, but the Linux kernel is still quite a big attack surface.

plicense• 3 years ago

I would suggest also looking at https://github.com/bazelbuild/remote-apis. Its essentially a standard API for remote (any binary) execution as a service and there are several reference implementations of it (Buildgrid, BuildBarn, Google's own service etc).

And you can consider using gVisor to minimize container breakouts to a great extent.

adamgordonbell10• 3 years ago

I'll checkout that remote-apis link.

gVisor was considered but so far it looks like the next iteration with be using firecracker vms. Our backend is buildkit and it can't run in gvisor containers without some work.

xxpor• 3 years ago

Did you consider using firecracker?

adamgordonbell10• 3 years ago

Yeah, we totally did and actually Corey has been playing around with a POC for backing with firecracker. It's likely the next revision of this will use firecracker.

So it wasn't so much disqualified as decided it wouldn't be the v1 solution. We wanted to get something out and get people on it and get feedback. So weren't afraid to spend some more computer dollars to do so.

cptnntsoobv• 3 years ago

One cool thing we learned was how quickly you can Hibernate and wake up x86 EC2 instances. That ended up being a game-changer for us.

Could you talk more about it? Are you keeping a cache of hibernated EC2 instances and re-launching them per request? What sort of relaunch latency profile do you see as function of instance memory size?

adamgordonbell10• 3 years ago

A specific Ec2 instance is always serving one customer max. And builds are highly cachable, so the ec2 instance has an EBS volume on it with a big cache that Earthly uses to prevent rework.

That instance is just sitting around waiting for gRPC requests that tell it to run another build. If it's idle for 30 minutes, it hibernates and then if another call comes back in a gRPC proxy wakes it back up.

I don't know if the wake up time increases per the size of the cache in memory, I can check with Brandon but its much faster starting up an instance cold, mainly because buildkit is designed for throughput and not a quick startup.

There are more details in the blog.

k__• 3 years ago

I had the impression that's what fly.io was doing.

Converting Docker images to run in VMs instead of containers.

slondr• 3 years ago

Fly uses Firecracker like AWS Lambda, not full-fledged VMs like EC2.

ones_and_zeros2• 3 years ago

Container break outs are rare and they typically require the attacker being able to control either the container creation parameters and/or the actual image being run. If you control those things and apply process isolation best practices (seccomp, cap drops, etc) then you are in pretty good shape.

Source: ran a container based RCE service that ran millions of arbitrary workloads per month. We had sophisticated network and system anomaly detection, high priced pentesters etc and never had a breakout.

justinclift• 3 years ago

... never had a breakout.

Would "never detected a breakout" be better wording? :)

ithkuil3• 3 years ago

Have you used things like gvisor?

adamgordonbell10• 3 years ago

Interesting! What was the service? IN our case we control the container, which is BuildkitD, but it has to be run privileged, which means lots of solutions are off the table.

preseinger3• 3 years ago

if your security model assumes user code is able to break out of containers, then it also has to assume user code will be able to modify the VM

if user code can modify the VM, and VMs are stateful, then how do you ensure hermetic builds?

adamgordonbell10• 3 years ago

So the builds are specified in Earthfiles, which are run by our buildkit backend.

Buildkit runs the builds in runC, so basically containers are used to keep things deterministic, but the buildkit backend isn't shared, each is in own EC2 instance.

So if you could break out or access cache entries that weren't correct you would have found a way to break reproducibility, but not access anything you shouldn't.

phphphphp• 3 years ago

Perhaps a dumb question but container breakouts are a problem for all sorts of services which have been addressed in different ways. Since your goal is not to prevent container breakouts but rather securely run third-party code, why did you choose to use EC2 over something like ECS or Lambda or Google Cloud Run which is already dealing with the security aspect on your behalf? Virtual machines seem less secure and less convenient.

adamgordonbell10• 3 years ago

Good question. Our goal is not just to run arbitrary code but to run it fast and cache rework. We are a CI service and speed is important. Brandon may be able to jump in with why not various options but it's hard to beat giving users powerful cloud machines to run their builds on.

I myself did try to run buildkit in a Lambda as I think that would be low cost option. But I found it you couldn't make gRPC calls against a lambda and that is a hard requirement for us.

brandonschurman2• 3 years ago

Some of the reason we went with EC2 over something like ECS is that we would need to run the container in privileged mode for some of our features to work. We also considered options like gVisor, but ultimately the EC2 route was a simple enough implementation that made it easy to manage the user's cache volumes, etc. We're also hoping to use Firecracker VMs in the near future.

AlexCoventry• 3 years ago

Are you concerned about AWS launching a competing service if you get significant traction?

rafark• 3 years ago

Aren’t servers remote execution services?

seanw444• 3 years ago

You mean a VPS? Yes. An HTTP server, no. At least not by default / design.

rkeene2• 3 years ago

I also allow remote code execution, though interactively [0]. Though I'm not using any VMs or anything like that for isolation.

Note: All the interesting stuff is in /opt/appfs

[0] https://rkeene.dev/js-repl/?arg=bash

v3ss0n• 3 years ago

You should try LXD containers. They are secured by default and very fast to spin up down. Can also snapshot

peter_d_sherman• 3 years ago

Remote Code Execution as a Service (AKA sandboxing distributed executable code remotely) -- is a brilliant business idea in my opinion, but IF AND ONLY IF such code execution can be guaranteed secure on end user's machines -- and my thoughts are that given all of the (lack of) security related issues that we read about in this day and age -- that it would be difficult, bordering on impossible, to create, much less rigorously prove such a security guarantee...

Still, if one were to attempt this exercise -- one path might be via extremely small emulated instruction set in an extremely tiny and open source (much easier to audit by end users) virtual machine where the emulated instruction set is NOT the same as the one on the host machine (i.e., RISC-V emulated instructions on x86 or ARM host...).

Would such an emulator be slower than native host execution?

Probably, possibly many times slower...

But, what might be lost in speed -- might be made up for in the ability to audit (at least better) for security concerns...

So in summation, is it a good idea?

Possibly -- but it needs that extra rigorous security guarantee which is very difficult to get exactly right in this day and age...

Also -- supposing that this business model was completely viable from a security perspective -- the next step to building such a system might be to build an auction site for CPU cycles (or compute ability/compute time) -- sort of like an "EBay for highly distributed but sandboxed code execution" (clients bid on renting compute resources, end users bid on selling theirs) which would be an interesting business model, IMHO...

Related: SETI@home: https://en.wikipedia.org/wiki/SETI@home

yetihehe• 3 years ago

Also -- supposing that this business model was completely viable from a security perspective -- the next step to building such a system might be to build an auction site for CPU cycles (or compute ability/compute time) -- sort of like an "EBay for highly distributed but sandboxed code execution" (clients bid on renting compute resources, end users bid on selling theirs) which would be an interesting business model, IMHO...

In Commonwealth saga from Peter F. Hamilton, people have access to nearby computing nodes like we have currently to free wifi, for accelerating any tasks they would need (like personal assistants, called e-butlers). That's not 100% secure and is sometimes used for nefarious purposes (and plot devices), but it's just accepted as a nuisance like other things that can be weaponised in our own reality (like cars).

19h• 3 years ago

Reminds me of the time in 2016 when I added a single line to one of our internal node modules to demonstrate how you can take over our entire Mesos cluster with a single compromised dependency.

Spawned a program on every single node that wrote "yes" into the file /etc/am_i_compromised..

Remote Code Execution as a Service

Related Stories