Mernit

We initially wrote this for a less technical audience (where we spelled out MCP), then edited it to post here - it's not AI, it's just bad editing from my part. Fixed now.

The only Beam-specific part are the sandboxes, but those can easily be swapped out for the vendor of your choice. The architecture we described isn't exclusive to our product.

Beam is fully OSS BTW: https://github.com/beam-cloud/beta9

what is missing for you?

beam.cloud (YC W22) | Product Engineer, SWE, SRE, Infrastructure, DevOps, Networking | NYC | Onsite

Beam is an ultrafast AI inference platform. We built a serverless runtime that launches GPU-backed containers in less than 1 second and quickly scales out to thousands of GPUs. Developers use our platform to serve apps to millions of users around the globe.

We’re working on challenging problems, including:

* Low-level systems development: working with container runtimes, OCI image formats, and lazy-loading large files from content addressable storage

* Efficiently packing thousands of workloads into GPUs across multiple clouds

* Working with cutting-edge technologies, like GPU checkpoint restore and CRIU

You don’t need prior experience with AI/ML, only an interest in working on hard problems and shipping quickly.

Email us at founders<at>beam<dot>cloud, or apply here: https://www.ycombinator.com/companies/beam/jobs/

You should look into beam.cloud (I'm the founder, but it's pretty great)

It lets you quickly run long-running jobs on the cloud by adding a simple decorator to your Python code:

  from beam import function

  # Some long training function
  @function(gpu="A100-80")
  def handler():
    return {}

  if __name__ == "__main__":
    # Runs on the cloud
    handler.remote()

You should checkout beam.cloud (I'm the founder). It's a modern FaaS platform for Python, with support for REST endpoints, task queues, scheduled jobs, and GPU support.

The major clouds don't support serverless GPU because the architecture is fundamentally different from running CPU workloads. For Lambda specifically, there's no way of running multiple customer workloads on a single GPU with Firecracker.

A more general issue is that the workloads that tend to run on GPU are much bigger than a standard Lambda-sized workload (think a 20Gi image with a smorgasbord of ML libraries). I've spent time working around this problem and wrote a bit about it here: https://www.beam.cloud/blog/serverless-platform-guide

beam.cloud | Founding Software Engineer, Infrastructure | Full-time | REMOTE | New York, NY USA

Beam is building a cloud runtime for running remote containers on GPUs. We’re used by thousands of developers for powering their generative AI apps, including companies like Coca-Cola, and we’re backed by great investors like YC and Tiger.

We’re building gnarly low-level distributed systems. You’ll have a major impact on the product and ship features directly to users. If working on a new Pythonic cloud runtime sound exciting, you might really like it here.

Apply here -> https://www.ycombinator.com/companies/beam/jobs/9fKNUsT-foun...

Yes we are hiring internationally

beam.cloud | Founding Software Engineer, Infrastructure | Full-time | REMOTE | New York, NY USA

Beam is building a cloud runtime for running remote containers on GPUs. We’re used by thousands of developers for powering their generative AI apps, including companies like Coca-Cola, and we’re backed by great investors like YC and Tiger.

We’re building gnarly low-level distributed systems. You’ll have a major impact on the product and ship features directly to users. If working on a new Pythonic cloud runtime sound exciting, you might really like it here.

Apply here -> https://www.ycombinator.com/companies/beam/jobs/9fKNUsT-foun...

There are a number of good options here. The different axes are cost of GPUs, performance, and ease of use / developer experience. You might consider beam.cloud (I'm one of the founders), which is oriented strongly on the performance and developer experience angle.

You should checkout https://beam.cloud (I'm the founder), it'll give you access to plenty of cloud GPU resources for training or inference.

Right now it's pretty hard to get GPU quota on AWS/GCP, so hopefully this is useful for you.

Cloudflare AI and Replicate are great for running off-the-shelf models, but anything custom is going to incur a 10+ minute cold start.

For running custom fine-tuned models on serverless, you could look into https://beam.cloud which is optimized for serving custom models with extremely fast cold start (I'm a little biased since I work there, but the numbers don't lie)

Serverless only works if the cold boot is fast. For context, my company runs a serverless cloud GPU product called https://beam.cloud, which we've optimized for fast cold start. We see Whisper in production cold start in under 10s (across model sizes). A lot of our users are running semi-real time STT, and this seems to be working well for them.

Sharing GPUs only really makes sense for GPUs that are large enough to share. MIGs can work for 80Gi A100s but won't work with smaller cards like T4s. It also adds latency to the GPU operations. Unfortunately there's not yet a silver bullet for this stuff.

What are you using for K8s autoscaling? We initially tried a few standard K8s scaling mechanisms and found that they didn't work well for GPU workloads. For example, if we were serving a low-RAM Huggingface model on GPU, it wouldn't trigger autoscaling. But since the GPU can only process one request at a time, the system would get bottlenecked while it waited to process requests one-by-one.

We wrote a bit about this here, if anyone is interested: https://www.beam.cloud/blog/serverless-autoscaling

You could easily host your model on https://beam.cloud (I'm a founder). You just add a decorator to your existing Python code:

    from beam import App, Runtime

    app = App(name="gpu-app", runtime=Runtime(gpu="T4"))

    @app.rest_api()
    def inference():
        print("This is running on a GPU")

You could try https://beam.cloud (disclaimer, I’m the co-founder)

You can run popular models on remote GPUs from your laptop with two commands. For example:

$ beam create-app stable-diffusion-gpu

$ beam start app.py

Slai | SRE / DevOps Engineer | REMOTE | Full Time

Slai is a tool to quickly build machine learning-powered applications. Our browser-based sandbox is the easiest way to build, deploy, and share machine learning models with zero setup [1]. We’re currently a team of four, and we’re looking to hire someone to help us with SRE / DevOps work.

You should have experience setting up and maintaining infrastructure at scale - ideally, you’ll have fluency with Docker/Kubernetes, EKS, KNative, Terraform, Terragrunt, Gunicorn, and Python. You should be able to communicate clearly in English, and you can work from anywhere (although if you prefer to work in-person, we have an office in Cambridge, MA)

We have a hackathon-inspired culture – the team works in one week sprints and has very few meetings besides daily standup and a Friday afternoon all-hands. If this interests you, please send a brief email with your resume to eli at slai dot io.

[1] Here's an example app built on our platform: https://www.slai.io/sandbox/62d86df31ba26200085cca99

On paper, Sagemaker does everything, but they don’t do many of those things well. I think Sagemaker is a great product for enterprises who want to maximize the products procured from a single vendor — it’s easy to buy Sagemaker when all of your infra is already on AWS.

It’s fairly painful to productionize a model on Sagemaker — they make you think about a lot of things and fit into AWS primitives. Besides the code for the model, we don’t force users to think about anything. Our focus is helping engineers get models into production, not reading documentation.

Using our tool, you can fork a model and deploy it to production right away — there’s no time spent battling AWS primitives. We’re focused on developer experience above everything else - which means we enforce sandboxes on our platform to be consistent and reproducible.

Hi! Feel free to shoot us an email, founders@slai.io

Yep! You can upload pre-trained models — just upload a pickled binary of your model into the “data” section of the sandbox and then load and return the object in the train function.

We chose to do it this way to ensure that the binary you upload is properly tracked in our versioning system, and that it can be integrated into your handler.

The core code editor itself uses Monaco (the same thing under the hood of VSCode), but everything else is custom (i.e. file browser, language server, syntax highlighting, tabs, etc.)

Would love to hear how it goes! Reach out if you have any questions - we've ported over quite a few text models.

Hi, thanks for checking it out! You bring up some great points.

When it comes to SWEs doing ML, our goal to bring together a bunch of apps that get people most of the way to something they can bring into production.

I appreciate your skepticism of online IDEs, and it’s unlikely we’ll ever completely replace notebooks or whatever IDEs people prefer hacking on locally. Instead, we’d like to take a hybrid approach, in which our online IDE is sufficient for making minor tweaks to a model after its in production, but the brunt of development can still happen locally, in which changes would be pushed to Git and synced with our online IDE.

It’s true that SWEs can deploy their own APIs, but that feels like an unnecessary annoyance to take for granted. At a high level you’re just setting up an API, but really you’re also going to setup some versioning system and a Docker file and monitoring, and all of that adds up to a lot of cognitive overhead.

BTW - the Jupyter-like cell execution can be turned on by clicking the “Interactive Mode” button on the bottom right corner of the IDE.

We think developer experience is the factor that has been sorely lacking from the ML tooling space. I’m curious how your experience using Sagemaker has been?

Hi, thanks! The main difference is that HuggingFace contains a huge repository of pretrained models, whereas we're providing the scaffolding to build your own end to end applications. For example, in Slai you could actually embed a HuggingFace model (or maybe two models), combine them into one application, along with API serialization/deserialization, application logic, CI/CD, versioning, etc.

You can think of us as being a store of useful ML based microservices, and not just a library of pre-trained models.

Thanks for checking us out! We sort of anticipated that — we just launched the unit test feature this week, and it’s not easily distinguished from the test panel. We’ll probably rename the test panel to something more descriptive, like API runner. Interactive mode is a feature to set interactive breakpoints in your code, it’s really useful for iterating on individual blocks of code without having to run the entire app E2E.

Hi, thanks for the questions! We're planning on adding a way to synchronize a git repo to a sandbox, should be out by April.

Right now, you can upload a trained binary to a sandbox and return it from the train function, and then use it for inference. So it's a bit manual at this point, but we're planning on improving that workflow shortly.

We built linting and testing into the sandbox, but testing is currently triggered manually - we're planning on building both into our CI/CD system (scheduled training)

Thanks, that means a lot! We've spent a lot of time thinking about the pitch, given how many other ML tools are out there. We're hoping to strike a chord with simplicity and developer experience.