Show HN: Neurox – GPU Observability for AI Infra

https://github.com/neuroxhq/helm-chart-neurox-control

by leeab • 1 year ago

25 22 1 year ago

leeab10• 1 year ago

GPU observability is broken, so we built Neurox.

When I co-founded Mezmo (a Series D observability platform), we obsessed over logs, metrics, and traces. I learned firsthand how critical app-level observability is for DevOps, cutting through logging noise and finding the needle in the haystack is everything.

But after diving into AI infra, I noticed a huge gap: GPU monitoring in multi-cloud environments is woefully insufficient.

Despite companies throwing billions at GPUs, there's no easy way to answer basic questions:

- What's happening with my GPUs?

- Who's using them?

- How much is this project costing me?

What's happening: Metrics (like DCGM_FI_DEV_GPU_UTIL) told us what was happening, but not why. Underutilized GPUs? Maybe the pod is crashlooping, stuck pulling an image, or misconfigured, or the application is simply not using the GPU.

Who's using the compute: Kubernetes metadata such as namespace or podname gave us the missing link. We even traced issues like failed pod states, incorrect scheduling, and even PyTorch jobs silently falling back to CPU.

How much is this gonna cost: Calculating cost isn't easy either. If you're renting, you need GPU-time per pod and cloud billing data. If you're on-prem, you'll want power usage + rate cards. Neither comes from a metrics dashboard.

---

Most teams are duct-taping scripts to Prometheus, Grafana, and kubectl.

So we built Neurox - A purpose-built GPU observability platform for Kubernetes-native, multi-cloud AI infrastructure. Think:

1. Real-time GPU utilization and alerts for idle GPUs

2. Cost breakdowns per app/team/project and finops integration

3. Unified view across AWS, GCP, Azure, and on-prem

4. Kubernetes-aware: connect node metrics to running pods, jobs, and owners

5. GPU health checks

Everyone we talked to runs their compute in multi-cloud and uses Kubes as the unifier across all environments. Metrics alone aren't good enough. You gotta combine metrics with Kube state and financial data to see the whole picture.

Check us out, let us know what we're missing. Curious to hear from folks who've rolled their own, what did you do?

Lee @ Neurox

nickysielicki2• 1 year ago

Everyone we talked to runs their compute in multi-cloud and uses Kubes as the unifier across all environments.

I categorically support any company willing to take a strong stance on the total irrelevance of slurm.

dharmab• 1 year ago

Is your comment pro-SLURM or anti-SLURM?

I took a serious look at SLURM for my problem space and among my conclusions were:

- Hiring people who know Kubernetes is going to be far cheaper

- Kubernetes is gonna be way more compatible with popular o11y tooling

- SLURM's accounting is great if your billing model includes multiple government departments and universities each with their own grants and strict budgets, but is far more complex than needed by the typical tech company

- Writing a custom scheduler that outperforms kube-scheduler is far easier than dealing with SLURM in general

firgrove• 1 year ago

this feels like grafana with extra steps

leeab10• 1 year ago

Haha...there is some truth to that. We do use Prometheus under the hood to collect metrics. However, our thesis is that metrics alone isn’t enough. We marry metrics + kube state + cost data to get the whole picture.

Also we're purpose built to monitor GPUs, so we have things like drilling down from a Kube cluster, down to GPU nodes, down to a GPU card.

freeatnet• 1 year ago

Interesting! A friend recently asked me if I knew of any tools to improve GPU observability across their deployments (primarily for cost tracking purposes, I think), but he was looking for an OSS solution. Do you plan to open source this?

leeab10• 1 year ago

We have considered this and may go down this route in the future. One thing we asked ourselves was what open sourcing provides. Usually it's a desire for privacy or cost in the form of self-hosting, among other reasons.

Currently, our free version is self-hosted and monitors clusters with up to 64 GPUs. We feel this will work for many use cases, especially just to try it out. Monitoring GPUs typically requires you to deploy something where your GPUs live. Since you’re already installing software on your cluster, you might as well keep your data there too.

fustercluck2• 1 year ago

Your Github repo says you need 120 GB of persistent storage, but our bare metal GPU clusters only have local storage. Would like to try your thing, but hosting the data with the GPUs is a pretty big blocker for us.

nicoslepicos• 1 year ago

I've heard a few folks at events mention curiosity about stuff like this.

Given you decided to start self-hosted, are you planning on a cloud version in the next while too?

I'm curious also who you think is the right fit for this right now in terms of initial users

leeab10• 1 year ago

One of the reasons we went down the self-hosted route is to ensure that your data remains on your servers. Since our architecture allows for separation between where our control plane lives vs where GPU workloads run, we can definitely host the control plane portion for you. Then you just need to run our agent only on your GPU cluster. Shoot me an email: lee at neurox.com and we can discuss!

zekrioca• 1 year ago

* Not open-source.

leeab10• 1 year ago

Someone asked about this earlier. We have considered this and may go down this route in the future. Was there something specific that you were looking for with open source? (ie: privacy, cost, etc)

Our solution is self-hosted and your data remains on your servers. And I think we do provide a fairly generous free limit of 64 GPUs.

28374654• 1 year ago

Arjikh

badmonster• 1 year ago

What metrics and Kubernetes runtime data does Neurox collect to provide its AI workload monitoring dashboards, and how customizable are these dashboards for different user roles like developers or finance auditors?

leeab10• 1 year ago

We collect a handful of metrics, but coming from our previous lives in DevOps, we only collect just what's needed to avoid unnecessary metrics bloat.

The main 3 are:

- GPU runtime stats from NVIDIA smi

- Running pods from Kube state

- Node data & events from Kube state

We have several screens with similar information intended for different roles. For example, the Workloads screen is mainly for researchers to monitor their workloads from creation to completion. The Reports screen shows mainly cost data grouped by team/project, etc.

mountainriver• 1 year ago

Kubernetes is now kinda a bad abstraction for accelerated compute with the GPU shortages

leeab10• 1 year ago

Well, it depends on how many GPU clouds you're managing. We've talked to a bunch of companies, some startups, some enterprises and the main trend we found was the sheer number of companies with GPU clusters from multiple clouds...likely due to GPU shortage.

And yes, it's nowhere near 100% but an overwhelming majority was running Kubes for GPU workloads...mainly cuz so they'd have a unifying layer that wasn't managing each cloud separately and being proficient with their respective tooling.

Are you using something else? Slurm, docker, etc?

nickysielicki2• 1 year ago

Are you concerned at all that your name is one letter away from neuronx?

https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neur...

leeab10• 1 year ago

I've heard of AWS Neuron...didn't realize they had a package called NeuronX. Tbh, I feel like many AI companies have similar names. Maybe the guys who own neuronx.io might be more concerned...

Show HN: Neurox – GPU Observability for AI Infra

Related Stories