New Relic to open-source Pixie’s eBPF observability platform

https://blog.pixielabs.ai/pixie-new-relic/

by htroisi • 6 years ago

345 63 6 years ago

blog.pixielabs.ai

I'm curious to hear people's thoughts on eBPF generally, it seems likely that this is where observability companies are headed. It's non-trivial to implement but monitoring from the kernel layer makes so much sense that I expect the tooling will come along quickly.

dan-buzzkill• 6 years ago

eBPF is great but it only works for linux and access to the kernel layer doesn't work for serverless environments, so it's definitely a piece of the puzzle but not a silver bullet IMO.

zasgar4• 6 years ago

Co-founder/CEO of Pixie here.

eBPF is indeed a part of the puzzle. It allows us to access telemetry data without any manual instrumentation when running on Linux machines.

Pixie itself is extendable and currently ingests data from many other sources as well. Joining forces with New Relic will allow us to focus on expanding the open-source project, but also expand our capabilities by plugging into other open APIs and frameworks such as OpenTelemetry, Grafana, Prometheus.

xiphias22• 6 years ago

Is there any technical reason why it couldn't work with serverless?

throwaway894345• 6 years ago

What's the advantage of monitoring from the kernel layer? It's not jumping out at me...

SEJeff4• 6 years ago

Tracing from user to kernel back to userspace in one single pane of glass. Have you never heard anyone hype about DTrace on Solaris? This allows building similar things on production systems with little to no impact on running production applications.

seneca2• 6 years ago

The kernel has a "god's eye view" of everything happening on its OS. With event based tracing in the kernel there's no chance of missing an occurrence because your sampling rate is too low, for example. You can also correlate and enrich data that just isn't available in userspace.

justicezyx2• 6 years ago

A few things:

* Non intrusive: meaning one can snoop info of application without changing application code.

* Deep visibility: function level and syscall/kernel functions reveal more context and are more accurate in a lot of cases.

* Low overhead: everything runs inside kernel space, no context switching compared to other kernel based/aided tracing.

* Expressiveness: eBPF is fairly expressive, can do many things that usually are exclusive to high level programming languages.

lbotos• 6 years ago

At GitLab we've built and used strace-parser to help us get to quick and deep debugging, but running strace is EXPENSIVE: https://gitlab.com/gitlab-com/support/toolbox/strace-parser

We've been eagerly awaiting some customers to adopt newer kernels so we can start leveraging eBPF because of the performance gains in these type of scenarios.

Getting down the the kernel often can help find problems with disk access or network issues.

One of our staff engineers is exploring it now for NFS stats: https://gitlab.com/wchandler/tracing-tools/-/blob/master/nfs...

In Support Engineering we often straddle the line of 'SRE style stare at graphs and configuration as code' and 'log on to the box and look at syscalls'. We are very very excited about eBPF.

boston_sre873• 6 years ago

It's extremely quick to do it there as well. Originally, bpf was created to replace tcpdump with a quicker less impactful alternative. People saw that it was a pretty neat alternative so they started extending it (the e in eBPF). Think they might have finally come round again and just call it BPF now again.

morelisp2• 6 years ago

For hardware management maybe but so much of "observability" in general is at the application layer I can't see BPF displacing anything more than a tiny corner of it.

On the other hand - it already is overhauling service meshes, VPNs and firewalls and network security policies, etc. The stuff fly.io is debuting now is probably going to be standard in few years.

seneca2• 6 years ago

Strongly disagree. BPF uprobes allow extremely fine grained tracing of userspace applications, and allow you to programmatically correlate them with kernel level information.

ssssttt• 6 years ago

seems ambitious. why would a public company like new relic do this?

imukherjee2• 6 years ago

Co-Founder/CPO of Pixie here.

This definitely is an unprecedented and forward looking investment by New Relic. While ambitious, it became evident in our conversations with them they we committed on standardizing on open source telemetry standards such as Prometheus, Open-Telemetry, Graphana etc.

We're not yet in a position to speak for them in detail but we believe this bet on our project reinforces their plan to open-source telemetry layer to accelerate the adoption of observability practices by developers.

Hope that adds some color! would be great to hear more thoughts.

pkaye• 6 years ago

How does Pixie technology compare to Sysdig? Do they do similar things?

PoignardAzur• 6 years ago

At a guess, to commoditize their complement.

avinassh• 6 years ago

What does signing a definitive agreement mean?

iampims• 6 years ago

Pixie agreed to be acquired by NewRelic

boston_sre873• 6 years ago

Yea, the deal is not yet closed, but assuming no regulatory hold up (think new relic is public, so if pixie is worth enough, they'll need to go through HSR) or similar stuff, they agree that the acquisition will go through.

raulreyes• 6 years ago

Pixie scripts FTW.

bogomipz2• 6 years ago

I was not familiar with this project. In looking at the architecture[1] documentation it indicates it uses the Pixie Cloud. Is is possible to use this without the Pixie cloud?

[1] https://docs.pixielabs.ai/about-pixie/how-pixie-works/

zasgar4• 6 years ago

Co-founder/CEO of Pixie here.

In the current form Pixie Cloud is required since it's a freemium product.

We are planning to open-source a self-managed version of Pixie early next year, and it can be used completely self-hosted.

bogomipz2• 6 years ago

Thanks, I will be looking forward to this announcement as the current requirement would prevent me from using it. Cheers.

neelaj• 6 years ago

My take is that eBPF is such a powerful technology that it holds the potential to fundamentally change how Networking, Observability and Security are delivered. Just like Virtualization and now Containers/Kubernetes challenged an entire industry focused on big servers, the ability to safely embed programs into the Linux Kernel w/ eBPF challenges every incumbent. We're seeing every incumbent vendor look hard at the technology and make build vs. buy calculations. New Relic probably compared acquiring Pixie this early to either building internally or waiting to see how things developed for both the market and Pixie. I'm guessing they made a bet that the longer they waited the more expensive it would be to buy an eBPF based company. It was a good fit so they pulled the trigger, de-risking things for both themselves and the Pixie team (Note I have no inside info this is all just a guess).

daxfohl2• 6 years ago

I wonder if there's anything in the pipeline for storage.

reacharavindh• 6 years ago

I heard NVMe over Fabric was that thing that would transform storage in the future. Network storage near the latency and throughput of local storage, but with the flexibility and redundancy.

cobbzilla• 6 years ago

For those like me who had no idea what eBPF is — https://blog.pixielabs.ai/ebpf and https://blog.pixielabs.ai/ebpf-http-tracing/

sys_call• 6 years ago

Also https://ebpf.io/

nui_manako• 6 years ago

Don’t see the source code in GitHub. When do you plan to open it up? Interested in digging into your engine code.

zasgar4• 6 years ago

Co-founder/CEO of Pixie here.

We are preparing to open source early next year, so stay tuned. The code will be posted on our repository, that currently hosts our community scripts: https://github.com/pixie-labs/pixie.

suthakamal2• 6 years ago

How helpful is this to some/any of hybrid/multi-cloud and serverless? Analysing most of the telemetry at the edge seems like it could be helpful there. Useful for instrumenting low-latency edge applications?

smithclay• 6 years ago

This adds to some interestsing acquisition activity in this area: Splunk, for example, just acquired Flowmill (also focusing on eBPF + observability).

https://techcrunch.com/2020/11/24/splunk-acquires-network-ob...

[blatantly promoting my substack] Been following this area about a year, first wrote about some of startups using eBPF here in late 2019: https://monitoring2.substack.com/p/ebpf-a-new-bff-for-observ...

alfanick2• 6 years ago

I know what eBPF is, but what is Pixie? A single sentence describing Pixie wouid be useful.

In general, academic rule applies: Never write an article without defining terms beforehand.

nserrino2• 6 years ago

Engineer from Pixie here. Pixie is an APM tool for Kubernetes leveraging eBPF for automatic data collection. You can use it to monitor and debug your application performance, without code or configuration changes.

alfanick2• 6 years ago

APM? /s

eatmyshorts• 6 years ago

You're being too modest. Pixie is a general purpose observability and monitoring tool, with especially strong attributes around APM.

hdz• 6 years ago

Any plans to bring this to Cloud Foundry?

atmosx• 6 years ago

For those who want to play with eBPF on kubernetes using an open source tool there is cilium & hubble[1], plug-and-play CNI & Network Observability tool.

[1] https://docs.cilium.io/en/v1.9/intro/

boston_sre873• 6 years ago

These iovisor guys write some cool stuff that uses bpf, like: https://github.com/iovisor/kubectl-trace

New Relic to open-source Pixie’s eBPF observability platform

Related Stories