Packetbeat: Open-source application monitoring

by mobiplayer • 12 years ago

89 23 12 years ago

aba_sababa• 12 years ago

An open source Boundary?

packetbeats5• 12 years ago

I haven't tried Boundary in a while, but I think they are focused on metrics. This indexes the transactions, so you can later search and do analytics on them.

ianmcgowan• 12 years ago

An open source Splunk?

Terretta• 12 years ago

Maybe Kibana + ElasticSearch + fluentd

lewaldman• 12 years ago

This would be kibana+elasticsearch+logstash :P

sdfjkl• 12 years ago

The output looks a lot like what New Relic does, but the capture process is entirely different: This seems to sniff network traffic, while NR wraps your application code.

greenyoda• 12 years ago

"Packetbeat agents sniff the traffic between your application processes, parse on the fly protocols like HTTP, MySQL or REDIS and correlate the messages into transactions.

For each transaction, the agents insert a JSON document into Elasticsearch where they are stored and indexed."

How much performance degradation could be expected from constantly inspecting every packet on your network with libpcap, correlating the packets into transactions and then storing information on every transaction in Elasticsearch?

Also: I noticed that the link to Elasticsearch on the "Getting Started" page is broken (it's missing the ":" after "http"). Here's a working link: http://www.elasticsearch.org

deeviant• 12 years ago

This is just a guess, but it shouldn't impact your performance at all. It's just a passive network capture mechanism like wireshark. It will need resources to do all the logging/categorizing/displaying and such, but the rest of the production environment should be unaffected.

packetbeats5• 12 years ago

Depending on the traffic, the CPU usage of the agent could indeed be significant. We're thinking to implement sampling to reduce the effect of this. On the other hand, the agent is not "in-line" so it cannot introduce any latency, only steal CPU time from them.

So, unless you have a single CPU core, it's pretty safe to run in production. It cannot affect the application performance by much.

Thanks for noticing the broken link, we fixed it.

zentrus• 12 years ago

What happens if more traffic is coming in than Packetbeat can process? Either you are dropping that traffic or you are (perhaps unintentionally) blocking it, right?

pyvpx• 12 years ago

you might look into sFlow and/or IPFIX.

milkshakes• 12 years ago

nobody said the monitoring service has to run on the same machine as the application. you could mirror the app server port and feed the traffic into a completely different monitoring machine.

SEJeff2• 12 years ago

I wonder if this would work with PF_RING[1] from the ntop guys? Regular libpcap drops packets quite a bit at 10G+ speeds when you have adaptive coalescing disabled on the network card.

[1] http://www.ntop.org/products/pf_ring/

packetbeats5• 12 years ago

PF_RING is an option but it requires a kernel module and specific network cards. Before then, the memory map sniffing feature of the Linux kernel [1] can improve libpcap performance. Not quite to 10G speeds, though, but I think enough for most cases :-)

[1] http://www.cs.fsu.edu/~baker/devices/lxr/http/source/linux/D...

SEJeff2• 12 years ago

Any plans to support it? Part of my job (historically) has been building line rate pcap appliances that do pcaps at nanosecond resolution (using endace cards, Mellanox, or solarflare). PF_RING is a bit tricky to get working, but is completely flawless when you get it running.

This software looks fantastic however.

knodi• 12 years ago

Wow such a brilliant idea. Can't wait to try it.

Nate75Sanders• 12 years ago

I believe http://www.extrahop.com/ is a commercial service/product in this space.

This looks very cool, though, and I want to play with it.

steedsofwar• 12 years ago

This looks interesting. I'm actually looking forward to trying this on a game server i'm working on at the moment. Thanks for making this freely available!

mobiplayer• 12 years ago

Hey guys, I'm the one that sent the link to HN, but I'm in no way related to the Packetbeat gals/guys, but I was so excited about this that couldn't help myself :) congratulations to the devs, this is amazing!

I've been lately playing with logstash+elasticsearch+Kibana and I think this is could be huge combined with Packetbeat agents. Can't wait to see what the people come up with!

packetbeats5• 12 years ago

Many thanks for posting it! :)

nls• 12 years ago

This looks great! Would be awesome if Packetbeat received Postgresql support too.

ausjke• 12 years ago

what's new here comparing to snmp and all those oss monitoring solutions(nagios,etc)?

Packetbeat: Open-source application monitoring

Related Stories