Introducing Vector: Netflix's On-Host Performance Monitoring Tool

http://techblog.netflix.com/2015/04/introducing-vector-netflixs-on-host.html

by r4um • 11 years ago

184 62 11 years ago

techblog.netflix.com

strunz• 11 years ago

Anyone care to compare this to something like collectd? https://collectd.org/

mspier6• 11 years ago

Goal is a bit different. Vector doesn't collect and persist metrics. We needed something that had as little overhead as possible so it could be deployed to all our hosts and simplify the process of analyzing those metrics.

toomuchtodo3• 11 years ago

If its not collecting and persisting metrics, is it more of a glorified htop?

falcolas• 11 years ago

Neither does collectd, it just gathers and transmits the data to an endpoint you specify. It's also quite lightweight (once you address the memory leaks, at least).

debaserab2• 11 years ago

What does this have that CloudWatch enhanced metrics doesn't? From the screenshots, the metrics look pretty similar. Not a slight at all against this project (it looks awesome), I'm just curious if your infrastructure is already AWS based what would cause you to choose a non-CloudWatch option.

toomuchtodo3• 11 years ago

CloudWatch charges you per instance if you want 1 minute metrics instead of the standard (free) 5 minute metrics. Any tool that collects the data for you gets you out of that $3.50/instance/month detailed monitoring charge [1].

[1] http://aws.amazon.com/cloudwatch/pricing/

brendangregg4• 11 years ago

It's one second metrics, and has been designed to allow us to add other metric sources and visualizations (using, say, ftrace, perf_events, SystemTap, etc), although those are not in the initial release.

thieving_magpie• 11 years ago

It looks like it has some metrics that aren't present in cloudwatch without a lot of scripting, such as Memory utilization. I'm new to cloud watch though, so someone please correct me if that is wrong.

philsnow• 11 years ago

I got down to

     you should be able to install PCP from binary packages made available by the PCP development team on:

    ftp.pcp.io

and that threw up a red flag in my brain. Then I noticed that techblog.netflix.com doesn't redirect to https (and indeed can't serve https) (I use https://www.eff.org/HTTPS-EVERYWHERE and did not get content over https).

The directions _I_ saw for building from source looked pretty innocuous, but you might see a different set of directions if you're being MITMed. Observe an appropriate amount of caution.

mspier6• 11 years ago

Happy to discuss your concerns. PCP 3.10 should be available on Ubuntu's official repo pretty soon too.

harshreality• 11 years ago

His concerns seem plain to me. Unauthenticated channels for software distribution or software installation instructions are bad.

The techblog isn't using SSL, and the git pull url for PCP is using the git protocol which is also unauthenticated, rather than the authenticated https transport (ssh is only an option when user accounts make sense).

Someone's at a conference and follows the link over public wifi. They get the same page but with "here's how to get PCP: ftp evil.io or git clone git://git.evil.io/pcp" Even if the webpage were ssl-enabled so that an attacker can't rewrite the pcp.io links, an attacker or evil network operator could MITM git.pcp.io or ftp.pcp.io. (FTP?!)

Being in Ubuntu's repo doesn't make it safe if Ubuntu's maintainers have no (semi-)trustworthy way of getting the code.

mrmondo• 11 years ago

Any chance you could provide standard Debian builds rather than Ubuntu specific?

nathan_scott4• 11 years ago

To get the sources, you can use non-ftp/non-git access methods via the github mirror - https://github.com/performancecopilot/pcp

capkutay2• 11 years ago

The charts look like they were built with nvd3. Can anyone confirm/deny?

mspier6• 11 years ago

That's right. Any suggestion of better reusable charts?

capkutay2• 11 years ago

IMO nvd3 is one of the better high-level charting libraries in terms of number of charts, out-of-the box styles and interactivity.

forrestthewoods• 11 years ago

Honestly? Just write your own SVG code. If you inspect the elements the output data is super simple and easy to understand. NVD3 just wraps D3.js which just wraps utilities that output relatively basic data. Well, d3js is a data-binding system that's way too god damned complicated if all you want to do is make some simple charts which is how it's used 99% of the time.

I've spent more time trying to manipulate chart libraries into doing almost the same thing but just different enough to cause pain and suffering. Output your own path data and it's a million times easier.

For reference here's what I made: http://forrestthewoods.com/unbalanced-design-of-super-smash-...

SEJeff2• 11 years ago

It isn't d3 based (uses canvas instead of svg), but flot is fantastic as well:

http://flotcharts.org

rubiquity2• 11 years ago

Am I wrong in saying this is a web interface that wraps PCP? So you can't really compare it to inspeqtor or collectd since those actually do the metrics collection.

mspier6• 11 years ago

True. But that's the first release of Vector. We expect to make our custom PCP agents public soon.

rubiquity2• 11 years ago

Sorry I should clarify I was making that statement for my understanding, not to degrade the project!

oimaz• 11 years ago

Where can I find deb packages for pcp version 3.10 or higher?

lscotte2• 11 years ago

See http://pcp.io/docs/installation.html#deb or direct link at ftp://ftp.pcp.io/projects/pcp/download/deb/ - but I only see packages for i386 arch, so you may have to compile from source if on amd64 or other architecture.

jmedefind• 11 years ago

The git repo has a MakePkg script that will generate deb packages for you.

I found it really easy to use.

xbryanx• 11 years ago

I'm curious how this compares to Zabbix's agent and server. Does PCP give you finer grained details, or is it possibly more lightweight?

nathan_scott4• 11 years ago

PCP is much finer-grained than Zabbix in terms of the metrics it makes available (esp. from the Linux kernel); not sure on Zabbix costs but PCP is quite light on all resources (mem, cpu, net) and very robust.

I've worked on production systems where everything else was failing (hardware, kernel, applications) but PCP kept chugging along, recording and telling the sad story to anyone that would listen.

tdicola• 11 years ago

Wow this looks great, thanks for releasing it! Nice to see even 'simple' stuff like this that can help people who aren't running at Netflix scale is still released.

CWuestefeld2• 11 years ago

It's enormously frustrating to me that the Windows platform is so far behind.

I've been researching for a rebuild of our ecommerce site with the idea of a modern microservice architecture. Team skillset and some other considerations dictate a .Net environment.

Netflix and other companies have created a really rich platform, between logging and monitoring technologies like this; message queueing; deployment; and so on. In the Windows environment there's precious little to compare to.

One bright spot is the news story also on HN today [1] about MS announcing Docker-related Hyper-V technology - but in the next version of Windows server.

It might be that to satisfy those .Net compatibility wishes, we just go to Mono and do everything else in Linux.

[1] https://news.ycombinator.com/item?id=9342369

voltagex_• 11 years ago

I suspect that you could extend PCP (the monitoring component of this dashboard) to support Windows' performance counters. The profiling capabilities are certainly there.

nathan_scott4• 11 years ago

PCP can export many hundreds of Windows-specific counters (which it accesses via the PDH APIs). The Windows build does need some love though, as the OP mentioned - its out of date and needs a regular contributor to bring it up to date with the latest PCP sources.

Then Vector should Just Work (tm) for Windows servers too.

tonyarkles• 11 years ago

Team skillset and some other considerations dictate a .Net environment.

Not to be a dick but... that doesn't sound like a Windows limitation.

dkhenry• 11 years ago

So I haven't looked into all the metrics that they are getting off of PCP but from an initial glance it doesn't look like they are getting a lot more information then you get off net-snmp.

In this one case the windows platform isn't going to hold you back too much. On top of that anything the .net stack on windows gives you access to WMI which can give you really simple really deep data from your application.

Don't buy into too much hype on what some of the big guys are doing on monitoring, most of their challenges seem to be around the scale they are monitoring at not with the depth of metrics.

brendangregg4• 11 years ago

For the initial release, you are right that you'll find most of these metrics in net-snmp. It's the selection that's more interesting, which is guided by the USE method: identifying utilization, saturation, and error metrics for key resources. We're missing some metrics (so do all monitoring tools), and we'll be extending Vector to include them (as can other people, as it's open source).

One challenge is that there are some metrics that the kernel doesn't make available, so we need to either use tracers to fetch them, patch the kernel, or use custom kernel modules. In recent weeks, I've been using (or trying to use): ftrace, perf_events, SystemTap, eBPF, and LTTng. I've also used others, but not in the last few weeks. I'm really looking forward to getting these advanced tools into Vector.

yareally• 11 years ago

Netflix and other companies have created a really rich platform, between logging and monitoring technologies like this; message queueing; deployment; and so on. In the Windows environment there's precious little to compare to.

Have you looked at any of the following for the various gripes you mentioned? I came from a primarily Linux background until recently, but I know .net and Windows have good tooling that extends beyond my modest list below. Looking to Microsoft to provide every option is probably not the best way to go about it, not sure if that was mostly what you were looking at so far.

https://www.rabbitmq.com/install-windows.html (company I work for uses it for messaging between their services)

http://nlog-project.org/ (we also use that as a base framework for our logging)

http://getakka.net/ (never used it on .net but can't do without it in the jvm)

https://github.com/dotnet/Orleans (sort of like Akka, but with additional features)

https://github.com/MSOpenTech/Redis (it's a bit behind the nix version, but still well maintained)

https://msdn.microsoft.com/en-us/data/gg577609.aspx (Netflix uses the Java port of that, fyi)

https://technet.microsoft.com/en-us/sysinternals/bb896653.as...

http://en.wikipedia.org/wiki/Resource_Monitor

https://www.jetbrains.com/dotmemory/

https://www.jetbrains.com/profiler/

http://www.appveyor.com/ (great for CI and easy to set up)

https://www.cygwin.com/ (when you miss a nix shell and tools)

http://en.wikipedia.org/wiki/Windows_PowerShell (for everything cygwin can't do easily)

wantab2• 11 years ago

The web was built, and runs on, *nix/BSD. Windows is an outlier. Windows can't even get the slashes going the right way. There's a reason 80% of internet traffic does not use Windows.

SEJeff2• 11 years ago

Likely more than that seeing as how the majority of the load balancers and many of the routers / switches out there run some form of BSD (netscalar) or Linux (Arista), etc.

CWuestefeld2• 11 years ago

I'm not sure how that was supposed to be helpful.

I also don't think it's strictly true. For sure the underlying networking came from Unix. But as for the web itself, once we got to real dynamic content, it seems to me that Microsoft were the ones that got things moving.

While the Unix world was mired in the awful world of CGI, Microsoft gave us high-performance ISAPI, and then Cold Fusion (also on the Windows platform) and Microsoft's ASP made programming a little more sane. While the Unix world tried to deal with JSP (which IMHO wasn't a very good solution), the Microsoft platform seems to have been the innovators for several years, until Ruby on Rails and then node.js and stuff started coming out.

Today, the Apache server powers far more sites than any other, it's true. But IIS shares the 2nd-place spot [1]. When you say "There's a reason 80% of internet traffic does not use Windows", that's pretty much true for the servers, but far off the mark when counting clients. And the reason for that is that Microsoft's strength hasn't historically been radical advances, but in figuring out ways to take the bleeding edge tech that doesn't really work quite right yet, and packaging it into commodity software that may not be as sexy as envisioned by those with the original ideas, but actually useful to the average guy.

[1] http://www.zdnet.com/article/web-servers-microsoft-iis-and-n...

EDIT - missing word "world" in 3rd para

AloisMayr• 11 years ago

Want to try https://ruxit.com for monitoring your .NET ecommerce shop on Windows? It also monitors your host and real user interactions with your site.

Thaxll• 11 years ago

Not sure what the difference is between that and Collect + Graphite / InfluxDB.

Florin_Andrei• 11 years ago

Make that Sensu + Graphite and you get fault detection and alerts too.

easytiger• 11 years ago

Yea, but i guess they connect directly to the daemon on each host so no intermediate db. Which sounds like a terrible idea

nathan_scott4• 11 years ago

There are many, many differences - there's some books about PCP that might help clarify PCP design points, see: http://pcp.io/documentation.html

vizzah• 11 years ago

Could anyone who tried this monitoring tool compare it to Munin?

fapjacks• 11 years ago

I just tried installing this on a test VPS. PCP went well, but whenever I tried to input a "hostname" in Vector's UI to get stats, it just kept telling me it couldn't connect and to check hostname, regardless of what I put in the box. PCP was available and of course ports open and whatnot. I'm not sure what the problem was, but that was a bit frustrating. It looks like a slick product otherwise, especially for a first version! Thanks for releasing projects like this! I'll be trying again in the future, for sure.

mspier6• 11 years ago

Can you open an issue on GitHub and post more details? Any errors on the JavaScript console?

victorhooi• 11 years ago

This is a slight tangent...but does anybody know what UI toolkit Netflix is using for this?

Or if it's in-house, any info on whether they have, or might release it?

I see bootstrap-submenu.css mentioned, but not Bootstrap itself:

https://github.com/Netflix/vector/tree/master/app/css

mspier6• 11 years ago

It's bootstrap (see bower.json dependencies), but with our own layer on top of that.

samstave2• 11 years ago

After using Stackdriver for the last 18 months, I would never go back to rolling my own monitoring infra if I could avoid it. I had nagios, cacti, munin, graphite all running and had two ops guys pretty much 80% of their time managing it.

Stackdriver with pagerduty and I have 250,000 custom metrics being published and hundreds and hundreds of graphs on dozens of dashboards.

Although, I am looking at SignalFX to give even better version of this, but I manage nearly 1,000 machines with only a staff of four.

makeitsuckless• 11 years ago

Never heard of it, so checked out Stackdriver. Seemed very interesting, especially since it's geared at AWS (which we use).

Until I noticed the big red flag on the front page "Stackdriver is now part of Google."

Surely Google is going to kill this product and use the knowledge to offer something similar for Google's cloud services.

samstave2• 11 years ago

We were concerned when.Google bought them and we had many meetings with them about this, I do not believe that will happen, but if that is still an issue for you, use signalFX.

Introducing Vector: Netflix's On-Host Performance Monitoring Tool

Related Stories