Netflix FlameScope

https://medium.com/@NetflixTechBlog/netflix-flamescope-a57ca19d47bb

by brendangregg • 8 years ago

389 74 8 years ago

smaili• 8 years ago

Link to the repo -- https://github.com/Netflix/flamescope

From a quick glance, it's interesting to see both Flask and Node.JS be used within the same project repo. Is this becoming more commonplace?

mspier3• 8 years ago

Not Node.JS specifically, but just npm for dependencies and task running to build the React client from source. If you're not looking to build the client from source, the distributed client bundle should suffice and only Python is required.

wooly_bully• 8 years ago

Even if you're using flask, there are plenty of reasons to have a package.json or other npm resources in your repo. Webpack is a good example.

ProAm2• 8 years ago

I wish they wouldnt host this on Medium, the pop-ups and layout are maddening and it almost seems unprofessional for a giant company like Netflix to use Medium instead of its own blog platform on their website.

soared• 8 years ago

https://outline.com/hpckBV

Although you lose the images so not that helpful in this situation, but usually good for medium.

herrvogel-• 8 years ago

Yes, Medium has terrible reading experience. I found this extension (https://github.com/thebaer/MMRA) a while ago. You should try it.

oarsinsync• 8 years ago

I keep reading comments like this, but I don’t ever see any particularly glaring issues when reading from my iPad. Is it a desktop / chrome thing that iOS / Safari is automatically mitigated against, or are my standards just so low that I don’t notice the issues?

eggoa• 8 years ago

Some employers will block netflix, which is one reason not to put their technical blogging on their own site.

wpietri• 8 years ago

Nice. I had some search cluster optimization problems that this would have been very handy for. I laughed when I got to "Ah, that’s Java garbage collection," as that turned out to be one of the big contributors to variance in response time.

wkz3• 8 years ago

Wow, great visualization! Thanks for sharing the source.

A minor suggestion: For the first part of the demo i was thrown of by the fact that each second as laid out bottom->top. Maybe moving the time labels below the plot would provide a visual cue for the direction.

mspier3• 8 years ago

Can you open a issue/feature request on Github?

wkz3• 8 years ago

Sure thing:

https://github.com/Netflix/flamescope/issues/1

lscotte• 8 years ago

I think this is intentional, as the norm is for 0 on a Y-axis to be at the bottom and increase going up.

I can understand why it's confusing, and perhaps could be a UI option to reverse it, but I do think that's the reason.

brendangregg8• 8 years ago

Yes, that's the reason! It means we get a t=0 origin for both the x- and y-axis. But yes, we can also add UI options to reverse it and other things.

wkz3• 8 years ago

Yes, I agree with it being that way. I just think my brain parses the origin as being the corner that is closest to the "0" label. Therefore, if the labels where below the plot, there would be no parse error :)

zawerf• 8 years ago

What's the purpose of visualizing it as grid? I thought it might be good for debugging periodic events (e.g., every 16ms for rendering at 60fps) but it seems like this is fixed to one sec for one column.

For example in the flame graph in chrome's performance tab, you can select/slice/zoom/pan arbitrary time ranges. If you want the high level view you just zoom out.

nsarafa• 8 years ago

I think it's helpful for visualizing the seconds as it's a time based focused debugger.

Maybe open a pull request for a graph based view and we can A/B test ️

brendangregg8• 8 years ago

Supporting different y-axis ranges is something we can add, and I was already thinking of adding a 1ms range for visualizing low-level CPU cycle activities (which span nano and microseconds). I like the idea of 16ms for 60fps analysis, where every column is one frame. Could help game development. :)

saulrh• 8 years ago

Actually, seeing the diagonal stripes in the examples [1], maybe it'd be useful to have a mode that lets you rescale the axes to line the stripes up? If you're using this to inspect patterns, it seems to me that the most useful time scale would not be a nice round human number, it'd be one that matches well with the patterns you're trying to inspect - one period, not one second.

Way out in the distance, maybe even a variable-interval mode, where you can click-select blocks and each one is given its own column, so you can directly compare them without the dead space in the middle and starting from the same baseline?

[1] Particularly striking: https://cdn-images-1.medium.com/max/2000/1*EypUnkPtayeKr9bJu...

kbenson2• 8 years ago

one second per column, fractions of a second per row, individual cells are fractions of that column's second.

In your example, I imagine you might see a periodic color shift along the Y axis as things aligned with that 16ms window. The example grid looks to have about 50 rows, so that would be ~20ms each, which is a bit too large for your use case. The example usage shows that you get to define the Hertz is samples at though, so you should be able to define exactly the period length you desire.

unictek• 8 years ago

Can this be used with Node.js CPU Profiles?

mspier3• 8 years ago

Depends on what you mean by Node.JS CPU profiles, if captured by `perf script`, then yes, otherwise, not yet.

brendangregg8• 8 years ago

Yes, I've already used it on Node.js profiles. :) Get any instructions for doing CPU flame graphs with Linux perf (I've written several), and take the output stage of "perf script" and you can browse it in FlameScope.

IamNotAtWork• 8 years ago

What was the root cause of the problem that motivated this visualization tool?

brendangregg8• 8 years ago

It's described in the blog post under "Origin", and the sample profile that we ship with FlameScope (examples/perf.stacks01) and which I used in the video is the first minute of the original problem.

It was intermittent latency that was hard to explain using existing tools.

coding123• 8 years ago

Pretty awesome - I recall a long time ago a few companies selling software that would do similar (but different) things, so this can help us get closer.

cortesi4• 8 years ago

This is lovely. An interesting thing to think about here is the layout of the sequential time-series data. This visualisation breaks the sequence out into columns, which means that squares that are close to each other in the sequence don't cluster visually.

I've played with using space-filling curves for this to visualise binaries:

https://binvis.io/

You can play around on there with the scan layout (similar to the Netflix layout) and compare it to the Hilbert curve. You can click and drag to select regions in both (similar to the video for this visualisation).

There are tradeoffs - the Hilbert layout is not intuitive, and you have to play around to select regions accurately. On the other hand, the clustering lets you visually pick hotspots in much greater detail, e.g.:

https://binvis.io/#/view/examples/tcpview.exe.bin?colors=ent...

I've found it useful to be able to toggle between both scanning and clustered layouts, and plan to explore some more interaction paradigms when I get time to turn back to the project.

brendangregg8• 8 years ago

I did binary visualizations many years ago too:

https://github.com/brendangregg/Dump2PNG/wiki

So with the subsecond-offset heat map, you get clustering from one column to the next for events that happen at the same time offset. Which is common with periodic tasks: see the example patterns in the post.

cortesi4• 8 years ago

It looks like we've traveled a similar path. The precursor to binvis is scurve, a (slow, dodgy) command-line tool to do basically what dump2png does:

https://github.com/cortesi/scurve

I have a blazing fast command-line version of binvis in Rust that I plan to release in the next month or so. I really need make some time to turn back to this.

kapitalx• 8 years ago

Longer video with more examples: https://www.youtube.com/watch?v=gRawd7CO-Q8

throwaway847425• 8 years ago

I’ve done something like this at Microsoft in 2002. Except in my case it was a multi-user system and each row represented a logical user, or a range of users in a range partitioned space. You could vary granularity, click through to see what’s busted, zoom into user ranges, discover correlated failures, discover cases when something is failing for just one user consistently, correlate all of it to performance counters, etc, etc. No one gave a shit.

mistermann• 8 years ago

What did you use to build it?

throwaway847425• 8 years ago

IIRC the standard MS stack at the time: C#, ASP.NET (which had just come out), MS SQL, IE, some in memory caching and bitmap rendering (canvas didn’t exist), AJAX. Pretty bleeding edge, but it all worked OK.

lma21• 8 years ago

Haven't followed up on this lately — is sampling java stacktraces with perf going well? what happened to the frame pointer issue? do we still have to use -XX:+PreserveFramePointer to get the full stack?

Awesome tool nevertheless. Always a pleasure to go through Brendan's utilities regarding performance analysis!

brendangregg8• 8 years ago

Yes, perf sampling of Java is still going well, via -XX:+PreserveFramePointer. Our one microservice that had high overhead with that option (up to 10%, which is rare) has improved their code, lowered the overhead, and now enabled this option by default.

But there's also a newer way that can do Java stack sampling from perf without the frame pointer: https://github.com/jvm-profiling-tools/async-profiler

We're not running it yet. I want to try it out. Note that the stacks with async-profiler are a bit broken -- Java methods become detached from the JVM -- but I'm hoping that's fixable (it should be with a JVM change, at least).

gregdunn2• 8 years ago

Awesome!

It'd be pretty sweet if you could send things sampled using Vector over to FlameScope. I know the flamegraphs PMDA is experimental but it's always worked fine for me :)

(It'd also be pretty sweet if there was a fleshed out PMDA for bcc tools. And Vector supported them. And, and, and... I have a really long wishlist.)

brendangregg8• 8 years ago

We'll get it done (although I'm not exactly sure who "we" will end up being yet!). I improved the flamegraph PMDA late last year, but how FlameScope works is really the direction Vector should go: being able to make an API call to get the flame graph stacks as JSON, and then render them in d3, along with buttons for transforms.

jsmeaton• 8 years ago

Does anyone have any good tools or processes for generating the kind of performance counters and information needed by this tool with a running python application?

I'd love to be able to generate a flamegraph for a django app, for example, and visualise it with flame scope.

https://mail.python.org/pipermail/python-dev/2014-November/1... discusses the problems with python and the `perf` tool. There may have been some patches applied recently to help, but it doesn't look like there was a full resolution.

gnufx4• 8 years ago

Not a direct answer, but there's Python support of some sort available for at least TAU <https://www.cs.uoregon.edu/research/tau/home.php>, Extrae <https://tools.bsc.es/extrae>, and Score-P <https://github.com/score-p/scorep_binding_python>.

akx• 8 years ago

I wrote https://github.com/akx/abyss for Django once upon a time. I'm not sure it works these days, but you could give it a shot.

itissid• 8 years ago

a) could not reliably capture a one minute flame graph covering the issue, as the onset of it kept fluctuating; b) capturing a two or three minute flamegraph didn’t help either, and the issue was “drowned” in the normal workload profile. Vadim asked me for help.

One interesting thing here is also the statistical nature of this problem: i.e. there are spikes but their duration are not exact or known when to start. Here[1] is some reference work done on Neural Spike Trains that seems relevant.

[1]http://www.stat.columbia.edu/~liam/teaching/neurostat-spr12/...

sandGorgon• 8 years ago

Given the existence of tools like this or https://github.com/uber/pyflame, I get really amazed at the monopoly of NewRelic. Nobody seems to be even attempting a start-up in this space, which is amazing given the amount of money that everyone coughs up for NewRelic.

hie2neiL2• 8 years ago

All the native linux thread stack visualizers I have used so far lack a way to show single-threaded bottlenecks in multi-threaded applications. Does this solve it?

In the java ecosystem this is solved with utilization timelines.

gnufx4• 8 years ago

I don't know which of the normal parallel performance tools support Java, but at least TAU <https://www.cs.uoregon.edu/research/tau/docs.php> does; not that I've worked on Java, which should preferably be kept well away from compute nodes.

hie2neiL2• 8 years ago

Oh, my question wasn't about java. Quite the opposite. I'm missing the perf equivalent of some java tools I am accustomed to.

gregdunn2• 8 years ago

Not directly related, but all of your posts on this subject contain broken links - please add spaces between your link and your delimiter, if you are going to use a delimiter, as doing <link> makes the > part of the link, requiring manual removal.

petters• 8 years ago

I know I am missing something, but passing -F 49 to perf will tell the kernel to adjust the sampling interval dynamically so approximately 49 events/s are recorded, right?

But then how does it make sense to plot the number of events per second in the overview? Or what exactly is being plotted in the overview?

Edit: Perhaps it is simply that idle events are discarded and multiple CPUs allow more than one event in the period?

gnufx4• 8 years ago

What's the advantage of this sort of thing over the trace and profile analysis tools that have been developed at least for HPC performance engineering over many years? (E.g. the BSC tools <https://tools.bsc.es/> amongst several others.) In particular, why are flame graphs preferred to the normal inclusive/exclusive views of profiles which seem more generally useful to me?

Netflix FlameScope

Related Stories