Poop: Performance Optimizer Observation Platform

https://github.com/andrewrk/poop

by davikr • 3 years ago

130 43 3 years ago

github.com

AndyKelley• 3 years ago

heh, I didn't expect to see poop on the front page of hacker news.

If you're curious about the poop-making process, I created it and released it all on one 6 hour twitch stream [1].

[1] https://www.twitch.tv/videos/1845437356

devsda2• 3 years ago

Even with the context, this comment sounded so intriguing and wrong at the same time that I hesitated while opening the link.

Watched a few mins and bookmarked it for another day. There were many other name suggestions that are good but I'm glad you kept the initial name.

notThrowingAway• 3 years ago

Note that twitch VODs don't stay up forever, Andrew did mention that he would upload to peertube, but I don't know if he did.

UniverseHacker• 3 years ago

Does this need a [NSFW] tag?

DANmode• 3 years ago

Twitch?!

pkphilip• 3 years ago

I know how to make poop, bro. Been doing it for a long time now.

dundarious2• 3 years ago

Kindof off-topic, but has anyone had any success in using perf_event_open from within WSL, or is that unsupported? I tried `sysctl kernel.perf_event_paranoid=0` and the following, which works under "real" linux, but fails under WSL:

    struct perf_event_attr attr = {
        .type = PERF_TYPE_HARDWARE,
        .size = sizeof(struct perf_event_attr),
        .config = PERF_COUNT_HW_INSTRUCTIONS,
        .disabled = 1, /* only need to read once, not updating values, but fails with 0 also */
        .exclude_kernel = 1,
        .exclude_hv = 1
    };
    int fd = perf_event_open(&attr, 0, -1, -1, 0); /* fails, returns -1 */

Microsoft's kernel appears to have the right CONFIG_* as well, so I feel like it must be supported in some capacity, even if limited.

birdiesanders• 3 years ago

I'll be making poop part of the CI development process starting Monday.

cmehdy• 3 years ago

Makes sense, it's for stuff nobody likes dealing with but everybody has to do.

warent• 3 years ago

Not to be rude but this source code... makes me wonder if I ever want to use Zig. It looks barely readable

https://github.com/andrewrk/poop/blob/main/src/main.zig

brosciencecode• 3 years ago

Out of curiosity, what makes it unreadable to you?

slekker• 3 years ago

Not OP but to me it is a bit difficult to read due to the syntax highlighting colours (mainly red and brown)

dinglejungle• 3 years ago

This might be more of an indictment on 'readable' being a meaningful concept (beyond simple personal preference) than it is of Zig's syntax.

proto_lambda• 3 years ago

I've never touched zig and know next to nothing about it, but that all looks reasonable to me at a glance. What exaxtly do you find so bad?

LanternLight83• 3 years ago

I've never used Zig or C++, mostly Bash, Python, and Lisps, but have recently began learning C. I see C++ in search results, and it looks like C with higher level features in a register that reminds me of Java. I see this, and it looks similer: C with higher level features that remind me more of Python. I'm sure it's got other important features and priorities that set it apart (oft' discussed here), but that's what I see, and I dig it.

To be clear there are definitely a few ideoms I'd need to learn before considering this readable (it ain't psudocode), but I generally see all the building blocks and recognize what they're achiving. Maybe I'd have put them together slightly differently, but that could be equally true in Python. I'd rather learn this next than C++.

The density of how it's been written reminds me of some lisp code I've read, some implementations of state machines can be hard to follow too (in the classic "but where does it do the thing?" sense). I attribute a lot of the stuff that I don't follow to the lower-level C influence; I'm working for the first time with memory managment, structs, static types, pointers, bitwise opperations, etc.

Think I made my point in there somewhere >z>

Would love to know where you're coming at it from, I'd imagine that the C or 'kinda functional control flow' aspects might explain what looks most forign to you too (unless you have broader experience with low level languages)?

lionkor• 3 years ago

Youre mostly right, but Zig is not higher level than C. In some ways, its lower level, as in C the allocator you use is mostly abstracted away, while in zig its usually required to be specified.

TeMPOraL• 3 years ago

Looks fine to me. Syntax is the run-of-the-mill post-Java style currently favored, as exemplified by e.g. Haskell or Rust. Not a fan personally, but this is what is popular now, so that's what we have to live with ¯\_(ツ)_/¯.

Style-wise, looks fine. Code split up into functions where it makes sense, inlined where it doesn't. I've seen worse - for example, code that takes the equivalent of that, and splits it into 100 functions across 10+ files.

wiseowise2• 3 years ago

I've seen worse - for example, code that takes the equivalent of that, and splits it into 100 functions across 10+ files.

This so many times. Will never understand why people are so afraid of multi thousand lines of properly structured code but will happily giggle when the same structure is split across 10-20 files. And no, it’s not for reuse sake.

j-pb• 3 years ago

Nobody should do benchmarking like this anymore.

Memory layout of your code has such a big impact on performance on modern computers that measuring performance without removing that variable leads to wild goose chases, where you think you improved something, but in reality you incidentally got the compiler to move the code around a bit.

Emery Berger has an excelent talk on this [1], and a causal profiler that they developed called Coz[2].

Branch miss-predicts might be somewhat invariant to that, but still, one of the main points of the talk is that people do too much eyeball statistics, mistaking the variance of the underlying stochastic process with actual signal.

Coz is pretty trivial to set up with Zig too.

1. https://m.youtube.com/watch?v=r-TLSBdHe1A

2. https://github.com/plasma-umass/coz

ororroro• 3 years ago

Sometimes you just need to check Poop to make sure your apps still healthy.

krona• 3 years ago

Causal profiling still uses hypothesis testing, so I don't get your point. Ultimately, one needs to know if software A is 'better' or 'worse' than software A' for any given definition of those terms.

adrian_b• 3 years ago

Coz appears to use a very good method for profiling multi-threaded applications, but which seems to be not applicable to single-threaded applications and I do not see how it would help to avoid the variability caused by different memory layouts, which, as you rightly point out, may confound profiling results.

dundarious2• 3 years ago

Yes, just because it's from a team connected with the old Stabilizer tool, it doesn't mean it shares the insights and benefits of that kind of testing. At least, that's my understanding based on my limited knowledge of coz, and a little bit more knowledge of https://github.com/ccurtsinger/stabilizer

MattPalmer1086• 3 years ago

What a great talk.

I've been working on algorithm benchmarking software, and I observed that small changes to algorithm code, even in parts that would not execute, would have a large effect on execution speed.

Even better they seem to have a solution to the problem. Going to explore further to see how we can use this technique.

CrampusDestrus• 3 years ago

Poop: it's the shit!

MeteorMarc• 3 years ago

The name poop does not hold in the intercultural sense. Indeed, in northern dutch poep means shit. In southern dutch poepen means having sexual intercourse.

sverhagen• 3 years ago

Southern Dutch as in the North of Belgium, right?

thawab• 3 years ago

Fun fact: the platform is built using zig, which means poop in Arabic.

yashBonde• 3 years ago

It's just not working, I `poop`ed all day long!

ftxbro• 3 years ago

    > "The MIT License (Expat)
    Copyright (c) Poop Contributors
    Permission is hereby granted..."

aprasadh• 3 years ago

should I use fork or spoon the poop? how I do merge my changes to poop? I don't want to make a pull request.

nine_k• 3 years ago

A perfect way to prevent adoption in larger and more serious companies, but become poopular among younger audience.

65102• 3 years ago

how young? I'm having a hard time containing my inner 5 year old. Apparently he has a poopulence of jokes ready for the making.

mcdonje• 3 years ago

A company refusing to use the most appropriate tool because of the name is much more offensive to me than a tool with a childish name.

devsda2• 3 years ago

Slim chance that it really gains wider adoption and becomes a standard tool but it will be an interesting conversation starter for sure.

Imagine you are struggling with performance of some tool and a more knowledgeable senior comes in and says "you need to use poop" or "just poop it!".

65102• 3 years ago

Who knows, peristaltics pushed it to the front page already.

mpweiher• 3 years ago

Big POOP by Sam Adams

http://www.youtube.com/watch?v=GBtqQwcJoN0

pacoverdi• 3 years ago

(phonetically) related: weboob

https://pypi.org/project/weboob/

matheusmoreira• 3 years ago

Such a cool project. Essentially a collection of custom HTTP clients for every supported web site.

quesera• 3 years ago

They've apparently renamed to Woob: https://woob.tech/

Woob (Web Outside of Browsers) is a library which provides a Python standardized API and data models to access websites.

Interesting idea -- basically an API on top of a passel of site-specific web scraper modules.

sirwitti• 3 years ago

Upvoted just for the name and acronym!

Poop: Performance Optimizer Observation Platform

Related Stories