maknee

Thanks! Looks like I have to request the whitepaper to take a look at the details.

How effective are LLMs at triaging issues? Has anyone found success using them to find the root cause? I've only been able to triage effectively for toy examples.

https://maknee.github.io/

Great paper! Love the easy to understand explanations and detailed graphs :)

Read through the comments. Added RSS and subscribe to my blog!

[1] https://maknee.github.io/blog/

It seems like the load_score serves a proxy for how much needs to be done. Is there a real value that could be used instead? The solution requires syncing with all of the GPU nodes anyways.

  Location: New York City, NY, USA (NYC)
  Remote: Yes
  Willing to relocate: No
  Résumé: https://www.linkedin.com/in/henry-zhu-347233121/
  Email: find email on my blog OR reach out on linkedin
  LinkedIn: https://www.linkedin.com/in/henry-zhu-347233121/
  Github: https://github.com/maknee

- Analyzed DeepSeek's distributed filesystem from the ground up in a multipart series

- Built a 100k+ (1TB+) RL dataset for a popular online MOBA game by reverse engineering data format (~50k -> ~100k downloads monthly on huggingface)

- Optimizing finetuning/RL on multi-node GPUs

- Built a framework that intercepts OpenGL pipelines and re‑renders them with NVIDIA ray tracing

I'm looking to work on optimizations and performance analysis of systems. This includes: building profiling tools, building and running large scale distributed systems (training, inference, vectordb, db, compilers, etc...), discussing napkin-math & identifying bottlenecks and writing about the work.

interesting results. why does reload/cross-tile have worse results? would be nice to see some examples of failed results (how close did it to solving?)

How does this compare against other DSLs?

Great to see another distributed file system open sourced! It has some interesting design decisions.

Have a couple of questions:

- How do you go about benchmarking throughput / latency of such a system? Curious if it's different compared to how other distributed filesystems benchmark their systems.

- Is network or storage the bottleneck for nodes (at least for throughput)?

  - From my observations from RDMA-based distributed filesystems, network seems to be the case.

- Very very interesting to go down the path of writing a kernel module instead of using FUSE or writing a native client in userspace (referring to 3FS [1])

  - Any crashes in production? And how do you go about tracking it down?

  - What's the difference in performance between using the kernel module versus using FUSE?

They're going to spend time and effort into making their optimizations public. Would you rather have them keep their changes internal?

Great work! It's nice seeing another observability tool. Demo is neat and easy to navigate.

Couple of questions:

What's the overhead of tracing + logging observed by users? I see many tools being built on top of the OpenTelemetry eBPF tracer, which is nice to see.

The OpenTelemetry eBPF tracer uses sampling to capture traces. Do other types of logging in the tool use sampling as well (HTTP traces)?

When finding SLO violations, can this tool find the bug if the latency spikes do not happen frequently (ie, latency spikes happens every 5minutes - 1hour)? I'm curious if the team have had experienced such events and even if those pmax latencies matter to customers since it may not happen frequently.

I see that the flamegraph is a CPU flamegraph - does off-cpu sampling matter (Disk/Network, etc...)? Or does the CPU flamegraph provide enough for developers to solve the issue?

Thanks! Those blogs are incredibly useful. Nice work on the profiler. :)

I have multiple questions if you don’t mind answering them:

Is there significant overhead to native unwinding and python in ebpf? EBPF needs to constantly read & copy from user space to read data structures.

I ask this because unwinding with frame pointers can be done by reading without copying in userland.

Python can be ran with different engines (cpython, pypy, etc) and versions (3.7, 3.8,…) and compilers can reorganize offsets. Reading from offsets in seems me to be handwavy. Does this work well in practice/when did it fail?

"All of this is made possible with the inclusion of frame pointers in all of Meta’s user space binaries, otherwise we couldn’t walk the stack to get all these addresses (or we’d have to do some other complicated/expensive thing which wouldn’t be as efficient)"

This makes things so, so, so much easier. Otherwise, a lot of effort has to built into creating an unwinder in ebpf code, essentially porting .eh_frame cfa/ra/bp calculations.

They claim to have event profilers for non-native languages (e.g. python). Does this mean that they use something similar to https://github.com/benfred/py-spy ? Otherwise, it's not obvious to me how they can read python state.

Lastly, the github repo https://github.com/facebookincubator/strobelight is pretty barebones. Wonder when they'll update it

Does anyone have blogs with benchmarks to show the performance of running smallpond let alone 3fs + smallpond?

A lot of blogs praise these new systems, but don't really provide any numbers :/

What are the results from running smallpond and 3fs?

Are claims valid for <10TB, 10TB -> 1PB and over 1PB?

You can't watch a replay from the league client that doesn't match the same version.

There's hundreds/thousands of generated one way decryption schemas for fields. However, it's not impossible to generate the decryption in another language with some effort.

Example:

A packet could be decrypted like this (the actual decryption takes more steps than this)

field1 = LOOKUPTABLE(XOR(ADD_CONST(... field2 = ADD_CONST(XOR(LOOKUPTABLE(... field3 = ADD_CONST(XOR(SUB_CONST(LOOKUPTABLE(... ...

We observe that each each operation is composed of ADD_CONST, XOR, SUB_CONST, LOOKUPTABLE and the lookup tables in the client which is ~256 bytes long.

We could extract these operations and generate a really long script in python.

Why didn't I approach it this way?

1) It's really fragile. League is an actively updated game and the decryption mechanism may change in the future. If the decryption adds another operation like MUL_CONST or DIV_CONST, I would need to account for that on my end. This is unlike the reverse engineering efforts for dead games/servers where the packets do not change.

2) I don't need to know how the decryption mechanism works. Building a game server would require decryption of packet necessary. I only need to observe game state.

As for understanding how it works, I have not put enough time/effort to give an answer. :)

Glad to see another community working on similar things!

I do not know how RIOT partners with external companies, so I do not know any analysis tools or datasets besides what is publicly available :(

At least, RIOT offers special endpoints/overlays for companies ~ [1][2].

[1] https://blitz.gg/overlays/lol [2] https://www.overwolf.com/browse-by-game/league-of-legends

That's pretty cool! It's quite similar to my tool in many ways. Parsing the file, setting up the packet context and using unicorn :)

The repo isn't on github. I might release it later, but I would want it to be in a better shape if I were to.

Oops, I didn't realize that the images are not visible in dark mode. I'll fix it. Thanks for pointing that out!