Show HN: bpfquery – experimenting with compiling SQL to bpf(trace)

by knuckleheads • 2 years ago

58 9 2 years ago

Hello! The last few weeks I've been experimenting with compiling sql queries to bpftrace programs and then working with the results. bpfquery.com is the result of that, source available at https://github.com/zmaril/bpfquery. It's a very minimal sql to bpftrace compiler that lets you explore what's going on with your systems. It implements queries, expressions, and filters/wheres/predicates, and has a streaming pivot table interface built on https://perspective.finos.org. I am still figuring out how to do windows, aggregations and joins though, but the pivot table interface actually lets you get surprisingly far. I hope you enjoy it!

password43212• 2 years ago

This seems like it will be awesome, kind of like WMI but ridiculously more capable. Don't be discouraged that it didn't catch on it's first time on HN!

I'm on Firefox Mobile on Android and the first time I visited the page the 3rd/data section just kept refreshing continuously rather than showing results (maybe showed results the first load, but then refreshes of just rows with a dash?). On the second visit to the page the 2nd/C translation section failed to load.

knuckleheads4• 2 years ago

Thank you very much for your very kind comment!!

And I will endeavor to fix those bugs. I put the web interface on over the last few days and when it works it is great, but often times, it does not work. Thank you for looking at it!

password43212• 2 years ago

Maybe you got a second chance bump back onto the front page, nice!

It looks like the issues I reported earlier have been resolved, though the second time it took a while to load (hopefully due to all the traffic!).

3r7j6qzi9jvnve• 2 years ago

--I ran ctags across the linux kernel source code, pulled out all the signatures and then reference the signature when compiling the query, so you don't have to do a bunch of casts in the query.

note you don't need the casts if you use kfuncs instead, which also let you reference arguments by name (from https://github.com/bpftrace/bpftrace/blob/master/man/adoc/bp... ):

    kfunc:tcp_connect {
        if (args->sk->__sk_common.skc_daddr == (uint32)pton("127.0.0.1"))
            ...
    }

With that said, kfuncs don't work (yet?) on aarch64, so this is great for me -- I'll definitely give it a try next time I need it.

(EDIT: formatting)

knuckleheads4• 2 years ago

oh sick! thank you for letting me know. I will look into using this more for sure. I am curious, do you know if there is a way to access the arguments for kfuncs ahead of time? does the kernel provide this information somehow at runtime, or is it all still implicit?

Also, you might also enjoy https://github.com/zmaril/hancock which is some of the code I've been using to run ctags across the versions of the kernel.

(edit: I checked bpftrace -l and saw that it does have the arguments and type structs ahead of time, which is absolutely perfect, thank you very much! Super helpful.)

evgpbfhnr• 2 years ago

Try this:

    bpftrace -lv kfunc:*

(or any specific function instead of wildcard) will get you all kfuncs and their respective arguments.

output snippet:

    kfunc:vmlinux:tcp_conn_request
        struct request_sock_ops * rsk_ops
        const struct tcp_request_sock_ops * af_ops
        struct sock * sk
        struct sk_buff * skb
        int retval
    kfunc:vmlinux:tcp_connect
        struct sock * sk
        int retval
    kfunc:vmlinux:tcp_create_openreq_child
        const struct sock * sk
        struct request_sock * req
        struct sk_buff * skb
        struct sock * retval

I'm not sure how to get this info without bpftrace itself - bpftool might have the info available somehow?

westurner• 2 years ago

RIL about how the ebpf verifier attempts to prevent infinite loops given rule ordering and rewriting transformations.

There are many open query planners; maybe most are hardly reusable.

There's a wasm-bpf; and also duckdb-wasm, sqlite in WASM with replication and synchronization, datasette-lite, JupyterLite

wasm-bpf: https://github.com/eunomia-bpf/wasm-bpf#how-it-works

Does this make databases faster or more efficient? Is there process or query isolation?

Show HN: bpfquery – experimenting with compiling SQL to bpf(trace)

Related Stories