alexflint

Thanks! Yeah linux network namespaces are a powerhouse that we're only just starting to fully utilize (outside of containerization).

Thank you! mitmproxy is fantastic - thanks for all the work that's gone into that project. Maybe we can get in touch and chat about all this stuff.

Nothing at all!

You do need write access to /dev/net/tun. This is standard for all users for the distros that I've looked into, but it is ultimately a distro-specific thing.

Unfortunately TLS happens inside the the application, not in the kernel, so using eBPF to hook syscalls to write won't help with TLS decryption.

Fascinating, I hadn't heard of WARC - thanks for the pointer and I'll consider it.

Yeah that photo is real! That's where I live!

Yes, it's true, actually doing it is hard, but to be honest not as hard as a lot of other stuff (getting a phd for example, or goodness gracious buying a house in San Francisco). I love getting up early. I love living out in nature. I love chanting and eating meals together and making a version of Buddhism for AI systems!

If you're interested in what it's like, we have written a bunch of very short few-paragraph stories about our time at MAPLE here: https://tales.monasticacademy.org/

Wireshark is awesome but yeah as others mentioned it's the TLS decryption piece that is difficult in that workflow

Woah, this is super helpful info. Thanks. That sounds like a real possibility for a macOS port actually.

This is a very interesting possibility actually. Not 100% sure what the implication would be but maybe there is something here.

Yeah mitmproxy is great. The main difference with httptap is that it's an HTTP proxy server, so you have to configure your program to use a proxy server. When I wrote httptap I wanted to be able to run `httptap <command>` and see the httptraces right there in standard output. There is an absolute ton of cool things that mitmproxy can do that httptap is not even close to, like interactively modifying HTTP requests and such. Very cool project.

Thanks!

I did try this with firefox but it doesn't work right now due to (I think) the user namespace messing with user IDs. I think I should be able to fix this, though. I will have to try it with other desktop apps soon too...

Same here actually -- have wanted this from time to time for years. Finally sat down and worked it out.

Yep. This is the first time I've mentioned this but there are actually two implementations of this in the codebase -- one uses gVisor and one is an incredibly bare-bones TCP implementation that I wrote myself in 550 lines of Go code (tcp.go). The home-grown one isn't used by default and it doesn't support much of TCP proper, but it actually works pretty well. You can use it with `--stack=homegrown`.

Very cool! Would love to hear how it goes, especially any features that would be useful in the context of real-world usage.

Yep, mitmproxy is fantastic IMO.

Yeah learning about gVisor was one of the most fun parts of this project. The gVisor devs are incredibly helpful. If you look through the gvisor-users mailing list you'll see me asking them various questions about this and they really helped out a bunch with some incredibly thorough answers.

Interesting. Care to share any info about your setup? Would it be a matter of httptap reading a certain environment variable and then forwarding traffic to a SOCKS proxy?

FWIW there is also the excellent tun2socks (https://github.com/xjasonlyu/tun2socks), which was a significant inspiration for this project, and is specifically designed to forward traffic from a TUN device to a SOCKS proxy.

Thanks! But yep I agree, you're exactly right, it's ultimately... frustrating that there isn't really an agreed-upon or system-enforced way to specify CA roots to an arbitrary process.

It's true that httptap mounts an overlay on /etc/resolv.conf. This is, as you'd expect, due to the also-sort-of-frustrating situation with respect to DNS resolution in which, like CA roots, there isn't a truly reliable way to tell an arbitrary process what DNS server to use, but /etc/resolv.conf is a pretty good bet. As soon as you put a process into a network namespace you have to provide it with DNS resolution because it can no longer access localhost:53, which is the systemd resolver, which is the most common setup now on desktop linux systems.

I do think it might help to mount /etc/ca-certificates as an overlay. When I started looking into the structure of that directory I was kind of dismayed... it's incredibly inconsistent from one distro to the next. Still, it's doable. Interested in any knowledge you might be able to share about how to add a cert to that directory in a way that would be picked up by at least some TLS implementations.

Very good to know about. But you still have the problem of decrypting TLS traffic.

Wow, did not know about this!

Super cool! Connecting what you capture to Chrome DevTools is fascinating, as is using eBPF. Great work getting the devtools to run as a standalone web app. You won't believe it but I have a half-finished attempt of the same thing for the firefox network tab - in the "networktab" dir of the repo!

Very cool project, would love to learn more and happy to chat more about it.

Wow that's an incredible quote! It feels like that to me too.

Agreed! So there isn't any interface for modifying requests/responses at present, but it's definitely possible given the underlying approach. If you consider [this line of code](https://github.com/monasticacademy/httptap/blob/main/http.go...) where you have an HTTP request parsed from the <command> that ran and are about to send it out to the public internet: you could modify the request (or the response that is received a few lines further) in just the way that you would modify a normal http.Request in Go.

Yeah, for other readers who are looking at this thread, the connection is just that this (httptap) is a Monastic Academy project, and what that means is that there is a group of people living on 123 acres in Vermont according to a fairly traditional Buddhist monastic structure (though we are not ordained monks), and during the day we work on a number of technology and non-technology projects together. The link to the readme that sevg posted above is a good overview:

https://github.com/monasticacademy/httptap?tab=readme-ov-fil...

Thanks!

Haven't measured performance at all. However when I decided on the approach I looked at the performance benchmarks from tun2socks, which uses the same gVisor code that httptap uses, and it seems that pretty great performance is possible with that code. Still need to do the work of actually achieving comparable performance.

Payloads and headers can be dumped with --body and --header respectively. There is an example in the readme of doing this (just search for --body) and I'll work more on documenting this in the coming days.

httptap is a process-scoped http tracer that you can run without root priveleges. You can run `httptap <command>` where <command> is a linux program and you get a trace of http/https requests and responses in standard output:

    httptap -- python -c "import requests; requests.get('https://monasticacademy.org')"
    ---> GET https://monasticacademy.org/
    <--- 308 https://monasticacademy.org/ (15 bytes)
    ---> GET https://www.monasticacademy.org/
    <--- 200 https://www.monasticacademy.org/ (5796 bytes)

Great point. We need to address this. Right now the examples are written and owned by us (Kite), so you would technically be in uncertain legal territory (depending on whether fair use was applied per-example or per-page). We will change this to make it crystal clear that copying examples is allowed.

Good idea - will look into it.

Right, but the python standard library docs are just a tiny tiny fraction of all the documentation in our index, and the documentation itself is just one corpus out of many.

It wouldn't be a viable business unless we thought we could add real value _and_ thought that we could capture some of that value. Yes, being on the cloud allows us to set up an attractive subscription model. But Kite won't succeed as a business unless it turns out to actually be a value-add over local solutions. That's what we're betting on.