tamnd

Fixed: https://github.com/tamnd/tomo-labs/blob/main/docs/content/ex...

I built these tools because I'm tired of recent tools being built for AI rather than for humans. More are coming soon!

It is a sibling project of https://github.com/tamnd/kage, in case you missed it.

Sorry, the docs come first, and I will set up Brew and Scoop later. In the meantime, you can use the pre-built images from the releases or use Go to build from source.

Possible, but currently I disable all large files, including videos.

For video downloading, I suggest wrapping around yt-dlp. It's an awesome tool.

If there's more demand for that, maybe I will implement a more relaxed version.

Currently, all of that is broken. At one point, I had a traumatic experience where an archived HTML file kept redirecting to the live site, even though I already had all the content rendered, so I ended up disabling all JavaScript entirely.

Good news for you: here is the command to clone Apple Docs:

```bash bin/kage clone https://developer.apple.com/documentation/ \ --scope-prefix /documentation/ \ --out /Users/apple/data/apple-docs \ --chrome "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome" \ --max-pages 0 --max-depth 0 \ --workers 3 --browser-pages 3 --asset-workers 6 \ --render-timeout 60s --settle 2s --timeout 30s \ 2>&1 | tee -a /Users/apple/apple-docs.log ```

Adjust it to your needs :)

I smoke-tested it, and all the content and CSS work, but I stripped all the JS, so the sidebar won't work.

If you run into any problems, feel free to create new issues in the repo. It helps me prioritize and know what should be fixed.

It should be fixed by https://github.com/tamnd/kage/pull/12

Thanks for nice trick.

Not yet supporting cookies, since I created this tool for shadowing public websites first. I will add options to pass cookies later. It will pass them to the underlying Chrome/Chromium process, so it should not be hard to do.

This could be a nice code golf project. It only needs a webview, a ZIM reader, and a way to append data to an existing binary and read it back.

I did something like that a very long time ago (Of course, I have forgotten)

For sharing, better use the html folder or zim format, Kage supports both of them.

I have a project for creating and archiving RSS feeds, keeping the full history from the time the crawler starts. I need to clean up a bit, then will open source it soon.

Exactly. For downloading, Kage requires Chrome or Chromium. Running it inside Docker makes setup easier and keeps cleanup simple:

https://github.com/tamnd/kage/blob/main/Dockerfile

Btw, let me think the way to only enable this when running inside Docker.

Making docs available offline was one of my main motivations for building this tool. I will try Apple Docs too.

I previously downloaded the Snowflake docs, and it was something like tens or even hundreds of thousands of pages, I do not remember exactly. The output ended up being very large.

By the way, I forgot to add zstd compression support to my ZIM reader/writer. I will implement that in the next version.

Thanks ;)

Kiwix has readers for almost every platform, Android, desktop, iPhone. That's why I made Kage produce ZIM file.

The executable file is mostly for people who don't have Kiwix installed yet, or just want to run the archive directly.

This brings back memories. Around twenty years ago, internet was still expensive dial-up, so I used to go to an internet cafe, run HTTrack to download websites and manga, copy everything onto my tiny 128MB USB stick (felt very large at that time), then bring it home and read offline ;))

You could use python -m http.server instead. I haven't tried it yet, but it should work.

Actually, Kage has two parts: a crawler that crawls pages and converts them to clean HTML by capturing the DOM after rendering in Chrome/Chromium, and a pack/serve component that packages the result as either a ZIM file for Kiwix or an executable file.

I have a bunch of opinionated/personal-use binaries like this in my $HOME/bin/, like delete-all-npm, clean-rust-cache, download-youtube-playlist, and get-markdown <url>. It feels good, and I don't need to remember any commands. Sometimes my coding agent can figure out how to call some of those tools too ;))

I tried to follow well-known formats first, such as WARC and ZIM from Kiwix, so we could benefit from existing tooling support.

For my own custom data format, I have a lot of private code that I plan to release soon. It is optimized for compression, fast lookups, and more. I have been working on it for two years. This is part of a larger, ambitious umbrella project: I am building Google from scratch (all open source), something that anyone can host, including the crawler, indexer, storage, and serving layers. Stay tuned!

Submitting this to Hacker News is the right place! Thanks for your idea. I will consider implementing that :)

Also, in my mind, I already have a script/program to convert HTML to Markdown, so it could actually store everything on disk as a folder of Markdown files, and then commit them to a Git repo.

I'm working on WARC too, with format from Common Crawl!

By converting it to Markdown, we save a lot of space, but it is for a different purpose and a different project: https://github.com/tamnd/ccrawl-cli

Yeah, it is. For example, openai.com is rendered with Next.js, so I will try to mirror it tomorrow.

Could you help create a new issue for that? I will do it later. It is already 1:00 AM my time, but I am happy that anyone is interested in it. : )

And thanks for the link. Let me implement this single HTML feature, it looks nice to have!

It seems this repo only saves one web page?

What I'm implementing here is mirroring a whole website, with all its subpages, so you can browse it all offline. For example, all essays from paulgraham.com.

Connecting directly with the author of the project that inspired me is awesome.

Let's collaborate and see how we can make our two projects work together. DuckDB has a feature that can write to SQLite: https://duckdb.org/docs/stable/core_extensions/sqlite. Starting from Parquet files, we could use DuckDB to write into SQLite databases. This could reduce ingress time to around five minutes instead of a week.

If I have some free time this weekend, I would definitely like to contribute to your project. Would you be interested?

As for my background, I focus on data engineering and data architecture. I help clients build very large-scale data pipelines, ranging from near real-time systems (under 10 ms) to large batch processing systems (handling up to 1 billion business transactions per day across thousands of partners). Some of these systems use mathematical models I developed, particularly in graph theory.

Happy to chat.

https://contabo.com/en/vps/cloud-vps-20/ - $8/month, 6 vCPU, 12 GB RAM, 200 GB SSD (or Hetzner servers, which offer good hourly pricing).

In my ongoing project, with 10 servers like this, I could index the large part internet (about 10 billion pages) using vector and full-text search.

That’s a silly bug in the "dynamic" README, fixing it now.