pronoiac

A Raspberry Pi can help debug and fix HDMI CEC flakiness, with the right cable - https://johnlian.net/posts/hdmi-cec/

I think SciOp is doing something in that area, with a catalog site and webseeds. https://sciop.net/

The Archive Team - not part of the Internet Archive - worked on a distributed backup of a portion of the Internet Archive - https://wiki.archiveteam.org/index.php/INTERNETARCHIVE.BAK

It's been dormant / on hiatus for a few years now.

I wonder if they'll go with "toploaders" - like Backblaze Storage Pods - later. They have better density and faster setup, as they don't have to screw in every drive.

They got used drives. I wonder if they did any testing? I've gotten used drives that were DOA, which showed up in tests - SMART tests, short and long, then writing pseudorandom data to verify capacity.

There's a flamewar detector, which triggers when there are far more comments than upvotes.

Kern Type, perhaps? https://type.method.ac/

Feel free to run EasyOCR against it and submit a PR

I attempted OCR, and while it's not great, it's a start. I considered adding a reference to "software wants to be free!" or the Open Letter, but I'm winding down for the night. https://github.com/pronoiac/altair-basic-source-code

I attempted OCR with OCRmyPDF / Tesseract. It's not great, but it's under 1% the size, at least. https://github.com/pronoiac/altair-basic-source-code

Checking diskprices.com - https://diskprices.com/?locale=us&condition=new,used&disk_ty... - there's a cheaper outlier for DVD-R, then it's 25GB BD-Rs for a bit.

LTO tape can be cheaper, but the cost of the drives has long been an obstacle to dabbling.

I've used ocrit, which uses those APIs. https://github.com/insidegui/ocrit

There are also:

* swiftocr - https://github.com/fny/swiftocr

* macos-vision-ocr - https://github.com/bytefer/macos-vision-ocr

They asked for something like Bluesky starter packs on Mastodon, not Bluesky starter packs on Bluesky.

I knew I'd seen something, but I just searched for for it; Fedidevs have something like that - https://fedidevs.com/starter-packs/

I think Pandoc and Calibre could work for you.

I've worked on PAIP, Paradigms of Artificial Intelligence Programming, and I might be able to help you a bit. It's around 1k pages long. I used Pandoc to generate an epub file, and then Calibre to turn that into a PDF file. I just tried using Pandoc to generate the PDF file directly, and it/LaTeX choked on some Unicode characters.

For internal ebook links, there's a Lua script. You'll have to keep anchors unique across the book for this:

* good: "chapter1#section1_1" and "chapter2#section2_1"

* bad: a "chapter1#section1" and a "chapter2#section1"

WIP: https://github.com/norvig/paip-lisp/pull/195

For line wrapping of code, there's CSS. I first used it over on "Writing an Operating System in 1,000 Lines"; here's the PR: https://github.com/nuta/operating-system-in-1000-lines/pull/...

I've run an X app from Docker, a Linux container on a macOS host. I was able to move the incantations to a Makefile: https://github.com/ryanfb/docker_scantailor

I've worked on PAIP, and I think the GitHub.com version - https://github.com/norvig/paip-lisp/ - gets more attention than the GitHub.io version linked here. The GitHub.io version automatically gets updates, I think, but I'm not verifying the Markdown works over there.

It's still in progress! It's looong - about a thousand pages. There's an ebook, but the printed book got more editing.

I made a high-quality scan of PAIP (Paradigms of Artificial Intelligence Programming), and worked on OCR'ing and incorporating that into an admittedly imperfect git repo of Markdown files. I used Scantailor to deskew and do other adjustments before applying Tesseract, via OCRmyPDF. I wrote notes for some of my process over at https://github.com/norvig/paip-lisp/releases/tag/v1.2 .

I'd also tried ocrit, which uses Apple's Vision framework for OCR, with some success - https://github.com/insidegui/ocrit

It's an ongoing, iterative process. I'll watch this thread with interest.

Some recent threads that might be helpful:

* https://news.ycombinator.com/item?id=42443022 - Show HN: Adventures in OCR

* https://news.ycombinator.com/item?id=43045801 - Benchmarking vision-language models on OCR in dynamic video environments - driscoll42 posted some stats from research

* https://news.ycombinator.com/item?id=43043671 - OCR4all

(Meaning, I have these browser tabs open, I haven't fully digested them yet)

The compression was trivial: store a suffix preceded by one byte that contained the length of the prefix that the word shared with its predecessor in dictionary order.

Oh, that looks familiar; the database for the locate command uses something similar - https://www.gnu.org/software/findutils/manual/html_node/find...

Covering all frequencies? No Blank Space?

Maybe yours did much worse because you aren't splitting files into subdirectories but creating them all in one?

No, and also, I'd expect that to be awful. 1000 folders, each with 1000 folders, each with 1000 files.

Those Arxiv and Phoronix links are great!

Is XFS well-regarded by others?

I was benchmarking filesystems by generating a billion empty files on them, and while ext2, ext4, and btrfs could finish in a day or two, xfs hit a wall in the first 4 million files, and was on track to take weeks. After hitting ctrl-c, it took 20+ minutes to get a responsive shell, and unmounting took a few minutes.

This surprised me, because it's been around for decades, and I expected scalability, performance, and robustness. A one-user accidental DOS was a surprising rake to trip over.

I remember tools that worked with the Wikipedia dumps, in bzip2, and built indexes to allow decent random access. Once you know where the compressed blocks are, and which Wikipedia entries they contain, you could start from a given block, something like 900k, rather than start at the beginning of the file. Compressing roughly a megabyte at a time, rather than a page, is a pretty solid win for compressibility.

I did some benchmarking of compression and decompression last year. Raspberry Pi 4, Debian, and my corpus was a filesystem with a billion files on it, as a sparse tar of a drive image, which I acknowledge is an odd choice, but that's where my focus was. I made graphs, because the tables were overwhelming. (Which also applies to this post, I think.) There's a blog post, but I think, more quickly useful:

* the graphs - https://gitlab.com/pronoiac/billion-file-fs/-/tree/main/grap...

* the Pareto frontier for compression vs time: zstd -1 and -9, plzip -0, xz -1 and -9. lzop -1 was a bit faster, and plzip -9 a bit smaller, but they had heavy penalties on the other axis.

I wasn't aware of Snappy.

I made:

* an issue, "make an ebook": https://github.com/nuta/operating-system-in-1000-lines/issue...

* an epub: https://github.com/pronoiac/operating-system-in-1000-lines/r... The epub is mostly ok but a bit broken.

You have a line per paragraph now, so a quick and dirty version is to add newlines after "[.!?] ", though code blocks might take some fiddling.

If you want PRs, I suggest you set up Semantic Linefeeds* sooner, rather than later. Each sentence gets its own line, and it really helps avoid merge conflicts.

* https://rhodesmill.org/brandon/2012/one-sentence-per-line/

I searched the Internet Archive for Les Mémoires de Saint-Simon - https://archive.org/search?query=Les+Mémoires+de+Saint-Simon... - skimmed the results for items with the right number of pages, and came up empty.

I uploaded a new item - https://archive.org/details/memoires-de-saint-simon-nouvelle... - though I made a mess of the metadata. It's still processing.

Pour obtenir un document de Gallica en haute définition, contacter utilisation.commerciale@bnf.fr.

roughly:

To obtain a Gallica document in high definition, contact utilisation.commerciale@bnf.fr.

My expectations would be very low, but I'd reach out to them anyway.

Page turning? So, non-destructive, with cameras? How's the quality?