snyy

Exactly this. Thank you for answering!

Ah yes, I meant accuracy.

Yes. I tried it with https://www.allbirds.com/products/womens-cruiser-canvas on our HF space and Pulpie worked great.

HF Space: https://huggingface.co/spaces/feyninc/pulpie

Images pass through as they are considered main content. Same with tables.

Pulpie will return all main content on a page as HTML/Markdown. I’m not sure I fully understand “which one this is good at?”. perhaps you can try the model on hugging face and let me know if the results look good?

https://huggingface.co/spaces/feyninc/pulpie

We see far better performance with models. Heuristics break on richer content like codeblocks, formulae, quotes, etc. In our testing, our model was 25 F1 points better than Trafilatura.

Fixed. Try again. Let me know if any other issues

Thanks! Good questions:

We haven't run a targeted eval against SEO spam yet. However, with Pulpie, each block gets labeled by what the text actually says rather than what the tags look like. Wrapping boilerplate in semantic tags fools rule based extractors precisely because they judge structure. Pulpie doesn't. The closest benchmark we have for this is the WebMainBench difficulty split, where pulpie-orange-small holds 0.813 on the hard subset. For comparison, trafilatura scores a 0.526.

For quantization, we haven't benchmarked INT8 or FP8. Everything in the post ran on L4 and A100. That said, I expect it to go well for a few reasons. It's a single forward pass over the page, so the workload is compute bound rather than bandwidth bound, which is why the L4 held up so well against the A100 and why cheaper cards should degrade gracefully. At 210M params the small model is roughly 420MB in FP16 and half that in INT8. So it fits on any consumer GPU with room to spare. Also, one pass classification tends to survive 8 bit quantization better than autoregressive generation since there is no error accumulation across decode steps.

Funnily enough, that wasn't my first choice either. I A/B tested it with a small group and people understood "up and to the right is better" faster.

You have the right understanding.

We've found that maximizing chunk size gives the best retrieval performance and is easier to maintain since you don't have to customize chunking strategy per document type.

The upper limit for chunk size is set by your embedding model. After a certain size, encoding becomes too lossy and performance degrades.

There is a downside: blindly splitting into large chunks may cut a sentence or word off mid-way. We handle this by splitting at delimiters and adding overlap to cover abbreviations and other edge cases.

As the other comment said, its a practice in good enough chunks quality. We focus on big chunks (largest we can make without hurting embedding quality) as fast as possible. In our experience, retrieval accuracy is mostly driven by embedding quality, so perfect splits don't move the needle much.

But as the number of files to ingest grows, chunking speed does become a bottleneck. We want faster everything (chunking, embedding, retrieval) but chunking was the first piece we tackled. Memchunk is the fastest we could build.

No, delimiters can be multiple bytes. They have to be passed as a pattern.

// With multi-byte pattern

let metaspace = "<japanese_full_stop>".as_bytes();

let chunks: Vec<&[u8]> = chunk(text).pattern(metaspace).prefix().collect();

A big chunk size with overlap solves this. Chunks don't have to be be "perfectly" split in order to work well.

Which language are you thinking of? Ideally, how would you identify split points in this language?

I suppose we've only tested this with languages that do have delimiters - Hindi, English, Spanish, and French

There are two ways to control the splitting point. First is through delimiters, and the second is by setting chunk size. If you're parsing a language where chunks can't be described by either of those params, then I suppose memchunk wouldn't work. I'd be curious to see what does work though!

Chunking is generally a one-time process where users aren't latency sensitive.

This is not necessarily true. For example, in our use case we are constantly monitoring websites, blogs, and other sources for changes. When a new page is added, we need to chunk and embed it fast so it's searchable immediately. Chunking speed matters for us.

When you're processing changes constantly, chunking is in the hot path. I think as LLMs get used more in real time workflows, every part of the stack will start facing latency pressure.

Memchunk is already in Chonkie as the `FastChunker`

To install: pip install chonkie[fast]

``` from chonkie import FastChunker

chunker = FastChunker(chunk_size=4096) chunks = chunker(huge_document) ```

We're the maintainers of Chonkie, a chunking library for RAG pipelines.

Recently, we've been using Chonkie to build deep research agents that watch topics for new developments and automatically update their reports. This requires chunking a large amount of data constantly.

While building this, we noticed Chonkie felt slow. We started wondering: what's the theoretical limit here? How fast can text chunking actually get if we throw out all the abstractions and go straight to the metal?

This post is about that rabbit hole and how it led us to build memchunk - the fastest chunking library, capable of chunking text at 1TB/s.

Blog: https://minha.sh/posts/so,-you-want-to-chunk-really-fast

GitHub: https://github.com/chonkie-inc/memchunk

Happy to answer any questions!

How did the OTP case work? Where was the OTP received and how did the browser know?

Or did it bypass it entirely with corporation from the website?

We're working on Mongo integrations!

We want to be the platform that connects documents to AI for all applications. Consequently, we want to cover all use cases, including the ones you mentioned :)

Yes :) Code chunker is fantastic for SQL

We don't yet, but our library comes with a visualization tool that you can use to compare chunkers directly. https://docs.chonkie.ai/python-sdk/utils/visualizer

Pretty much what you described. Convert the PDF to Markdown, join content across pages so that its all one string, then chunk it. Our evals show this approach works best.

you may want to remove errors while retaining the correction

Double clicking on this, are these messages you’d want to drop from memory because they’re not part of the actual content (e.g. execution errors or warnings)? That kind of cleanup is something Chonkie can help with as a pre-processing step.

If you can share an example structure of your message threads, I can give more specific guidance. We've seen folks use Chonkie to chunk and embed AI chat threads — treating the resulting vector store as long-term memory. That way, you can RAG over past threads to recover context without redoing the conversation.

P.S. If HN isn’t ideal for going back and forth, feel free to send me an email at shreyash@chonkie.ai.

Awesome! Keep us posted :)

Great questions!

Chunking fundamentals remain the same whether you're doing traditional semantic search or agentic retrieval. The key difference lies in the retrieval strategy, not the chunking approach itself.

For quality agentic retrieval, you still need to create a knowledge base by chunking documents, generating embeddings, and storing them in a vector database. You can add organizational structure here—like creating separate collections for different document categories (Physics papers, Biology papers, etc.)—though the importance of this organization depends on the size and diversity of your source data.

The agent then operates exactly as you described: it queries the vector database, retrieves relevant chunks, and synthesizes them into a coherent response. The chunking strategy should still optimize for semantic coherence and appropriate context window usage.

Regarding your concern about large DB records: you're absolutely right. Even individual research papers often exceed context windows, so you'd still need to chunk them into smaller, semantically meaningful pieces (perhaps by section, abstract, methodology, etc.). The agent can then retrieve and combine multiple chunks from the same paper or across papers as needed.

The main advantage of agentic retrieval is that the agent can make multiple queries, refine its search strategy, and iteratively build context—but it still relies on well-chunked, embedded content in the underlying vector database.

In addition to size and speed we also offer the most variety of chunking strategies!

Typically, our current users fall into one of two categories:

- People who are running async chunking but need access to a strategy not supported in langchain/llamaIndex. Sometimes speed matters here too, especially if the user has a high volume of documents

- people who need real time chunking. Super useful for apps like codegen/code review tools.

Thank you :)

Not sure what HN uses :)

If you want agents/LLMs to be able to find relevant data based on similarity to queries, vectorDBs like Chroma (or even pgVector) are great.

Node and Bun should work. Haven't tested on Deno yet.

We rely on the huggingface/transformers library which might be too heavy for a react-native app.

As mentioned in the other reply, we have a cloud/on-prem offering that comes with a managed ETL pipeline built on top of our OSS offering.