jonathan-adly

I can’t believe this has 1 point, and buried.

This is great find, very juicy, and would kill a few hot startups.

Basically- the same math as modern automated manufacturing. Super expensive and complex build-out - then a money printer once running and optimized.

I know there is lots of bearish sentiments here. Lots of people correctly point out that this is not the same math as FAANG products - then they make the jump that it must be bad.

But - my guess is these companies end up with margins better than Tesla (modern manufacturer), but less than 80%-90% of "pure" software. Somewhere in the middle, which is still pretty good.

Also - once the Nvidia monopoly gets broken, the initial build out becomes a lot cheaper as well.

We have been running this playbook for the last 2 years in healthcare, and we have been super successful. Doubling every quarter over the last year. 70%+ profitability, almost 7 figures of revenue. 100% bootstrapped.

People are still mentally locked in to the world where code was expensive. Code now is extremely cheap. And if it is cheap, then it makes sense that every customer gets their own.

Before - we built factories to give people heavy machinery. Now, we run a 3d printer.

Everyday I thank SV product-led growth cargo cults for telling, sometimes even forcing our competition to not go there.

One of the most pleasant experiences I had writing code, is early AI days when we did hyperscript SSE. Super locality of behavior, super interesting way of writing Server Sent Events code.

eventsource demo from http://server/demo

    on message as string
        put it into #div
    end

    on open
        log "connection opened."
    end

    on close
        log "connection closed."
    end

    on error
        log "handle error here..."
    end

Lots of YC companies copy each other process and selection criteria. Basically- they all have the same blind spots and look for the same type of engineer.

So, super easy to scam all of them with the same skillset and mannerism.

I send this article as part of onboarding for all new devs we hire. It is super great to keep a fast growing team from falling into the typical cycle of more people, more complexity.

The “equivalent” here would be Jina-Clip (architecture-wise), not necessarily performance.

The ColPali paper(1) does a good job explaining why you don’t really want to directly use vision embeddings; and how you are much better off optimizing for RAG with a ColPali like setup. Basically, it is not optimized for textual understanding, it works if you are searching for the word bird; and images of birds. But doesn’t work well to pull a document where it’s a paper about birds.

1. https://arxiv.org/abs/2407.01449

So, there is a whole world with vision based RAG/search.

We have a good open-source repo here with a ColPali implementation: https://github.com/tjmlabs/ColiVara

Great work! Honestly it helps so much just explaining these metrics for folks.

Early on RAG was an art, now when things are stabilized a bit, it’s more of a science - and vendors should at a minimum have some benchmarks.

I would like to through our project in the ring. We use ColQwen2 over a ColPali implementation. Basically, search & extract pipeline: https://docs.colivara.com/guide/markdown

Here is a nice use-case. Put this in a pharmacy - have people hit a button, and ask questions about over-the-counter medications.

Really - any physical place where people are easily overwhelmed, have something like that would be really nice.

With some work - you can probably even run RAG on the questions and answer esoteric things like where the food court in an airport or the ATM in a hotel.

Hi HN!

We benchmarked two ways to improve the latency in RAG workflows with a multi-vector setup. Hybrid search using Postgres native capabilities and a relatively new method of token pooling. Token pooling unlocked up to 70% faster latency with <1% performance cost.

So - the synthetic QAs datasets in the Vidore datasets are exactly like that 90% text, 10% charts/tables. OCR + BM25 is at ~90% NCDG@5 which is pretty decent. ColPali/Ours is at ~98%.

It is a small upgrade, but one nonetheless. The complexity, and the cost of multi-vectors *might* not make this worth it, really depends on how accuracy-critical the task is.

For example, one of our customers who does this over FDA monographs, which is like 95%+ text, and 5% tables - they misses were extremely painful - even though there weren't that many in text-based pipelines. So, the migrations made sense to them.

I would like to throw our project in the ring that solves this problem: https://github.com/tjmlabs/ColiVara

1. Uses half-vecs, so you cut down everything by half with no recall loss 2. Uses token pooling with hierarchial clustering at 3, so, you further cut down things by 2/3rd with <1% loss 3. Everything is on Postgres and pgvector, so you can do all the Postgres stuff and decrease corpus size by document metadata filtering 4. We have a 5000+ pages corpus in production with <3 seconds latency. 5. We benchmark against the Vidore leaderboard, and very near SOTA

You can read about half-vecs here: https://jkatz05.com/post/postgres/pgvector-scalar-binary-qua...

Hierarchical token pooling: https://www.answer.ai/posts/colbert-pooling.html

And how we implemented them here: https://blog.colivara.com/

I will do it - you are right. Lots of materials in the space is basically people selling their complex tools w/ learning as a lower priority

I would strongly advise against people learning based on LangChain.

It is abstraction hell, and will set you back thousands of engineers hours the moment you want to do something differently.

RAG is actually very simple thing to do; just too much VC money in the space & complexity merchants.

Best way to learn is outside of notebooks (the hard parts of RAG is all around the actual product), and use as little frameworks as possible.

My preferred stack is a FastAPI/numpy/redis. Simple as pie. You can swap redis for pgVector/Postgres when ready for the next complexity step.

I would check out vision models as a technique to go around OCR errors.

ColPali is the standard implementation & SOTA. Much better than OCR. We maintain a ready to go retrieval API that implements this: https://github.com/tjmlabs/ColiVara

We have traditionally used Django in all our projects. We believe it is one of the most underrated, beautifully designed, rock solid framework out there.

However, if we are to be honest, the history of async usage in Django wasn't very impressive. You could argue that for most products, you don’t really need async. It was just an extra layer of complexity without any significant practical benefit.

Over the last couple of years, AI use-cases have changed that perception. Many AI products have calling external APIs over the network as their bottleneck. This makes the complexity from async Python worth considering. FastAPI with its intuitive async usage and simplicity have risen to be the default API/web layer for AI projects.

I wrote about using async Django in a relatively complex AI open source project here: https://jonathanadly.com/is-async-django-ready-for-prime-tim...

tldr: Async django is ready! there is a couple of gotcha's here and there, but there should be no performance loss when using async Django instead of FastAPI for the same tasks. Django's built-in features greatly simplify and enhance the developer experience.

So - go ahead and use async Django in your next project. It should be a lot smoother that it was a year or even six months ago.

If you are interested in that space, would throw our project in the mix which uses ColPali under the hood transparently.

https://github.com/tjmlabs/ColiVara

The main benchmark for this is the Vidore leaderboard. Where we would love to see where VoyageAI performs compared to the more open-source implementations.

Super cool- love this.

Yea - 100% agree. Migrating old projects is really hard. DRF paradigms is just too opinionated and it will be basically a rebuild.

What we did for a client was to do api/v2 - and essentially do one endpoint migration at a time. api-v2 was basically its own independent application within the Django monolith.

One thing I would highly recommend is to know familiarize yourself with what tools exist. For example, people spend thousands of hours trying to make PDFs with lots of charts and tables work via chunking and splitting. They aren't even aware that we have end to end visual pipeline where you don't have to chunk or split - treating everything as an image.

My startup using htmx was there - and it was acquired successfully.

They migrated pretty quick to react after the acquisition due to team dynamics (offshore big teams - seeped into JS heavy client, thin server culture). htmx was a struggle there as well.

It worked amazing for us as a small team where everyone was full stack and I always build using htmx-first now. But, it is a struggle for folks who have been working in React-like patterns for 5+ years and never experienced the bless of MVC apps.

This is excellent. I have been running a very similar stack for 2 years, and you got all the tricks of the trade. Pgvector, HyDe, Web Search + document search. Good dashboard with logs and analytics.

I am leaving my position, and I recommended this to basically replace me with a junior dev who can just hit the API endpoints.

it is not about liking it. They won't like it even with determinism. The idea is to NOT learn new things, and keep doing things the old inefficient way. More headcount and job security this way.

I built and sold a company that does this a year ago. It was hard 2 years ago, but now pretty standard RAG with a good implementation will get you there.

The trick is, healthcare users would complain to no end about determinism. But, these are “below-the-line” user - aka, folks who don’t write checks and the AI is better than them. (I am a pharmacist by training, and plain vanilla GPT4-turbo is better than me).

Don’t really worry about them. The folks who are interested and willing to pay for AI has more practical concerns - like what is my ROI and the implementation like.

Also - folks should be building Baymax from big hero 6 by now (the medical capabilities, not the rocket arm stuff). That’s the next leg up.

I used this as part of my bigger write up on how to open source a Python package in 2024.[0] It is really great and takes away a lot of pain.

0. https://jonathanadly.com/open-sourcing-a-python-project-the-...

I have something similar[0] but with a different philosophy. Basically, a docker container running and you can execute code against with ability to set timeouts, auto install and uninstall dependencies and a bunch of other cool stuff.

The pain point of all of this is dependencies and making sure someone doesn’t use your infrastructure to DDOS other folks.

0. https://github.com/Jonathan-Adly/AgentRun

Great article. It’s not so bad as a user of python honestly. Venv goes a long way, and with docker its get the rest of the way there.

The problem is when you want to open-source a package. I consider myself a professional Python developer, yet I was completely overwhelmed with all ways you can manage and release a package. Hatch, poetry, rye, setup, and a million other things that can make you a waste days fighting the language.

I highly recommend this cookiecutter from Simon Willison: https://simonwillison.net/2024/Jan/16/python-lib-pypi/

I just open-sourced a project last week[0] and it took 5 minutes (after 2 days fighting other tools).

0. https://pypi.org/project/agentrun/

Very Cool! Would it work for Pydantic out of the box? Or that's something coming along?