arauhala

Author here. I wrote this because "AI database" keeps getting used as one category when there are at least four architecturally different approaches, each with a different inference mechanism. Vector DBs do embedding similarity, ML-in-database is train-then-predict in SQL, LLM-augmented routes to an LLM per query, and predictive databases do Bayesian inference at query time with no model lifecycle.

Full disclosure: I'm the co-founder of Aito, which falls in the predictive database category. I tried to cover all four fairly, including where our approach falls short (latency scales with dataset size, smallest ecosystem of the four). Happy to discuss any of the categories or the taxonomy itself.

One thing that stood out while writing: the major cloud platforms (Databricks, BigQuery, Snowflake) now span multiple categories at once, which makes "AI database" as a single label pretty useless. Also, the BayesLite work from MIT CSAIL deserves way more attention than it gets. Their BQL syntax is really nice.

Hmm.. So, I'm correct, you are maintaining QA suites for 10k agents, their prompts, toolboxes and some scenarios.

That sounds like an absolutely massive scale to do QA over. Running the test suite must cost fortune and take ages.

As such, I don't think the data storage is that big problem. E.g. If you have 10-100 requests stored per agent, it's 100k-1M snapshots. Booktest normally stores the states in Git, but there is also some kind of DVC support. If you need to recreate all snapshots regularly, e.g. of you change some system wide properties often, that may became problem or not. Git and DVC can manage quite high scale. The Git PR reviews won't work though with e.g 1M changed files.

Our scale in mono repo was maybe 10k LLM snapshots in several hundred files, which worked technically. Recreating all evaluations was a bit slow (e.g 5-10 minutes) and merges often forced recreation of snapshots and re-reviews. This is ofc massively smaller scale than what you are having. We did use booktest for assistant, but it was just one assistant over dozens of use case flows.

I guess it could be somehow manageable, if you try to avoid test level manual review and only review some aggregared results in the tool, but I cannot really promise anything. It may be a worth a try.

One of the key techniques is snapshotting the LLM (or any HTTP) request. This means that if the inputs won't change, the LLM will not be called. This will also snapshot /cache LLM verifications steps.

This doesn't only saves costs, but it's main goal was to force determinism and save time. Limited changes may need only the new/changed tests to be rerun with LLM. CI typically don't have LLM API keys and only rerun against snapshots with zero costs and delays

All LLM operations tend to be notoriously slow, and at least on our side: we are often more interested of how our code interacts with the LLM. Having the LLM being fully snapshopshotted does iterating the code delightfully fast.

If you want do sampling, this can be implemented in the test code. Booktest is a bit like pytest in the sense, that the actual testing logic heavylifting is left for the the developer. Lot of LLM test suites are more opinionated, but also more intrusive in that sense

Booktest is build based of 2 decade-career in data science. It has been used to support RnD on numerous LLM, ML, NLP, information retrieval and also more traditional software engineering.

It was partly inspired by earlier examples (kudos for Ferenc), but especially real pains with how to assert ML QA with regression testing, transparency and iteration cycle speed.

So, in systems where correctness is fuzzy, evaluation is expensive, and changes have non-local effects, a failing test without diagnostics often raises more questions than it answers. This is a painful combination, if left unsolved.

Booktest is now on its 3rd or 4th iteration of the same idea, and as such it addresses most common needs and problems in this space.

It is a review-driven regression testing approach that captures system behavior as readable artifacts, so humans can see, review, and reason about regressions instead of fighting tooling.

This approach has been used in production for testing ML/NLP systems processing large volumes of data, and we’ve now open-sourced it.

I'm curious whether this matches others’ experience, and how people handle this today.

I'm bootstrapping my predictive database startup https://aito.ai/ :-)

More specifically, I have worked on the demo https://github.com/AitoDotAI/aito-demo to make use cases visual and well described. E.g. smart search use case is here https://github.com/AitoDotAI/aito-demo/blob/main/docs/use-ca...

Claude Code is doing absolute wonders on setting things up. One has to just check out for hallucinations and made-up stuff in any written content.

If I'm reading this correctly, a few themes PG touches here are:

1) loss of control when hiring a professional manager without intrusion to sub organization, because you rely on the manager provided information. If the manager is not straightforward, the sub organization may become a black box, where issues can go unseen

2) Lack of access to the frontline people and their understanding, which is opened by the Jobs like key people meeting across the org

3) Id also imagine that if you have a founder with deep domain knowledge, who has worked across all aspects of business, going fully hands off with the details, and replacing decision makers with more generic managers potentially from other industries, means that lot of expertise gets disconnected from the relevant decisions.

Ultimately the outcomes are all about the decisions and the decisions are all about understanding, which is all about the information. As such, it's not surprising if cutting the seasoned founder from both key mid level decisions and the firsthand information flows brings disadvantages.

I'd over all interpret the article to be about how hands on the founder should be about the different aspects of the business rather than the leadership, as implied elsewhere

Author here. The article seeks to answer the question "How to make ML / LMM development faster with higher quality?"

The long format article seeks address the pain points in current tooling (Jupyter Notebook, unittests) used to support intelligent application RnD. It proposes a new approach for tooling and development, which combines the benefits of notebooks (review driven process, caches) with benefits of unit testing (repeatiability, regression testing)

The tool has been succesfully used to support the developmvent topic models, analytics and GPT based functionality. Here's an example of how to create a simple test that both creates a snapshot for results that also snapshots the environment and (e.g. GPT) API calls so that the test interaction can be replayed e.g. in CI.

    import booktest as bt
    import os 
    import requests
    import json

    @bt.snapshot_env("HOST_NAME")
    @bt.mock_missing_env({"API_KEY": "mock"})
    @bt.snapshot_requests()
    def test_requests_and_env(t: bt.TestCaseRun):
        t.h1("request:")

        host_name = os.environ["HOST_NAME"]
        response = (
            t.t(f"making post request to {host_name} in ").imsln(
                lambda:
                requests.post(
                    host_name,
                    json={
                        "message": "hello"
                    },
                    headers={
                        "X-Api-Key": os.environ["API_KEY"]
                    })))

        t.h1("response:")
        t.tln(json.dumps(response.json()["json"], indent=4))

Regarding this topic, I found the following paper on TNT (three nightmare traits) interesting and bringing some clarity to the issues/confusions you are mentioning:

https://www.frontiersin.org/articles/10.3389/fpsyg.2018.0087...

Hi,

Author here! :-)

The project feels transformative, because the ML part was implemented an RPA developer using predictive database queries. It wasn't that complex data science project, but it demonstrates that simple ML can be done by (RPA) developers with somewhat tight budgets and schedules

If you have questions, feel free to ask! We@Aito also love feedback!

Regards, Antti & aito

I wonder if you are familiar with the predictive databases?

We at Aito.ai have gotten lot of interest from different RPA/no-node users and providers, and predictive database queries seem like a best intelligent automation.

https://aito.ai/blog/could-predictive-database-queries-repla...

It would be interesting to deeply integrate predictive functionality in your system, especially as it integrates a DB naturally. This could be used to offer predictive functionality from the plarform out of the box.

Isn't this the same as the inverted honesty/humility personality trait from the HEXACO personality model?

https://en.m.wikipedia.org/wiki/Honesty-humility_factor_of_t...

Hi,

One of the aito.ai founders here! :-)

We feel that the project was transformative, because the ML project was done end-to-end by RPA developers without data science team help. One of the RPA developer comments about Aito was 'that it was easy to use', which is a big thing in the machine learning space and in the context of ML democratization.

If you have any questions, I'm happy to help.

Regards, Antti & Aito

I feel this is an interesting and a rather stake on the innovation / startup aspects.

I got reminded of this Rand's blog post after PG's blog post and its comments, and also after the why companies stop innovating article.

It seems that there is a contradiction between people who are very high on openness (the personality traits) and people, who are very low on openness and also high on risk aversion and orderliness (as pointed out in Rand's article).

I feel it's still less about good vd bad, but more about explorative vs optimizing & conforming tendencies as in Rands post. The same topic has been also discussed multiple times before, once with pioneers, settlers and town planners article.

While there are obviously risks in the conformism going wild / aggressive, in general it strikes me that Rands is right in the sense that you need both 'explorers' and 'optimizers' in any successful organization.

True, but it is familiar and quite useful for it's purpose (running scripts) :)

For build it mostly works fine. Doing anything else with it would be a total pain.

Preach to the choir! :)

Make does so many things correctly:

- it gives user total freedom to modify the built

- yet it has excellent defaults for most situations

- it is full blown programming language,

- yet it's syntax is extremely specialized for the purpose and familiar at the same time (it's bash)

- it's basic assumptions/structure are extremely simple (timestamps, dependencies)

- yet it's extremely powerful and can take into account most situations.

It's be extremely Unix-style tool with very simple building blocks that combine together in extremely powerful way. Its fast to learn and easy to master and as such, the best kind of design.

Sometimes making a build with it is a small programming project, yet after using all kinds of built tools, I end up just wishing that I could use make instead.

This is a good point, and Aito's inference engine has lot of similarities with search engines. As an interesting, we can provide TF-IDF scored full text search functionality from the same indexes we are also using for inferences.

Still, while there are tons of similarities, I feel that the inferences engines are fundamentally different from search engines. The data structures are different, and I can see them diverging even more in the future. The algorithms and modes of operations are very different, even if there is some overlap.

From the user point of view, there is still a striking similarity between Aito and ElasticSearch. Both act now as auxiliary databases (all thought we would like to make Aito fully ACID with an SQL interface in future) and provide more search engine / inference engine-like functionality than full database functionality.

Would you be our first customer, if we do it? ;-)

The fact of the matter is that, while it is a tempting idea it's far from easy. The interfaces may not be that hard, but the storage itself will have its challenges and building a fast ad-hoc inference & representation learning layers on top of it is a huge project.

After working on Aito's DB and ML parts for several years, I can promise: it's more work & harder than it looks :-)

Please correct me if I'm mistake, but doesn't that require a separately prepared model in both Postgres and madlib?

You cannot simply request for an arbitrary prediction in a single & simple query.

No scripts. The change is much deeper, because Aito uses ad hoc / lazy models to provide the predictive query capabilities. If you would thinly integrate some 3rd party ML library, you would end up with separate 1) model definition and 2) training steps as an addition to 3) the prediction. Aito's database is specialized for counting statistics, so that it can create ML models in millisecond scale to answer pretty arbitrary prediction queries instantly.

There is quite normal query engine working inside Aito, but the basic database query capabilities haven't been our focus right now. We have an SQL API on our roadmap, but it will likely take time, before we can even start working on it.

I can see, why having predictive queries in solutions like Snowflake or Postgres would be extremely tempting.

Still, the problem of integrating the Aito like functionality in existing databases is that it requires lot of specialized data structures to work fast enough. While getting it to work in existing DBs is plausible, at minimum it would require completely new storage engine, or at least a wide refactoring to the old ones.

Regarding the analyst workflows: could you tell more about the main use cases? We haven't had Analyst / BI customers (yet), while they seem like a plausible audience for an Aito-like solution.

llarsson, I think you are spot on with this comment :-)

Also thank you for the advice. We have indeed gotten a good response in the less-technical, but more business oriented, RPA developer community, as an addition to appealing to the traditional software developers. (Many data scientists can obviously appreciate the idea, but the added value of this kind of tool is not that high, if you already have a routine for doing custom models & analytics)

Still, we have also faced situations, where the users have been too non-technical. I think the jury is still out on, whether these kind of solutions can be easily used by the average accountant or a business analyst.

We have few customers in production. One is about smart purchase invoice automation and - IMO - the real world data set is not that different from those StatLog / UCI / Kaggle datasets.

Of course our customer datasets tend to be on the easier side of the ML application field, but like you mentioned: the easy / fast or the low / mid-value ML applications are the place, where Aito / predictive database strengths play out and where they can carve out space.

The big thing in Aito is still, that it is 'modelless'. You don't need a prepared model to do a prediction, because the model is done ad hoc / lazily for the query. It is a much easier workflow, because you don't need to care about the models and you can have very free format / templated queries like this:

  predictCaseFieldForCustomer(customer, department, case, predictedField) {
    return aito.query({
      from: f'customer-$customer',
      where: {
        $on : [
          { case : case }, 
          { department : department }
        ]
      },
      predict : predictedField
    })
  }

The reason, why the predictive queries create value relates to the simplified workflow and the simplified architecture.

Instead of defining model, training model and using model, you merely ask for an arbitrary unknown variable, based on any arbitrary facts. This provides much easier interface, much faster iteration cycle and other technical benefits like the ability to create generic query templates. These benefits stand even when compared to the AutoML platforms (which also do lot of heavy lifting to simplify the workflow).

Regarding the architecture and process: the system has a lot of resemblance to normal databases (and especially to the Lucene like search engines), but in order to serve arbitrary predictive queries, the entire database is specialized in-and-out for counting statistics and doing millisecond time-frame ML modeling. The things are somewhat described in the article, but I'm also happy to answer to additional questions about the system.

As interesting details the underlying database is very functional programming oriented and build on a Git-like system. We'd like to expose the database's snapshot and branching abilities in the future.

I do understand your point. There are definately tons of hard data science problems, which are simply not suitable for the predictive query kind of approach.

At the same time there are tons of ML problems e.g. in process automation or user interaction, which have extremely strong patterns and very easy to treat with sophisticated enough ML model.

Regarding your list of items. Feature engineering is greatly managed by the user selecting relevant facts in the query, by analyzers, by MDL based feature learning and by information theory based feature selection. I feel this approach is pretty robust for many problems, all thought not complete. There are special queries like $on for making conditional variables of form A|B, and $numeric to deal numeric data, that can be used manually.

Model debugging can be partly done with $why explanations, that are easy to create with the Bayesian approach. I feel that model debugging has been good enough.

Latency requirements and constant updates are more about software/database engineering and they are solvable, but right now we do recommend batch updates and applications, that can deal with sparsely occurring multi second latency. And OFC if you have limited data sets (less than 100k), there shouldn't be such problems.

I feel that all the problems you listed are solvable, but they are of course hard problems and we are still on the roadmap on fully solving those issues for larger set of applications. For many applications (like RPA, internal tools, analytics) these are not real issues, while the benefits (easiness, speed) are extremely concrete and relevant.

It's not just different interface, but different workflow.

In traditional ML you need to a) define model to do prediction A -> B b) train the model, which may take minutes c) then do the predictions form A -> B, which takes (1, 10, 100) microseconds

With predictive queries, you: a) Ask prediction for any X based on any A, B and C and expect answers in (1, 10, 100) milliseconds

You basically trade throughput and latency to get higher productivity, faster iteration and simplified overall sysstem

Here's an early article on predictive databases:

https://aito.ai/blog/introducing-a-new-database-category-the...

The big difference of the predictive queries / databases compared to trained models is that predictive databases are prepared / optimized to allow making arbitrary predictions without pretrained model. So you can basically as to predict any X based on any A, B and C and expect a more or less immediate answer.

The benefit of not having pre-trained models relates to workflows and architecture, as described in the article. The disadvantage of having such instant generic prediction capability is that it's technical hard to implement, as described in the 'Quality' chapter.

I would say, that the answer is partially yes and partially no.

We have done several projects with simple machine learning problems, where e.g. semi-technical RPA developers have been able to implement ML based automation just fine. We have gotten compliments that Aito is easy to use, and one intelligent automation demo was implemented in 5 hours with some integrations by 2 RPA developers. It's worth noting, that there is an absolute abundance of ML problems (especially in domains like automation or UIs) that are simple to understand and easy as ML problems.

At the same time, we have run into many ML problems, which require data scientist to even formulate the problem and to think about it. There are also problems, where Aito's Bayesian approach is inadequate and you need a data scientist to do good amount engineering to make it possible to model the patterns and then find the right model.

So TBH: I don't think the predictive queries can fully replace the traditional models or data science work, but there are large application domains, that can be handled just fine with predictive queries and even by normal developers.

Regarding text: Aito can already handle simple texts just fine, and with representation learning based 'world modeling' approaches: I believe that we can do also more complex analysis on text.

Overall, Aito does not seek to provide the best models or solve the hardest problems, but it's value proposition is on speed and easiness. We focus on investment instead of return in the return-on-investment equation. It gives an advantage on the lower-value 'tail' of the ML market, where the importance of costs is higher, and where the traditional data science approach is economically not that attractive.

It does sound appealing, doesn't it :-)

Regarding the model: as described in the article, you can create specialized ML models in millisecond scale. This means that all computational, latency and consistency issues disappear, as you don't need to update models.

The not so nice side of this approach is that there are currently limitations on scaling, so that it may work fine with 100k samples, but becomes somewhat slower with 1M samples and likely won't work in 10M and 100M scale. Still, like described in the article, I believe this issue is solvable within near future, and it's not an issue for many/most applications.

The author here. If you have any questions about the article, I'm happy to help. :-)

I do believe, that the query based ML will replace the trained model based ML in the long run. I believe this not because the results would be better, but because it offers higher productivity and greater simplicity.

What are your thoughts? Does the query based ML makes sense?