timeserious

Can you ELI5 why an embeddings database helps here? Can pinecone/milvus be used to 'extend memory' of OSS and vendor LLMs without retraining?

Hi Steve. See my reply in above thread to Shin which calls out alot of the operational improvements we've made. Just an FYI, there's also been significant changes in how we price so it's now alot easier to start small, get started quickly and decrease the time to value with more mainstream skillsets. With the new product strategy we're seeing alot more buy side clients (hedge funds/asset managers) use kdb+/q for their workloads with KX Insights.

Hi Shin - KX employee here (the creators of q/kdb+). We launched a new product to take our time-series tech to new markets and usecases with high performance analytic needs. Over the past 2-3 years we've listened closely to the community and have made huge improvements to make kdb+ easier to develop with using options like ANSI-SQL/Python/Stream Processing DSL/microservices/Open API etc. The banks are adopting all of the new kit quickly and leverage the cloud native capabilities (like kubernetes operators/object storage) to help scale and take their deployments to new heights without deep technical kdb+ expertise. Thought it was worthwhile calling out the adoption and product strategy here at KX.. lots more to come!

https://code.kx.com/insights/

Let's get a write up of your backtesting framework too please! Terrific post @muggermuch - thank you!

The implementation is similar to before where the sql support is loaded as a library that translates to k - however it goes a few steps further by supporting (not an exhaustive list):

- Intermingling q and sql in the same expression. e.g SELECT * from qt('marketSnap`IBM')

- extensive joins, aggregates, sub-selects, cte, parameters etc - whereas the SQL92/'99 support was v. limited

- Distributed support for executing/joining across multiple targets - allowing a gateway sql query to run against in-memory (rdb) and on-disk (idb/hdb) databases. We are also testing support for querying external databases through 'virtual' tables (as if the data was stored natively in kdb+). The sql support will never replace Q APIs but we're really keen to see what our users will try to do with it. e.g single-pane governance/data mesh patterns

- PostgreSQL dialect and wire support to allow sql/no-code users to connect from 3rd party tools using the ootb PostgreSQL connectors (tableau/powerbi/glue etc) without loading and maintaining special odbc/jdbc drivers for kdb+. Queries generated via tools will push down aggregates/predicates to execute directly against kdb+ (versus the legacy import model)

Specifics aside, I hear all your points loud and clear - and agree with sql being inferior to q-sql (opinions are my own!) - but the intention is not to replace q-sql here. We're simply opening/democratising the technology for less advanced users by enabling new options for them to get more value on day 1 without the typical on-ramp usually required to start using/writing analytics in KX.

With the ultimate aim being that they see huge performance benefits when using KX (even via SQL) vs. other databases with minimal changes to their applications.

From there we hope they will begin to use more q-sql and realise how much easier and succinct it is to express questions against their time-series data e.g window functions, order and temporal based queries.

For some colour on the topic, SQL is by far the biggest request we get from new users (especially outside finance) - so this will help us grow the community and create more opportunity for careers in new sectors for kdb+ developers.

p.s We've invested heavily in better tooling for kdb+ this past 2 years which you may not have seen but I'd mostly love to discuss your ideas/requests and share with Product Strategy who I spend a lot of time working with.

Anytime! FYI we've released a new product KX Insights which is a cloud native version of kdb+ that supports ANSI-SQL (PostgreSQL) and has a bunch of ease-of-use and interoperability improvements to open the technology to a much wider user base :)

fyi kdb+ now supports ANSI-SQL. https://code.kx.com/insights/1.1/core/sql.html

Including support for the PostgreSQL wire protocol https://code.kx.com/insights/1.1/core/pgwire.html

Get in touch! I work within KX as a Solutions Architect and would love to see how we could help you out. [redacted]