yevpats

Nice! I think this space is growing. There are a few others Im aware off in the space worth checking out: https://julius.ai/, https://cipher42.ai (I've built the early version of this).

complementary hitler uses docker https://www.youtube.com/watch?v=PivpCKEiQOQ . but yeah I think infrastructure is hard but at the end it's just a tool for a job, you can use terraform, cdk, pulumi, yoke - the cycle of X for Infrastructure is bad continues :)

This is why we built CloudQuery (https://github.com/cloudquery/cloudquery) an open source high performance ELT framework powered by Apache Arrow (framework is open source, our connectors closed source). You can run local pipeline and write plugins (extractors) in Go, Python, Javascript and any other language and save data to any destination (files, SQLite, DuckDB, PostgreSQL, ...)

(Founder here)

Check out CloudQuery - Arrow powered ELT framework (Author here :) )

I wish you all the best but unfortunately I think you will find out that this business model doesn't work and if you give something for free most companies won't pay for it even if they have the money as it doesn't even worth to go through procurement if it is free. You will have to lock some features in an open core way or other way to paywall companies to pay-up.

Hope Im wrong but I think in the last few years all companies realised that FOSS is not a business model.

Congrats! What is the business model? Also, I see that the license is permissive which is great but how do you protect yourself from other companies hosting your solution and competing your cloud offering?

Exactly. open source and free tiers in b2b are dead business models for the most part in the post ZIRP Era.

planetscale - https://news.ycombinator.com/item?id=39618815

Here is a free tip to fix terraform business model (no MBA required): - Terraform is one of the most valuable tools for IaC - it's all free and it shouldn't - it is not a consumer product, it is not running ads and 99% of the users are business users with money. - Keep Terraform SDK free and open source, close source plugins, charge per resource. - Invest much more into Core and plugins.

all forks are doomed with the same destiny, business model is not sustainable with negative margin - as long as you have such a generous free tier and storage can run on S3 the price you can ask for a managed version is incredibly low.

You welcome :)

Firstly, congrats :) (Generalized) ingestion is a very hard problem because any abstraction that you come up with will always some limitations where you might need to fallback to writing code and have full access to the 3rd party APIs. But definitely in some cases generalized ingestion is much better then re-writing the same ingestion piece especially for complex connectors. Take a look at CloudQuery (https://github.com/cloudquery/cloudquery) open source high performance ELT framework powered by Apache Arrow (so you can write plugins in any language). (Maintainer here)

Check out CloudQuery (https://github.com/cloudquery/cloudquery) for the cloud version :) (Maintainer here)

Firstly, Big Kudos if you were able to bootstrap a big business this is very very hard (actually doing this with VC money is also hard). As someone who did both VC funded startup and self-funded (fuzzit.dev acquired by GitLab), I would say doing a self-funded startup even though it sounds good (because you are an underdog and you win on the hardest level if you succeed), a lot of time is really impossible or if possible there is information people are not sharing - for example for self-funded - did you include 300k-600k you had as saving that you used while working on it (or did you consulting while working on it). Did you take loans to grow faster? - the details are really important.

Personally I got to a conclusion that the common ground is a lot of time what makes sense. If you can validate as much as possible before raising the first round, raising reasonable amount of what you need to prove or get to the next milestone and do a lot of first principal thinking that could be better then beiing on each side of the extreme - let's raise as much as possible or Im not taking any VC money because it is all bad. Just my 2 cents but yes if you can grow a business with your own money - then it's amazing - just think this is not possible in a lot of cases and products.

Check out CloudQuery - https://github.com/cloudquery/cloudquery for an easy cloud asset inventory.

Check out https://news.ycombinator.com/item?id=38812087 for high performance and high load connectors. (Maintainer here :) )

Unified API is a holly grail but as many said quite difficult to abstract every use case in a scalable way that won't break. At CloudQuery (https://github.com/cloudquery/cloudquery) we focus solely on the ELT use-case(Founder/Maintainer here).

Congrats!! How does it compare to the ELT space and the modern data stack where you have ingestion/storage/visualization layers decoupled?

Asking as the founder of CloudQuery (https://github.com/cloudquery/cloudquery), Saw Datasette quite a few times around data exploration but curious to hear about the most popular use-cases of Datasette!

I think it's not only the issue with terraform but also the underlying infrastructure. AWS should've never have imperative APIs in the first place. Or at least it's time for AWS V2 APIs

IMO Terraform providers should've never been free. It should've been open-core, whether you are running on prem or on the cloud. There are multi-billion dollars companies using Terraform and pay exactly 0$ (yes, some that are generous are paying for support, great but you don't build a business on charity). Maintaining 3000 APIs for GCP, AWS, and Azure is costing at the very list $20M/year - trying to drive everyone to the cloud offering instead of charging for whatever people already use is the wrong way around imo. You can charge less but charging nothing doesn't gonna work. Heck, even a restaurant is charging a bit less for food and then charges more on beverage but it never gives the food for free.

I think today with things like CoPilot that speeds up development significantly the era of python and untyped languages is past its peak.

Congrats!! We also focus on performance at CloudQuery (https://github.com/cloudquery/cloudquery) by using Golang, gRPC and still trying to be abstract enough to support different databases :)

In any case good luck!

I see that PRQL is compiled to SQL. How do you deal with different type of SQLs? i.e Snowflake/Postgres/Clickhouse? Isn't it a potentially impossible abstraction or at least there will be places where you will need to use the native support sql language (unless of course databases will adopt that in the first place which might be great)?

Looks interesting but I wonder if this abstraction doesn't leak and it doesn't just become a more limited version of vanilla python + vanilla SQL.

Congrats!! Check out also CloudQuery (https://github.com/cloudquery/cloudquery) an open source ELT that support DB->DB without Kafka or Debezium (Founder here) :)

Yeah, Go is not a great language for frameworks as it usually works until it absolutely doesn't and then you have to rip that out.

We also adopted arrow recently for CloudQuery - our open source ELT framework https://arrow.apache.org/blog/2023/05/04/adopting-apache-arr...

Looks cool. What is the difference between this tools and benthos (https://www.benthos.dev/)?

Quite interesting. It's basically like a centralized digital currency which is legal. most of the time folks think Fiat vs Crypto because it's de-centralized and it kinda promise the promise land but I wonder if we will be moving to just a world with centralized central bank currency and no banks.

Or you are stupid enough to do it and end up in jail for the rest of your life instead of putting your own imagination and skill into something a bit nicer

R.I.P.

His book delivering happiness is a good one with a shadow story about depression, and inability to find piece and happiness for oneself.

Ironically this will most probably help as most companies now understand that putting more people on the same piece of software doesn't really help but make it worse.

Thanks everyone for sharing thoughts. Pretty interesting to see very different thoughts and ideas. I guess now only option is to wait and see.