HN user
I've noticed that there's another problem with microservices as well. People tend to tie microservices and multi-repo into the same strategy.
Multi-repo appears to make teams faster (builds are faster! fewer merge conflicts!) but, like micro-services, they push complexity into the ether. Things like updating service contracts, library updates, etc. all become more complicated.
First thing I noticed too.
It sounds like this framework is susceptible to head of line blocking. In my experience, that significantly reduces the utility of any applications written choosing this framework. What’s the benefit being delivered?
I noticed the same thing. I would have expected an Arc<Mutex<…>> or something similar for safe concurrency. Not sure what value is delivered by a single threaded, blocking web server.
You bring up an interesting question -- I think Tilt works best for those that find themselves in an environment where product is delivered using a service-oriented architecture deployed to Kubernetes. It's also easy to get started within a small team in a big company.
To your other point, Tilt is to development as ArgoCD is to deployment. Tilt enables on-demand, reproducible development environments that are sufficiently high-fidelity that you can often replace your shared and/or long-lived testing clusters.
With Tilt, I test my application using the same Kubernetes specs / Kustomizations / Helm Charts that you use to deploy into production. When it comes time to deploy my application, I supply these same specs / kustomizations / charts to ArgoCD.
Because I can reuse the specs for both testing and production, I enjoy far greater testability of my application, improving quality and time to market.
I think Tilt Extensions highlights the combined power and composability that using Starlarks brings to Tilt: https://docs.tilt.dev/extensions.html.
Want to create a Kubernetes secret? It's as simple as:
load('ext://secret', 'secret_yaml_generic')
k8s_yaml(secret_yaml_generic(...))
load('ext://vault_client', 'vault_read_secret', 'vault_set_env_vars')
vault_set_env_vars('https://localhost:8200','mytoken')
my_foo = vault_read_secret('path/myfoo', 'value')
my_bar = vault_read_secret('path/mybar', 'foobar')Thank you for sharing this! I think your Tiltfile just showed me how to solve something that's been bugging me for a while!
I see that you also have docker-compose files -- are those for different tasks or for developer preference?
I'm also curious to understand why you have different build scripts for CI (`buildx`) vs local (regular docker build)? In our team, we use the same build processes for both.
I've always appreciated that Tilt chose Starlark instead of YAML. Makes things so much cleaner!
I think that's a fair point -- you're making a tradeoff. And the best part is that you don't need to choose one or the other.
In my case, I find that I prefer having higher fidelity and simpler service code by using Tilt to avoid mocks. It's also nice for frontend development because, using a Kubernetes ingress, you can avoid the need for things like frontend proxies, CORS and other development-only setup.
My understanding is that dev containers are more about configuring your development environment with the right toolchains to build and run services.
Tilt is a monitor process that builds and starts your services, with a hot-reload loop that rebuilds and restarts your services when the underlying code changes. The hot reload loop even works for statically compiled languages.
Yes, Tilt really shines when you’re testing interactions with Kubernetes, such a APIs. But also things like your services’ ingress configuration and metrics scraping.
By default, Tilt is actually intended for local development using kind, minikube or other similar tooling. It supports developing against a multi-node cluster but it requires extra configuration and slows down iteration time.
Tilt can also help validate things like:
- Service discovery
- Volume mounts
- Ingress and Certificates
- Metrics scraping and configuration
- Dashboards
It’s really quite powerful and replaces the need to mock things out with docker compose. If you’re deploying to Kubernetes, Tilt gives you the option to avoid “development-only” setups like docker compose.
Interesting to see this pop up here! I’ve been using Tilt for multiple years now but the pace of development seems to have slowed down after the Docker acquisition.
I love how Tilt enables creating a local development environment that lets my services run the same in production , test and development. Greatly simplifies my service code and improved my quality.
In particular, I’d love to see Tilt be better around handling things like CRDs (there’s no way to mark a k8s_yaml as depending on a CRD being available, a frequent source of broken tilt up invocations).
Having said that, the first thing I do, when working on any new project, is to get “tilt up” working.
Things I’ve used for testing include: eBPF-based collectors for security and observability, data pipelines, helm chart development, and Kubernetes controllers. It’s very flexible and powerful for a wide range of development.
Sorry, I should have added my second thought. Your original comment about isolating MCP servers is also good!
These are tools where the AI may tell you it’s doing one thing and then accidentally do another (I had an LLM tell me it would make a directory using mkdir but then called the shell command kdir (thankfully didn’t exist)). Sandboxing MCP servers is also important!
I believe echoangle’s concern is about the security and privacy of the LLM using the data, not the MCP server itself.
I think you might be off by a factor of 10. Alphanumeric would be at least 36 characters, which would imply 2.8 trillion combininations (36 ^ 8).
At the moment, we are focused on scaling up with a single instance. While there are a couple of strategies for scaling out, none of our current methods support scaling a single view beyond a single instance. This isn't really a technical limitation but rather more of a testing and supportability limitation. Timely and Differential (the incremental computation frameworks used by Materialize) suport scale-out and have been extensively tested in scale-out scenarios.
Do you have a limit in mind on how large you're willing to go? A brief bit of napkin math shows that, at 1KB per record, your largest view should fit within 100GB of memory. This is easily supported by Materialize and I can certainly understand if this exceeds your appetite!
If you're interested in reducing the size of the in-memory dataset, does your fact table have a temporal dimension to it? By default, Materialize stores data for all time (like a database) and you can write views in such a way that it will only materialize recent data (like a stream processor). Our co-founder Frank wrote a blog post[1] detailing how to do this.
I'm honestly not sure how TAIL-ing a non-materialized view would perform. Sounds like something that would be fun to test!
Happy to chat about this further in our community Slack channel if you have more questions.
You are correct! Updates are a expressed as a retraction and an insert that happen within the same timestamp.
An example may not be necessary but it might also help clarify. Assuming you're using the psql client to run "TAIL WITH (PROGRESS)", the logical grouping for a single update will be a set of rows like the following:
...
1608081358001 f -1 ['Lockal', '4590']
1608081358001 f 1 ['Epidosis', '4595']
1608081358001 f -1 ['Matlin', '5220']
1608081358001 f 1 ['Matlin', '5221']
1608081359001 t \N ['\\N', '\\N']
...
I find that this stream of rows is very easy to convert to a data structure "{timestamp, inserts[], deletes[]}" and this, in turn, maps naturally onto reactive APIs, such as React or D3. My blog post, linked above, delves into this in more detail. Hope this explanation helps!
In the simplest case, I'm talking about regular SQL non-materialized views which are essentially inlined.
I see that now -- makes sense!
Wish we had some better database primitives to assemble rather than building everything on Postgres - its not ideal for a lot of things.
I'm curious to hear more about this! We agree that better primitives are required and that's why Materialize is written in Rust using using TimelyDataflow[1] and DifferentialDataflow[2] (both developed by Materialize co-founder Frank McSherry). The only relationship between Materialize and Postgres is that we are wire-compatible with Postgres and we don't share any code with Postgres nor do we have a dependence on it.
[1] https://github.com/TimelyDataflow/timely-dataflow [2] https://github.com/TimelyDataflow/differential-dataflow
Yep. I meant it was easy to do it the inefficient way where you just refresh the entire query when any table mentioned in the query changes. You would just have to also check if something was a view and recursively parse the SQL that is used in the view. Just use Postgres `LISTEN` and triggers or the WAL for change monitoring.
Unfortunately, it turns out that recursively refreshing views still leads to surprising behavior. I think post summarizes the problem quite nicely: https://scattered-thoughts.net/writing/internal-consistency-.... If you cannot refresh all of the views, at a single point in time, then there will be internal inconsistencies in your dataset.
When looking at automatic refreshing, simple triggers and `LISTEN/NOTIFY` don't scale, as was mentioned in the comment regarding Hasura's multiplexing. I think, in the absence of incrementally maintained views, their multiplexing strategy is a good compromise for databases like postgres. However, it should be noted that continuous query / subscription of views is the exact scenario under which incremental computation will provide both lower latency and greater resource efficiency.
Disclaimer: I work at Materialize.
This is the exact problem that we are solving here at Materialize! I wrote an example blogpost that details to how to subscribe to a SQL query in practice: https://materialize.com/a-simple-and-efficient-real-time-app...
Regarding your comment about "focus on streams", it's true that we first focused on extracting event data from other sources (Kafka, Debezium, direct change data capture from external databases such as Postgres, S3, etc). Over time, however, we plan to add additional features that will allow users to also treat Materialize as a general purpose database.
Hope this helps and happy to answer any questions!
edit: I was imprecise in my usage of the term event streams. Materialize supports inserts, updates and deletes at its core (the topK query shown in the blog post above shows this). Materialize is a more general solution than something focused on append-only event streams.
My understanding is that ksqlDB is a read-only interface on top of streams and only helps people write better consumers. The problems mentioned in the blog post relate to producers.
Please do!