mattaitken

This is cool. I think for our use case this wouldn’t work. We’re dealing with billions of rows for some tenants.

We’re about to introduce alerts where users can write their own TRQL queries and then define alerts from them. Which requires evaluating them regularly so effectively the data needs to be continuously up to date.

ClickHouse is a high performance OLAP database. It’s good for analytics and search.

We use it (I’m the author or the article) so users can search every run they do and graph all sorts of metrics.

Yes it’s exactly this. There’s not a neat permission boundary when you have users, orgs, projects, environments. Let alone when you add RBAC too.

You’re right RLS can go a long way here. With complex RBAC rules it can get tricky though.

The main advantages of a DSL are you can expose a nicer interface to users (table names, columns, virtual columns, automatic joins, query optimization).

We very intentionally kept the syntax as close to regular ClickHouse as possible but added some functions.

In a system with organizations, projects and advanced user access permissions having separate databases doesn’t full solve the problem. You still need access control inside each tenanted database. It also makes cross-cutting queries impossible which means users can’t query across all their orgs for example.

The DSL approach has other advantages too: like rewriting queries to not expose underlying tables, doing automatic performance optimizations…

Yes, you can actually do this already because we expose a REST API and TypeScript SDK functions to execute the queries.

We think Zapier is a fantastic product and we've used it ourselves many times. But it's more focused on simpler use cases and we found ourselves hitting the wall and then being frustrated that there wasn't a good alternative that could live in our code.

You can use JavaScript and will have a great experience – all of our code is in TypeScript which means you get a really nice experience as either a JS or TS developer.

Really sorry about this – we are working on a guide and it will be released soon.

We're using the open core model that Gitlab use. It's popular because it mean 95% of the code is MIT (good for everyone) and a small number of enterprise features are under a different license. This puts off bad actors from building a commercial competitor with zero effort. The alternative we considered was AGPL but that felt worse for our open source users.

All the code that has been pushed so far is under MIT and we currently have no enterprise features (/ee folders). The majority of future code will fall into this same bracket.

Some features that are for "enterprise" will be put in /ee folders – ideally we will put all of that in a single /ee folder in the root but we wanted to cover the case where that's non-trivial to implement.

This open core model (that Gitlab use) is popular because it strikes a nice balance between having an open source project (good for everyone) and it puts off bad actors from building a commercial competitor with zero effort.

First off, thanks for this feedback – it makes it clear that we need to do a better job.

We're following the GitLab open core model. Our licensing is the same as their's (and many other open core companies).

This means that all existing code and the vast majority of future code will be under the MIT license. At some point, we will add some enterprise features that are in /ee folders and they will be licensed differently. This is to protect us against a competitor taking the code and launching an alternative hosted service without any effort on their part. We felt that this was a friendlier option than going with something like AGPL.

Sorry that our self-hosting guide isn't available yet, we will get it up really soon. We want people to use Trigger.dev and that includes self-hosting.

There are lots of open core companies that have a cloud version and the ability to self-host. I don't think that our business model conflicts with being open source – anyone contributing will make the experience better for everyone (cloud or self-hosted).

Hopefully this answers your questions and helps to alleviate your worries about the project. We are big believers in open source and have thought hard about how we can create something valuable for everyone.

Thanks, this is great feedback. You're right that these examples don't highlight the real advantage of our platform vs no-code tools. We're going to add some new examples to our site based on what our customers are building.

The three categories of problems you identified that are best solved with a code tool are what we're seeing with our early customers. Plus a lot of notification use cases like when developers want to be notified in Slack when something important/bad happens.

Hey, Inngest looks impressive, congrats! We're tackling similar problems. Our actual inspiration to build this was Interval back in November. They make it easy to create internal dashboards inside your own codebase by using their SDK. We wanted to offer the same great developer experience but for creating workflows.

The website is correct, it's just a bit confusing so I'll explain more here:

The workflow code is in your codebase and runs on your servers, we don't host that.

We host the service that triggers your code (using events like scheduled, webhooks, customEvents) and that you can call using our SDK to do requests, logging and delays inside the workflow (e.g. using our Slack integration or using our fetch that auto-retries with exponential back-off). We also host the web app that you use to authenticate with APIs, show all of your runs with associated data and any errors.

Soon we will add a self-hosted guide so that you can also choose to host the Trigger.dev service yourself too. It's a bit hard to explain, hopefully this clears things up!

We are using Nango (formerly called Pizzly) for our OAuth integrations. It's a fantastic way of making it easier to do OAuth!

Thanks! That's been our experience exactly and is the reason we're building it. The experience of AWS step functions on one extreme and Zapier on the other.

Ugh that sucks, sorry. We love the name but the only thing holding us back was the association with guns.

"A simpler alternative to Temporal" is a good description for us. We'll get a self-hosting guide up soon for you. We're adding integrations that people ask for as fast as we can build them and Stripe is high on that list :)

Thanks for such an amazing reply, I'll do my best to answer everything. I am also on the Trigger.dev team btw.

1. Your API key with us can either be passed into the Trigger constructor or you can use our TRIGGER_API_KEY environment variable. For the API integrations we provide, we handle API keys for you and they're added inside the dashboard UI. For everything else: as the workflows and files are just code on your server you can use environment variables (or your preferred alternative) to inject secure values.

2. When a workflow step that requires OAuth is hit, the workflow pauses and prompts you to sign in. After you've signed in the workflow continues where it left off. You only need to connect each service once per organization. You can sign optionally in multiple times to the same service and switch connections where needed, e.g. multiple separate Slack workspaces.

GitHub star to Twitter follow is a great idea. You could achieve that right now by using our GitHub integration and our fetch call (which auto-retries with exponential back off and logging in the dashboard). We'll add a Twitter integration soon so this is really easy and publish it as an example :)

3. We have detailed error messages attached to the step that failed but don't currently have a way to hook into this like you describe. This is a great idea, I've added it to our task list.

4. Exactly right, each "step" is a block in the UI with clear inputs and outputs. We would really like to have a graph view. That'll be especially useful when you put loops in the code, and of course branching.

5. 100%, making this work well with AI code generation would be great.

6. This is a great idea, we'll investigate. Rather than just mapping 1:1 with an SDK we're trying to make the experience better. For example, with the normal Slack API you can't post to a channel name, you have to use an id. We make it so you can use the name and we deal with the hassle for you.

7. We'll look into this too. Integrations for interoperability could make this easier.

8. We're going to follow the GitLab/PostHog model. Like GitLab, the repository as a whole is MIT licensed but we can add some /ee folders in the future with enterprise features. We feel it gives us the right balance of being open source and gives us some protection from a competitor hijacking the project.

9. We will separate out the dashboard from the runners – this was a compromise so we could ship the first version faster.

10. We can relatively easily ship other languages SDKs and Python is probably where we'd start. Our core backend code will most likely remain in TypeScript.

Thanks again for so much feedback!