MrSaints

I can't remember what my terminal was like before this. It's so fast, seamless and part of my day to day workflow. Solid open source project.

Wouldn't there be latency overhead between the workers (edge) and Postgresql? I guess the post alludes to relying on caching on the edge to mitigate such roundtrips.

But I suppose that as long as you are dealing with data that is not easily cached, the bottleneck with using serverless edge will still be between the worker, and the database?

+1 for this. The great thing is you still have the option to incrementally opt for SSR and/or enriching statically generated pages with build-time data.

Is my understanding correct in that a "Pane" is essentially a UI component? And I am assuming there are tests to catch cases where a new pane is defined in the backend, but not in the SDK (or I suppose, how do you deal with any discrepancy)? Do you lazy-load the components?

I'm quite curious about your scale / size of your team.

Most of the moving parts for k8s are easily handled if you go for a managed service.

Using Nomad alone is fine, but with Consul in the mix, it requires quite a bit of set-up (especially for some degree of HA), and from my experience, is no less harder than using something like k3s.

Overall from my perspective, it just seems like Nomad + Consul sits in two places.

Either for an org. small enough where HA is not a concern, so setting it up, and running it "on-premise" is trivial. Or, for an org. large enough where you can have a team dedicated to setting it up, and managing it to ensure it meets various SLAs.

Genuinely curious to know what's your experience been like, and if it matches up with this.

Agree on all points (and definitely, a mostly happy customer too).

(3) was the cause of numerous production incidents for us. We had to contact support to have it scaled up, and sometimes they'd take up to 3 working days to get back to us. Happily paid more for AWS to get better support, and stability.

The functions take about 10s-15s to execute on cold start

This may be a bit of an exaggeration, and may vary depending on your deployment (from experience, it takes up to 5s at most), but I agree. There is a very noticeable cold start time which makes it not ideal for any business critical services. It is probably only good for things like document conversions.

I'm quite curious about the URL service when deploying the server onto Kubernetes. How is the public `waypoint.run` able to access deployments in a Kubernetes cluster? I know it uses Horizon. But, are the requests somehow proxied through the deployed Waypoint server?

What you have described is quite similar to what Lyft's Flyte is trying to accomplish https://flyte.org/

A lot of Tensorflow inspired DAGs approaches the described node processing in the same way.

You raised a pretty good point about being able to review the BPMN. I did not immediately think of this, but now that you have mentioned it...

1. It was good for communicating the engine room

I remember demo'ing the workflows within my team, and to non-technical stakeholders. It was very easy to demonstrate what was happening, and to provide a live view into the state of things. From there, it was easy to get conversations going, e.g. about how certain business processes can be extended for more complex use-cases.

2. It empowered others to communicate their intent

Zeebe comes with a modeller which is simple enough even for non-technical users to stitch together a rough workflow. The problem is, the end-result often requires a lot of changes to be production-ready. But I have found that this still helps communicate ideas, and intent.

You do not really need BPMN for this, but if this becomes the standard practice, now you have a way of talking on the same wavelength. In my case, we were productionising ML pipelines so data scientists who were not incredibly attuned to data engineering practices, and limitations, were slowly able to open up to them. And as a data engineer, it became clearer what the requirements were.

On the point about testing, the test framework in Zeebe is still a bit immature. There is quite a few tooling / libraries in Java, but not really in other languages. The way we approached it was lots of semi-auto / manual QA, and fixing live in production (Zeebe provides several mechanisms for essentially rescuing broken workflows).

The testing in Cadence / Temporal is definitely more mature. But you do not have the same level of simplicity as Zeebe. That said, the way I like to see it / compare them, you could build something like Zeebe or even Conductor on Cadence / Temporal, but not vice versa.

I've used Conductor, Zeebe, and Cadence all in close to production capacity. This is just my personal experience.

Conductor's JSON DSL was a bit of a nightmare to work with in my opinion. But otherwise, it did the job OK-ish. Felt more akin to Step Functions.

Arguably, Zeebe was the easiest to get started with once you get past the initial hurdle of BPMN. Their model of job processing is very simple, and because of that, it is very easy to write an SDK for it in any language. The biggest downside is that it is far from production ready, and there are ongoing complaints in their Slack about its lack of stability, and relatively poor performance. Zeebe does not require an external storage since workflows are transient, and replicated using their own RocksDB, and Raft set-up. You need to export, and index the workflows if you want to keep a history of it or even if you want to manage them. It is very eventually consistent.

With both Conductor, and Zeebe however, if you have a complex enough online workflow, it starts getting very difficult to model them in their respective DSLs. Especially if you have a dynamic workflow. And that complexity can translate to bugs at an orchestration level which you do not catch unless running the different scenarios.

Cadence (Temporal) handles this very well. You essentially write the workflow in the programming language itself, with appropriate wrappers / decorators, and helpers. There is no need to learn a new DSL per se. But, as a result, building an SDK for it in a specific programming language is a non-trivial exercise, and currently, the stable implementations are in Java, and Go. Performance, and reliability wise, it is great (relies on Cassandra, but there are SQL adapters, though, not mature yet).

We have somewhat settled on Temporal now having worked with the other two for quite some time. We also explored Lyft's Flyte, but it seemed more appropriate for data engineering, and offline processing.

As it is mentioned elsewhere here, we also use Argo, but I do not think it falls in the same space as these workflow engines I have mentioned (which can handle the orchestration of complex business logic a lot better rather than simple pipelines for like CI / CD or ETL).

Also worth mentioning is that we went with a workflow engine to reduce the boilerplate, and time / effort needed to write orchestration logic / glue code. You do this in lots of projects without knowing. We definitely feel like we have succeeded in that goal. And I feel this is an exciting space.

after several attempts to cancel the contract, I cancelled the direct debit.

They did do that.

But, the issue is if it accumulates without a resolution, it'll reflect badly on your credit score.

They will be doing an AMA on Reddit again soon according to https://youtu.be/y4xBFHjkUvw?t=674

There's a relatively new one by Camunda (https://zeebe.io/) that also works quite well in cloud native environments.

We are currently experimenting with workflow engines for orchestrating different components, e.g. data ingestion, data preprocessing, feature engineering, scoring, automated decision making / escalation. Namely, Argo for offline processing, and bulk processing; Zeebe, and Cadence (trying both out) for online processing, and business logic / application services.

We don't yet have a polyglot architecture, but we do have the requirement of running distributed services (partly because there are certain components of the pipeline that needs to be run on-premise), and we have found that workflow engines / orchestrators definitely makes it a lot easier to reason about the wider architecture / have a bird's eye view. It works for us. No need to handle callbacks, events, queues, etc. We also do have the potential to run a polyglot architecture.

We tried out Celery Workflows, and struggled to get it "production ready", so I'd advise against this for complex workflows. We also found the visibility lacking.

We have yet to fully try out Kubeflow, and MLflow. What is not quite working at the moment is creating, and deploying portable models. And I don't mean simply pickling, and storing an artifact.

Leveraging containers (Docker), and slapping simple anti-corruption layers (e.g. simple web APIs) has also helped. We have a more consistent way of deploying, and isolating code without having to rewrite much.

We want to look into using Nuclio, and/or knative to ease the process of deployment, and to empower the data scientists to deliver without much engineering expertise.

Others have mentioned using base classes or standard interfaces for their models. We tried this too, but it didn't work. The generalisation early on was met with conflicting requirements, and broke the interface segregation principle (not that it matters too much, but it can be confusing to not know precisely what is being used or not used). We figured it's much easier to procrastinate any abstractions. Let the data, and it's flow do the talking.

Interestingly, Googling `ProjectedVolumeTokens` yields this very post. I think that says something about its maturity.

A pretty simple, and configurable one: https://github.com/casbin/casbin

Hey! And congrats, hope you're well :) The company I was previously working for is still using Commento, great job!

Wow, never thought someone would pay attention to that! I was the original contributor (https://github.com/adtac/commento/pull/109) for that change, so happy to answer any questions.

This might be a very specific question, but any plans to include packages in the Go runtime for headless Chrome? I know it is already available in Node.js.

BTW, I think it might be worth including "commands" in the usecases, and repository (looking quickly at the code on GitHub). At the moment, the only usecase is fetching data.

I've recently also come across https://pydantic-docs.helpmanual.io/ which appears to be useful for enforcing simple constraints on domain entities.

Awesome. I've always been curious over examples of clean architecture in Python. I had a cursory glance. As a suggestion, I think the `Room` domain model could be improved with dataclasses (https://docs.python.org/3/library/dataclasses.html).

I've been using a similar one: https://github.com/gopasspw/gopass. Only problem is getting non-technical people to use it is hard. And it is not super intuitive to get it working on mobile.

This is akin to the other "codemod" facilities Facebook already uses for large-scale refactoring in busy codebases (particularly, their JS codebases), but for Python (https://github.com/facebook/codemod, https://github.com/facebook/jscodeshift).

I thought it might've just been me at first since I normally use Twitter to confirm if X is down for everyone else. It doesn't help that their status page shows:

All Systems Operational

We have an issue already filed for that, and unfortunately no. That's something CSS Paged Media is supposed to solve.

That's interesting feedback! Thank you :)

We actually went with Docker for the set up because it simplified dependency management tremendously, and it allowed us to deploy on platforms like Kubernetes, Swarm, and ECS. As a plus, it gave us some confidence that if it works for us, it should work for others (obviously, we have come across cases where Docker behaves differently across platforms).

I consider asynchronous processing (in this context) as advantageous in some cases. Indeed, when we were refactoring `athenapdf`, we considered introducing a message queue for workers to pull work from, and to put back when the work completes. The problem with this however, is that we can't as easily scale horizontally (i.e. introduce node replicas behind a load balancer), as if we tried to get / update a job, we may not get the same node we originally got. I mean, the solution can be as easy as introducing a centralised message queue of sorts (or even a sticky session), but that complicates the set up process, so we decided against it.

Taken together, for our specific use cases, we believe it is a lot simpler to consume a synchronous API. No webhooks / callbacks. No polling. No concerns over acknowledgement. If a HTTP call fails, we will know about it immediately. If a complex retry mechanism is needed, we think this should be accomplished in the client application.

In the long term, I believe we should have a toolkit that can easily be plugged into a wider orchestration engine like Conductor (https://netflix.github.io/conductor/). That way, anyone can develop their own conversion process pipeline with ease.

Core developer of `athenapdf` here :)

I had a quick look at `pdf-bot`, and though we both rely on the same underlying technology (we are only just moving to headless Chromium; we were on Electron before), I believe we have slightly different ambitions with our respective project. But, I may be biased.

For example, `pdf-bot` seems to be tied exclusively to a specific converter, and storage backend. With `athenapdf` however, we are moving more, and more towards building a toolkit or rather, framework for other people to construct their own conversion processes (or even microservice)[0].

Consequently, we are working towards general abstractions like fetching, converting, and uploading, that can have different implementations (e.g. wkhtmltopdf, LibreOffice, Weasyprint, etc).

With our microservice assembly as well, we are focused heavily on ensuring we have:

1. Instrumentation, and metrics (which `pdf-bot` appears to currently lack)

2. Support for different retry mechanisms (e.g. retry using the same converter or retry using a different converter)

3. Support for multiple input MIME types

4. Synchronous API calls (`pdf-bot` appears to be mostly asynchronous, with batch processing, and callbacks)

5. Ease of installation (e.g. Docker), and configuration

We also have a CLI assembly[1] that can support custom JavaScript plugins[2] (e.g. Markdown -> PDF, Readability, etc). So you don't need to run a service or make API calls for conversions.

[0] https://github.com/arachnys/athenapdf/tree/v3/pkg

[1] https://github.com/arachnys/athenapdf/blob/v3/cmd/cli/main.g...

[2] https://github.com/arachnys/athenapdf/tree/v3/pkg/runner/plu...

We use a combination of Lambda, ECS (for slightly larger services), and Heroku to host them. And, we use Mashape's Kong in front of them for authentication / authorization, and logging (see https://blog.arachnys.com/how-we-solved-load-testing-scalabi...). We found it far easier to configure than API Gateway.