Yml Coding

https://cloud.google.com/workflows/docs/reference/syntax

by undefinednull • 6 years ago

97 82 6 years ago

cloud.google.com

tuankiet65• 6 years ago

Trying to turn YAML into a programming language reminds me of Ansible playbooks.

dhosek2• 6 years ago

We went through great lengths to be able to do parameterized YAML using jinja2 templates. I designed a system that let us effectively create a new yaml out of two other yamls which let us handle repetitive configuration tasks reasonably well but there were occasional unexpected challenges thanks to the syntactically significant white space of yaml.

hbogert• 6 years ago

sincerely, why do this in the first place?

IshKebab2• 6 years ago

That sounds ridiculous.

kabacha• 6 years ago

Isn't that what Helm[1] is?

1 - https://helm.sh/

oblio• 6 years ago

http://ant-contrib.sourceforge.net/tasks/tasks/if.html

taylorlapeyre• 6 years ago

I am surprised at how well this actually reads!

perfunctory2• 6 years ago

And here we come. Google, in the 21st century, is trying to sell us BASIC

  - define:
    assign:
        - array: ["foo", "ba", "r"]
        - result: ""
        - i: 0
  - check_condition:
    switch:
        - condition: ${len(array) > i}
            next: iterate
    next: exit_loop
  - iterate:
    assign:
        - result: ${result + array[i]}
        - i: ${i+1}
    next: check_condition
  - exit_loop:
    return:
        concat_result: ${result}

edit: it's not April 1st yet, is it?

recuter• 6 years ago

Come on, this is bound to revolutionise your Kubernetes configuration.

baq4• 6 years ago

i've seen this before, as has my father and probably my grandfather, except it was thought out better and was called lisp instead of yaml.

mahmoudimus• 6 years ago

I completely agree with this statement and made a similar realization as well. Basically, modern devops and application development boils down to writing Lisp in YAML.

Config languages for Ansible, K8S, etc.are just basically a bad implementation of a Lisp-ish language.

qppo• 6 years ago

Well at least it's NoCode

1337shadow• 6 years ago

I fail to see how better it is than:

    array = ["foo", "ba", "r"]
    result = ""
    i = 0
    while len(array) > i:
        result = result + array[i]
        i += 1

exactly 3 times less code

orf4• 6 years ago

I like this. Go "har har har yaml" all you like, but this reads a damn lot better than AWS step functions and seems way more powerful.

There are a number of tasks that benefit from something like this, and there are a huge number of advantages from being able to encode steps in domain specific languages such as YAML and have them execute with truly "no code".

baq4• 6 years ago

yaml is not a language. it's a serialization format.

this thing here is a language that serializes into yaml. there are zero benefits of using it except perhaps 'it is yaml so i don't have to compile my configuration'.

note that xml with xslt would have handle it better.

gnulinux2• 6 years ago

Well, you don't have to write a parser. The parser is already written for you (yaml parser). But I have no idea why that'd be an advantage, especially for Google, they surely know how to write a parser...

orf4• 6 years ago

there are zero benefits of using it

There are many, many benefits from having steps defined in a restrictive, tightly-controlled language that doesn't require sandboxing.

There are also many benefits to not using XML and XSLT, not least of all user experience.

osmarks7• 6 years ago

This is not "no code". This is code, imperative code even, in a really bizarre format.

orf4• 6 years ago

You're not wrong, but there is always going to be code involved when instructing a machine to perform some actions. By "no code" (and the reason it was in quotes) was that it's almost no superfluous code at all. No libraries to bundle, functions to define, bundles to upload, etc.

"I want to call an API, and if it's a Tuesday and the response contains 'foo' trigger another API" seems pretty simple with this. That's not to say it wasn't "simple" before, but there was definitely _more code_.

abeppu• 6 years ago

I get the appeal of something like this. It's locked down enough that it's perhaps challenging to do something really unsafe with it. It's maybe usable by someone who doesn't feel comfortable with a "real" programming language.

But it seems like this falls apart as soon as one service you need to interact with creates a requirement not anticipated by this very constrained tool-set. You need to query service A, extract something with a regex, base64 encode something else before you post to service B? Well we didn't include regexes, a module/import system, or the ability to introduce UDFs in a different language.

And if you had the resources to make all your services play into the expectations of this workflow system, you might not need to use this workflow system.

rightbyte• 6 years ago

I don't see how writing a kinda abstract syntax tree with the opacity of Enterprise JavaBeans is easier than say writing the workflow in some Basic dialect with only GOTOs and IFs.

dpc_pw• 6 years ago

Peak Yaml?

throwaway4889• 6 years ago

Yaml is the new xml

IshKebab2• 6 years ago

I dunno, XML may have been verbose and overused but at least it had a fairly solid design. YAML is terrible.

zo1• 6 years ago

I always saw it as "JSON was the new XML". And JSON Schema was the new XSD. And then they went ahead and also re-invented SOAP using "Swagger" / OpenAPI.

Luckily, JSON/YAML is mostly interchangeable these days as it's just a nested hierarchy of a few basic types. Heck, I mostly treat XML the same way as well.

osmarks7• 6 years ago

This seems like some sort of horrible inner platform effect. It would be nice if they would use an actual embeddable programming language, like Lua.

baq4• 6 years ago

turing complete configuration? no thanks. i like it when my configs are guaranteed to halt.

osmarks7• 6 years ago

Fair, and there's this neat Dhall (https://dhall-lang.org/) thing for config files (not really this, since it seems like it is designed to actually encode some amount of logic) which enforces that while still being pretty programmable. This might actually already be TC, though; at the very least it is able to do infinite loops.

perfunctory2• 6 years ago

what makes you think this yml language is not turing complete?

ufo• 6 years ago

It is possible to use Lua hooks to protect against programs that accidentally take too long too run. Lua can be told to stop the script after so many steps.

This technique isn't fool proof against maliciously crafted programs, but it might be sufficient depending on what you are doing.

amyjess• 6 years ago

Lua isn't homoiconic, though.

This is basically Lisp without the parentheses.

osmarks7• 6 years ago

This doesn't have any of what makes lisps nice, in that you cannot, as far as I can tell from those docs, operate on the data structures the code consists of at all.

Jtsummers3• 6 years ago

This language isn’t a lisp, nor is it homoiconic. It’s just a tree-structured language.

It may be fair to compare the syntax and structure to s-expressions, but not to lisp itself. In the same way this language could as easily be expressed as XML.

keymone• 6 years ago

This is basically half assed bug ridden implementation of half of a lisp.

thangalin• 6 years ago

Tangentially related is my YAML pre-processor that performs string interpolation:

https://bitbucket.org/djarvis/yamlp/

Along with Pandoc, it allows common prose from Markdown to be de-duplicated, as described in my Typesetting Markdown series:

https://dave.autonoma.ca/blog/2019/07/06/typesetting-markdow...

Obviously it's laborious insert to YAML keys everywhere, so I'm developing an editor that integrates YAML and plain text document formats (such as Markdown):

https://www.youtube.com/watch?v=u_dFd6UhdV8

jpxw• 6 years ago

Even writing CI configuration is onerous enough in YAML, let alone a program

wickedOne• 6 years ago

despite the readability of the yaml syntax, for me the big disadvantage is the inability to use meta / type info.

for a straightforward collection of strings: probably good enough, for anything else xml is more likely to cover the usecase better

Spivak3• 6 years ago

YAML has really good typing support with type annotations being a first class citizen. But you’re dependent on the hosting software (adding or understanding) the types.

For example

    start: !date 2020-09-12

could actually map to a native date object.

bonestormii_2• 6 years ago

Data is data. Code is logic. So the the thought process goes...

...what if I could store my configuration (data! right?!) in a way that is nicely separated from the logic?! No more scripting for me!

Oh wait. Some of the configuration can't be generalized about universally. Configurations fundamentally contain logic, I guess.

Oh, well then why don't I just represent the logic in the data?! That will be much better than representing the data in the logic!

....but now, you are back where you started, only instead of using something nice, standard, and powerful like Python, you have to use this... language... thing. This YAML convention you cooked up.

Separate the data from logic. Read the data into the logic. Make a nice organized place to call custom logic from... like, you know, a file directory full of scripts, which are called according to some scheme. It could be another data file. Stop there.

Like this:

    $ ls -Ra
    ./config/do_something.yaml
    ./config/do_something_else.yaml
    ./config/config.yaml
    ./logs/ping.log
    ./scripts/do_something.py
    ./scripts/do_something_else.py

---

    $ cat ./config/config.yaml

    do_something:
        target: my.stupid.server
        exec_frequency: daily
        ...
    do_something_else:
        target: my.stupid.server
        exec_frequency: monthly
        ...

---

    $ cat ./config/do_something.yaml

    ping: true
    output: 
       directory: ./logs/ping.log
       mode: append

---

    $ cat ./config/do_something_else.yaml

    delete: ./logs/ping.log

---

    $ cat ./scripts/do_something.py

    from stupid.library.task import Task, subscribe

    class DoSomething(Task):

        @subscribe
        def ping(self, target):
            return super().ping(target)

Then you code the program that runs collects all of the task methods, put them in an ordered list, and run them if the conditions in the config.yaml is met.

Every other thing is done in a task method in python, or something like it. I don't think we need more abstraction than that.

pydry2• 6 years ago

This is more or less the structure I followed with this testing framework: https://github.com/hitchdev/hitchstory

The whole idea being that you don't want the story to be turing complete (there are no loops or conditionals with a story), but the code that executes it needs to be turing complete.

bonestormii_2• 6 years ago

Right, and it's not like it's ground breaking thinking here. It's just a matter of some personal restraint on your part to say, "No, I will not turn python into an interpreter for some disgusting yaml-based language."

Everyone agrees we hate this, until one day it's us who is dreaming it up, and we just can't resist! There should be no concept of "flow of execution", except for maybe expansion of previously declared variables just so you can do something like concatenating two lines together. Otherwise, the lack of logical operations will make even that unnecessary.

xiaq6• 6 years ago

Saying this is reinventing BASIC in YAML seems to miss the point here; the programming model here is essentially a finite state machine with optional data flowing between nodes, and AFAIK there isn't really a widespread language that targets this model.

And a restricted model is not there just so that non-programmers can use it. Restricting what you can do in the program means that there are more things you can do with the program. I haven't checked if GCP's Workflows supports everything below, but here are some things you can do in principle:

* You can visualize the entire program as a flowchart, and visualize the state of the program simply by pointing at a node in the flowchart. This is not possible with a general purpose language since there could be arbitrary levels of call frames.

* You can implement retry policies for each step entirely transparently, and possibly other things like authentication. Aspect-oriented programming is more practical when the programming model is restricted.

* You can schedule the steps onto different hosts, possibly in parallel.

* You can suspend and resume the workflow, since its entire state is just which step is being executed, plus a handful of variables (which presumably are always serializable).

Re the problem of extension: the idea seems to be that you put all "smartness" inside HTTP services that are written in real languages and only use this as a dumb glue language.

mfateev6• 6 years ago

Look at my project temporal.io. It does the same thing using general purpose programming languages. Currently Java and Go are supported with Python, Ruby and C# under development. This is absolutely valid workflow code:

     public void execute(String customerId) {
       activities.onboardToFreeTrial(customerId);
       try {
         Workflow.sleep(Duration.ofDays(60));
         activities.upgradeFromTrialToPaid(customerId);
         while (true) {
           Workflow.sleep(Duration.ofDays(30));
           activities.chargeMonthlyFee(customerId);
         }
       } catch (CancellationException e) {
         activities.processSubscriptionCancellation(customerId);
       }
    }

xiaq6• 6 years ago

But what this does under the hood is still sending instructions to a workflow engine that does the actual work, as opposed to this code doing the actual work directly, right?

If that's the case, then what you could do is still restricted by the protocol of the workflow engine - building the instructions in code gives you some dynamicism, but not a whole world of difference, and complicates things that are easier done statically. It is definitely a valid approach, but it doesn't invalidate the approach of writing out the workflow definition statically, especially if the paradigm is "put all smartness inside HTTP service and only use the workflow as a glue".

lowercase12• 6 years ago

Have you moved on from Cadence to this? What differences are there? Looks interesting.

cle• 6 years ago

This looks very similar to Amazon Simple Workflow, which has been effectively superseded by Step Functions, which uses static workflow definitions that look very similar to GCP Workflows.

Why do you think AWS moved away from this model?

lowercase12• 6 years ago

This is similar to AWS step functions which kinda descended from more general/powerful AWS Simple Workflow.

roenxi• 6 years ago

That is good, but why code all that up in YAML? This looks to me like a programming language with great efforts made to obfuscate the fact.

We've had 70 years to figure out what good programming language layout looks like. I think there is almost a consensus that it doesn't look like the linked code-wearing-a-fake-moustache .

Are people supposed to code these state machines using some sort of real language then translate it to YAML?

xiaq6• 6 years ago

Yeah, I agree that YAML is not the best language for embedding a DSL, but in general GCP seems to have a preference for using YAML in its configuration languages.

I can imagine that the developers chose YAML based on its popularity elsewhere (especially in the CI/CD space), not its merits.

(Oh, and a belated disclaimer: I work for GCP but not on this product. All opinions are my own, obviously.)

indymike• 6 years ago

Actually, it isn't even as capable as BASIC, but it is very cool and probably really useful.

xiaq6• 6 years ago

A belated disclaimer: I work for GCP, but not this product.

cordite• 6 years ago

How about we use resumable lua scripting

swiley2• 6 years ago

IMO: most of these weird DSLs come from wanting a data structure but getting confused and writing a bad programming language instead.

jpalomaki• 6 years ago

Do they have a graphical tool for defining these workflows? For me this looks like the business process management tools from the past. With those the idea was not to use them to write program logic, but connect the dots on very high level. See for example the use case pictures at [1].

[1] https://cloud.google.com/workflows

fmakunbound• 6 years ago

Good god it’s stuff like this makes me contemplate leaving professional programming.

Yml Coding

Related Stories