joeduffy

[Pulumi founder here] Sorry to hear you don't particularly like Pulumi---any/all feedback welcome. If nothing else, we do listen and we do try to get better. -Joe

Note that you can now use TF providers "on the fly" from Pulumi: https://www.pulumi.com/blog/any-terraform-provider. No provider bridging/conversion necessary.

I just tried and it seems to have worked (though I haven't tested any specific resources yet):

$ pulumi package add terraform-provider lxc/incus

[Joe, Pulumi Founder here.]

Said well (and thank you for being a customer and valuable member of our community!)

The analogy I draw sometimes is that our open source infrastructure as code SDK ("Pulumi") is like Git, and our commercial offering ("Pulumi Cloud") is like GitHub.

Like GitHub, the Pulumi Cloud offers valuable features that go beyond the open source project for teams looking to manage lots of projects securely at scale, but we definitely love our open source community and want folks to have the choice to use Pulumi however makes the most sense for them.

This approach also has the nice consequence that we can be fully transparent with our community at all times while also building a strong, long-term business. If a new feature is part of the infrastructure as code SDK, it's open source and free; if it's part of the Pulumi Cloud SaaS, it's part of our commercial offering. This avoids needing to do things like artificially hold back features (like open core) or violating our commitment to the open source community (like Hashi's new license).

Pulumi Founder/CEO here.

The blog post is disingenuous. We tried many times to contribute upstream fixes to Terraform providers, but HashiCorp would never accept them. So we've had to maintain forks. They lost their OSS DNA a long time ago, and this move just puts the final nail in the coffin.

Thankfully over time, they already pushed responsibility for most Terraform providers back onto their partners, so I'm hopeful the ecosystem of providers can still stay vibrant and open.

We are deep believers in open source---heck my last project at Microsoft was to take .NET open source and cross-platform, our CTO helped found TypeScript, and Pulumi is an Apache open source project---it seems HashiCorp no longer is.

As a concrete example of what this enables, the Pulumi Automation API lets you embed IaC straight into a bigger program. Folks have used this to create infrastructure-oriented SaaS products, self-service portals, and new higher level abstraction frameworks and tools, for instance, often spanning multiple cloud resource types (AWS, Azure, Kubernetes, Cloudflare, others) -- https://www.pulumi.com/automation/. Transpiling to YAML and handing it over to CloudFormation is clumsy and wouldn't work for these cases among others.

Pulumi | Many Roles, Engineers, Engineering Managers | Remote | https://www.pulumi.com/careers/

Pulumi is an open source infrastructure as code platform enabling you to program the cloud in your favorite languages. We help developers and infrastructure teams build better software together. Pulumi is polyglot in many ways, so you'll have the opportunity to work with Go, Node.js, Python, and other language ecosystems, on a daily basis; Pulumi is also multi-cloud, meaning you'll also get to work with Kubernetes, serverless, AWS, Azure, and much, much more. We have open roles at all layers of our stack, including our open source platform in addition to our SaaS application. We are a remote team, growing fast, and also hiring engineering managers. If this sounds fun, apply online, or email me at joe AT pulumi DOT com!

[caveat, Pulumi co-founder here]

Pulumi is fully functional in open source form. The analogy I like to draw is Git and GitHub. You can use Git fully independent of GitHub, or you can choose to use them together, for a seamless experience within a team. (Not a perfect analogy since we built both Pulumi open source and the Pulumi SaaS, which causes this very confusion!) We don't hold anything back, if it's in the SDK, it's open.

We recently added concurrency control to the alternative backends. I'm sorry the docs are confusing on this matter -- we will get that fixed up. We also have many large customers in production on the open source alone. It's easier with the SaaS just because we handle security, reliability, and sharing with your team along with access controls, auditing, etc. But if you prefer to roll your own there we are entirely happy to have you in the community and help out. Admittedly our marketing materials aren't super clear here and we are working to fix this.

Hope this helps to clear things up and again apologies for the confusion.

[caveat, Pulumi co-founder here]

You are right that it's not easy. Thankfully the cloud providers themselves have moved in the direction of auto-generation for their own SDKs, documentation, etc., which has forced this to get better over time. This is motivated by much the same reason we've done it -- keeping one's own sanity, ensuring high quality and timeliness of updates, and ensuring good coverage and consistency, when supporting so many cloud resource types across many language SDKs.

Microsoft, for instance, created OpenAPI specifications very early on, and standardized on them (AFAIK, they are required for any new service). Those API specifications contain annotations to describe what properties are immutable (as you say, the need to "re-create" the resource rather than update it in place). Google Cloud's API specifications similarly contain such information but it's based on the presence of PATCH support for resources and their properties. Etc.

The good news is that we've partnered with the cloud providers themselves to build these and we expect this to be increasingly the direction things go.

Excellent topic. [caveat, Pulumi co-founder here]

Indeed what you say is true of many other "multi-language" platforms. I was an early engineer on .NET at Microsoft, and although it was multi-language from the outset (COBAL.NET was a thing!), the reality is most folks write in C# these days. And yet, you still see a lot of excitement for PowerShell, Visual Basic, and F#, each of which has a rich community, but uses that same common core. A similar phenomenon has happened in the JVM ecosystem with Java dominating most usage until the late 2000s, at which point my impression is that Groovy, Scala, and now Kotlin won significant mindshare.

I have reasons to be optimistic the infrastructure language domain will play out similarly. Especially as we are fundamentally serving multiple audiences -- we see that infrastructure teams often elect Python, while developers often go with TypeScript or Go, because they are more familiar with it. For those scenarios, the new multi-language package support is essential, since many companies have both sorts of engineers working together.

A "default language for IaC" may emerge with time, but I suspect that is more likely to be Python than, say, HCL. (And even then, I'm not so sure it will happen.) One of the things I'm ridiculously excited about, by the way, is bringing IaC to new audiences -- many folks learn Python at school, not so much for other infrastructure languages. Again, I'm biased. But, even if a default emerges, I guarantee there will be reasons for the others to exist. I for one am a big functional language fan and especially for simple serverless apps, I love seeing that written in F#. And we've had a ton of interest in PowerShell support since many folks working with infrastructure for the Microsoft stack know it. And Ruby due to the Chef and Puppet journeys.

I also won't discount the idea of us introducing a cloud infrastructure-specific language ;-). But it would be more general purpose than not. I worked a lot on parallel computing in the mid-2000s and that temptation was always there, but I'm glad we resisted it and instead just added tasks/promises and await to existing languages.

As to the Pulumi schema, you're right, that's a step we aim to remove soon. For TypeScript, we'll generate it off the d.ts files; for Go we'll use struct tags; and so on. Now that the basic runtime is in place, we are now going to focus there. This issue tracks it: https://github.com/pulumi/pulumi/issues/6804. Our goal is to make this as ridiculously easy as just writing a type/class in your language of choice.

The defining characteristic about Pulumi compared to other tools is that it's not a transpiler, in fact. It's a multi-language runtime written in Go that can host many language plugins (Node.js, Python, .NET, Go, etc), as well as many resource provider plugins (native ones, OpenAPI-based, Terraform-based). So although yes it can use Terraform providers -- great for coverage across many infrastructure providers as well as easy portability if you're coming from Terraform/HCL -- it's not correct to say that it's "just using Terraform" or is a "transpiler".

Pulumi is open source on GitHub if you want to check it out: https://github.com/pulumi/pulumi.

Pulumi (https://pulumi.com) | Remote (Seattle HQ) | Full Time

Pulumi is a cloud engineering startup whose flagship infrastructure as code platform helps infrastructure teams and developers create modern cloud architectures. Our open source SDK uses general-purpose languages and ecosystems of tools to deliver apps and infrastructure across any cloud -- including AWS, Azure, Google Cloud, and Kubernetes -- with software engineering practices like testing, sharing and reuse, and more. We offer a SaaS for teams and enterprises using this SDK at scale.

If you enjoy working at the intersection of cloud infrastructure and developer platforms, you'd love it here!

We have several positions open, including:

* Platform Engineer (Go/Python/JavaScript/TypeScript/C#)

* Full-Stack Engineer (JavaScript/TypeScript/Angular/Go)

* Customer Engineer (pre- and post-sales for AWS, Azure, Kubernetes, and more)

Our careers page has more details: https://pulumi.com/careers. Feel free to apply there or just email joe at pulumi dot com (I'm a founder).

is it possible to do that without having to use the CLI? Are you doing any sort of state locking here?

Yeah it's just a file if you prefer to edit it. By default, Pulumi uses our hosted service so you don't need to think about state or locking. That said, if you don't want to use that, you can manage state on your own[1]. At this time, you also need to come up with a locking strategy. Most of our end users pick the hosted service -- it's just super easy to get going with.

Do you have any plans of releasing something like the Terraform Registry to help with discoverability?

I expect us to do that eventually, absolutely. For us it'll be more of an "index" of other package managers since you already have NPM and PyPI, etc. But definitely get that it's helpful to find all of this in one place -- as well as knowing which ones we bless and support.

Also, do you have any docs on writing providers?

We have boilerplate repos that help you get started:

1) Native providers: https://github.com/pulumi/pulumi-provider-boilerplate

2) Terraform-based providers: https://github.com/pulumi/pulumi-tf-provider-boilerplate

These packages are inherently multi-language and our code-generator library will generate the various JavaScript, Python, Go, C#, etc, client libraries after you've authored the central Go-based provider schema.

Is that the preferred workflow here for providers that already exist in the Terraform ecosystem?

Yes. We already have a few dozen published (check the https://github.com/pulumi org when in question). In general, we will support any Terraform-backed provider, so if you have one that's missing that you'd like help with, just let us know. We have a Slack[2] where the team hangs out if you want to chat with us or the community.

[1] https://www.pulumi.com/docs/intro/concepts/state/#self-manag...

[2] https://slack.pulumi.com

You're right, the Pulumi example is a project, not a reusable module. There are a few approaches to making it modular:

1) The project does support config. So if you want to change (e.g.) the number of AZs, you can say

    $ pulumi config set numberOfAvailabilityZones 3
    $ pulumi up

2) You can make this into a library using standard language techniques like classes, functions, and packages. These can use a combination of configuration as well as parameterization. If you wrote it in Python, you can publish it on PyPI, or JavaScript on NPM, or Go on GitHub -- or something like JFrog Artifactory for any of them. This makes it easy to share it with the community or within your team.

3) We offer some libraries of our own, like this one: https://github.com/pulumi/pulumi-awsx/tree/master/nodejs/aws.... That includes an abstraction that's a lot like the Terraform module you've shown, and cuts down even further on LOC to spin up a properly configured VPC.

I am a big Go fan too, so I very much know what you're saying. (In fact, we implemented Pulumi in Go.) Even with Go, though, you've got funcs, structs, loops, and solid basics. Simply having those goes a long way -- as well as great supporting tools -- and you definitely do not need to go overboard with abstraction to get a ton of benefit right out of the gate.

Again, I'm biased and YMMV :-)

[Disclaimer: also member of the Pulumi team.]

We've worked with a lot of end users to migrate from Terraform, and we honestly do see a lot of copy-and-paste. I agree that it's not as rampant as with YAML/JSON, however, in practice we find a lot of folks struggle to share and reuse their Terraform configs for a variety of reasons.

Even though HCL2 introduced some basic "programming" constructs, it's a far cry from the expressiveness of a language like Python. We frequently see not only better reuse but significant reduction in lines of code when migrating. Being able to create a function or class to capture a frequent pattern, easily loop over some data structure (e.g., for every AZ in this region, create a subnet), or even conditionals for specialization (e.g., maybe your production environment is slightly different than development, us-east-1 is different, etc). And linters, test tools, IDEs, etc just work.

For comparison, this Amazon VPC example may be worth checking out:

- Terraform: https://github.com/terraform-aws-modules/terraform-aws-vpc/b...

- Pulumi (Python): https://github.com/joeduffy/pulumi-architectures/blob/master...

- CloudFormation: https://github.com/aws-quickstart/quickstart-aws-vpc/blob/ma...

It's common to see a 10x reduction in LOCs going from CloudFormation to Terraform and a 10x reduction further going from Terraform to Pulumi.

A key importance in how Pulumi works is that everything centers around the declarative goal state. You are shown previews of this (graphically in the CLI, you can serialize that as a plan, you always have full diffs of what the tool is doing and has done. This helps to avoid some of the "danger" of having a turing-complete language. Plus, I prefer having a familiar language with familiar control constructs, rather than learning a proprietary language that the industry generally isn't supporting or aware of (schools teach Python -- they don't teach HCL).

In any case, we appreciate the feedback and discussion -- all great and valid points to be thinking about -- HTH.

Our free tier is still there and here to stay. What did we do to make you think it's been sunsetted? :-(

We have ksonnet expats on the team (we're all in cloud city -- Seattle), and I've been keeping an eye on that project myself, since I think it got a lot of things right and frankly many of the ideas for Pulumi were inspired by early chats with the Heptio team. But, as you say, why create a new language when an existing one will do -- that was our original stance and it's working great in practice.

Joe Beda will be doing a deep dive on Pulumi on the TGIK videocast tomorrow, so it's a timely opportunity to check it out: https://twitter.com/jbeda/status/1092963296565587969

Definitely. I was a part of C# in the early days, so little else would make me happier than awesome class .NET support. This'll be great for Azure folks -- who knows, PowerShell too?

We are actively working on https://github.com/pulumi/pulumi/issues/2430, which will make it easier for our small team to manage multiple languages. Once that lands, I would expect this to be high priority.

Some of our amazing community members have been prototyping this, and it's looking pretty promising: https://twitter.com/MikhailShilkov/status/109278757393889689....

Joe

My belief is that we've been slowly building up to using general purpose languages, one small step at a time, throughout the infrastructure as code, DevOps, and SRE journeys this past 10 years. INI files, XML, JSON, and YAML aren't sufficiently expressive -- lacking for loops, conditionals, variable references, and any sort of abstraction -- so, of course, we add templates to it. But as the author (IMHO rightfully) points out, we just end up with a funky, poor approximation of a language.

I think this approach is a byproduct of thinking about infrastructure and configuration -- and the cloud generally -- as an "afterthought," not a core part of an application's infrastructure. Containers, Kubernetes, serverless, and more hosted services all change this, and Chef, Puppet, and others laid the groundwork to think differently about what the future looks like. More developers today than ever before need to think about how to build and configure cloud software.

We started the Pulumi project to solve this very problem, so I'm admittedly biased, and I hope you forgive the plug -- I only mention it here because I think it contributes to the discussion. Our approach is to simply use general purpose languages like TypeScript, Python, and Go, while still having infrastructure as code. An important thing to realize is that infrastructure as code is based on the idea of a goal state. Using a full blown language to generate that goal state generally doesn't threaten the repeatability, determinism, or robustness of the solution, provided you've got an engine handling state management, diffing, resource CRUD, and so on. We've been able to apply this universally across AWS, Azure, GCP, and Kubernetes, often mixing their configuration in the same program.

Again, I'm biased and want to admit that, however if you're sick of YAML, it's definitely worth checking out. We'd love your feedback:

- Project website: https://pulumi.io/

- All open source on GitHub: https://github.com/pulumi/pulumi

- Example of abstractions: https://blog.pulumi.com/the-fastest-path-to-deploying-kubern...

- Example of serverless as event handlers: https://blog.pulumi.com/lambdas-as-lambdas-the-magic-of-simp...

Pulumi may not be the solution for everyone, but I'm fairly optimistic that this is where we're all heading.

Joe

Pulumi | Developer Advocate, Software Engineers | Seattle, WA or Remote (US) | Full Time | https://pulumi.io

Pulumi is a VC funded open source cloud company in Seattle, WA. Pulumi lets you write infrastructure as code in your favorite language (Go, JavaScript, TypeScript, Python, ...), and supports many clouds (AWS, Azure, GCP, Kubernetes, ...). We have a complementary SaaS product that helps with continuous deployment and management of your cloud apps and infrastructure.

The heart of the system -- the engine and its providers -- is written in Go, but being a multi-language system, we all program in many languages regularly. The SaaS product is a modern Angular app written in TypeScript. Our team is a unique combination of language and cloud geeks, and after just launching out of stealth, we are growing in all areas of the company.

More info at https://www.pulumi.com/careers/ or just email jobs@pulumi.com.

congrats on the launch.

Thanks -- we're very excited to be out and talking to the community!

What is difference in the use case for Pulumi compared to scripting infrastructure using vendor provided libraries

The major difference is that Pulumi does immutable infrastructure. The code describes a goal state and our system manages your environment to ensure that it matches the goal state. This means you can preview changes before making them, and that, if you do make them, you've got a full audit trail of who changed what and when. Rollbacks are trivial because you just point us at an old goal state worked and we can chew on it until your live environment matches. We can even detect drift, so if someone manually updates in the cloud console, we can tell you about it. As a result, we always have a full object graph of your cloud resources, and can tie it back to the source code which created it, opening up some unique diagnostics capabilities.

The difference with scripting libraries, like AWS's Boto, or the Azure PowerShell SDK, is that they mutate cloud resources directly, and in an ad-hoc manner. So, you don't know what they are going to do before you run them. And in the event of failure, you're more likely to be in a corrupt state and unable to recover. Rollbacks are difficult. There's also no audit trail beyond the cloud access logs, and information like dependencies are lost, so resources end up disassociated from the code and workflow that created or updated them. Many people encounter these problems and need to build complex systems on top to address them. Or they end up using a solution like CloudFormation or Azure Resource Manager, which eschews code in favor of JSON/YAML/templates.

The hope here is that we've landed on a sweet spot between immutable infrastructure and fully programmable languages.

Pulumi doesn't seem to be designed to support deployment onto bare metal.

In principle, there is no reason we couldn't support bare metal, provided we have an orchestrator. Pulumi bottoms out on raw CRUD operations that can do anything. It turns out that, when targeting existing cloud providers, that means invoking their APIs. As you say, this is just a layer of abstraction, but the key aspect at the center here is diffable immutable infrastructure.

I was, in fact, inspired to pursue Pulumi in part because of the experience of building a distributed OS that ran on bare metal. Many of the same challenges Pulumi solves around configuration, composition, and the application model, were inspired by this experience, and we've taken some of those lessons and applied them to cloud programs.

you can keep more of the abstraction in the language, and thereby avoid complexity in the underlying architecture.

I look at other ways we've integrated with systems as inspiration here. For instance, we can productively code using tasks, and async await, these days, and reap the benefits of concurrency, despite there being massive differences between threading, I/O, and scheduling behavior across different systems. There was abstraction in the language, but it bottoms out on a set of primitives.

In a sense, I view the current APIs we're using as the syscall layer for a given cloud, Kubernetes included. (Also note, we don't require depending on Kubernetes.) That's not to say everybody will want to eschew the syscalls -- there's always a time and a place -- but we can be more productive by using the abstractions.

I believe that frameworks are languages, in that they give you a set of nouns and verbs you can use to express yourself. I do think there's opportunity for language-level innovation here, but if there's one thing I've taken away from our industry experiences building PaaS's, it's that it's a losing battle to bet against the incredible momentum of the public cloud vendors.

I'd be much more interested to see an attempt to replace the Kubernetes API with a library.

We are just getting started, so expect to see a lot more progress in this direction in the weeks to come. The foundation of languages is a huge enabler for us, and we already have several projects in the works that I think will make you happier than this initial release.

YMMV, this is just my opinion, and I really love hearing the feedback. Thank you for sharing. This is a vibrant space and time and I'm enjoying the diversity of perspective.

The strength of this system is that map is type checked to be provably side-effect free. The mapping function is free to internally allocate and mutate objects to get its job done, e.g. a RegexMatcher, but it may not mutate existing (non-isolated) state. This composes transitively, so even wrapping side-effects in abstractions and closures is insufficient to hide them.