Pulumi 3.0

https://www.pulumi.com/blog/pulumi-3-0/

by bovermyer • 5 years ago

321 136 5 years ago

Can someone who has used both Pulumi and AWS CDK describe the differences between the two?

I am using Terraform 100% now, but sometimes wish I had more than the HCL (hashicorp configuration language) syntax available to use in my code.

aequitas• 5 years ago

You could try Terraform CDK[0] which supports TypeScript, Python, Java, and C#.

[0] https://github.com/hashicorp/terraform-cdk

k__2• 5 years ago

The CDK synthesizes CloudFormation templates (for AWS) and Pulumi seems to use the AWS APIs to provision and doesn't touch CloudFormation.

bloopernova2• 5 years ago

Right - I was wondering about the developer "experience" between the two.

satya712• 5 years ago

CDK maintains the state with AWS itself, Pulumi operates their own state service (with option of self management). I have never used CDK, but I've heard that it's hard to recover from state corruption in CDK.

jen202• 5 years ago

While this is true, the reason is not called out: CDK simply generates a CloudFormation Template, and thus uses CloudFormation for state management, rather than using the cloud APIs directly.

nhoughto• 5 years ago

CDK is just a different way of writing cloudformation essentially, its code that is transformed into cloudformation templates, so inherits all its problems.

Pulumi is more like Terraform but without HCL. So imagine instead of having to find-the-HCL-way-to-do-things and work within those constraints to make terraform work, you can use one of a few languages to build up resources (similar to you would in HCL) and have pulumi provision/act on them for you including managing their state.

Quite different to CDK/CloudFormation and much more like Terraform without the handcuffs of HCL. (sometimes handcuffs are useful tho..)

mdeeks2• 5 years ago

In addition to what others have said, it is important to point out that the CDK is AWS specific. Your "infrastructure" is rarely just "AWS" even if you're not multi-cloud. Terraform and Pulumi can operate on things outside of AWS like Github, Okta, Cloudflare, Datadog, Hashicorp Vault, etc.

It is nice to have one system or even repo for managing all of that.

https://www.pulumi.com/docs/intro/vs/cloud_template_transpil...

cmclaughlin4• 5 years ago

In addition to what others have said, it is important to point out that the CDK is AWS specific.

As mentioned in other comments here, CDK for Terraform is another option and it's not AWS specific - it should work with any Terraform provider and there are heaps available.

https://github.com/hashicorp/terraform-cdk

rossmohax• 5 years ago

I used both in depth. CDK as it is now is a wrapper around CloudFormation and therefore suffers from same limitations: AWS only, 'static' templates.

Pulimi on the other hand is closer to terraform, except you can write in a language which actually doesn't stand in your way and allows you to programmatically access values available during execution only. With CDK you can only reference them in CF template (lets say instance ID which you just created), but these are never "lifted" to your code. Also CDK misses native 'datasources' and offers limited mechanism for lookups.

Pulumi also has a killer Automatioin featre, where you can code infrastructure migrations, not unlike you'd do it for SQL. Neither TF nor CDK allows you to do that, and you'd need to code infrastructure state transitions in error prone bash wrappers.

As a developer, I quite liked Pulumi resource model, documentation and team responsiveness on Github and slack channel. I am working with CDK now, because of costs and I'd prefer Pulumi if there was a choice.

tasssko• 5 years ago

When i looked the SDK was only available in certain languages which was a issue for me at the time. I just checked and they have python and csharp. Pulumi has potential to reduce the verbosity of infrastructure configuration and enable engineers to tailor it more to the application on demand. Adding a new queue can be done as part of a app release and not handled in a separate pipeline. It isn’t fun to wrangle infrastructure pipelines and when necessary orchestrate changes.

fulafel• 5 years ago

The migration style changes sound really interesting.

CDK indeed is just another layer of abstraction on top of CloudFormation, itself already a leaky complex abstraction, and it bites you in a lot of ways.

pm903• 5 years ago

The migration as code feature sounds the most promising from your comment. I use terraform heavily, and while its good to capture infra state, there's no good mechanism to automate infrastructure migration from one state to another in code. I think this would be incredibly valuable to capture that.

I must admit that I'm skeptical on how well this would actually work though. Infrastructure API's tend to be flaky, and I'm not sure how reusable the migration code would actually be. If its too flaky, people will just not use it.

junon• 5 years ago

Thanks for the writeup. Terraform has only ever worked against me, and hadn't heard of Pulimi until now.

Will have to look into it now.

carlosf• 5 years ago

I migrated from Terraform to AWS CDK and I'm very happy.

The good:

- Tooling for any major language (Typescript, Python)… is lightyears ahead of anything you will find for Terraform / HCL.

- You can write your code as declaratively as possible (like you would do using TF), but you always have the escape hatch of using all libs available to your language of choice.

- AWS CDK uses CloudFormation under the hood, so you get cool stuff like automatic rollbacks in case of failure.

- You have access to much more mature testing frameworks, compared to what is available to TF. Because AWS CDK synthetize CF templates, you can also snapshot those for regression testing. Applying good software engineering practices is overall much easier. Most languages are much easier to extend than HCL.

- AWS is constantly releasing higher level libraries so you don't need to fiddle with low level API details. When using Terraform, you generally must understand your cloud provider API at a very fine level to implement IaC.

The bad:

- AWS CDK is written in Typescript and automatically translated to other languages. Even though you can use Python, C#, etc... Most examples and tutorials exist in Typescript. Also, you always must install npm + cdk + the libraries in the language you are using, so using any language other than Typescript means supporting two toolchains, which is a pain in the ass. I started with AWS CDK in Python and now I'm migrating to Typescript.

- Some modules only exist for Typescript.

- Since CF is used under the hood, it only supports resources supported by CF. Sometimes CF support takes a while. Since TF is just a wrapper for a Cloud Provider API, it generally implements new resources much quicker.

- It has its own vocabulary and and ways to do stuff (L1/L2 Construct? Stack? Retention Policy?)... even if you've used a lot of TF and know AWS very well, there's a learning curve.

Some cool videos:

- An AWS dev shows how to create your own construct and test it (that's the less "toy example" tutorial I have found): https://www.youtube.com/watch?v=cTsSXYOYQPw

- Same guy shows how to contribute to AWS CDK. I've found that looking at their source code is a great way to learn about good practices and patterns: https://www.youtube.com/watch?v=OXQSSibrt-A

manigandham• 5 years ago

All those frameworks have their own DSL/config languages that lack expressiveness and inevitably become a really bad scripting language to support everything people want to do.

Pulumi started with actual programming languages instead which gives you full expressiveness, IDE integration, and more, without learning yet another new language. And since infrastructure is just represented as objects in the language, creating and combining them in complex ways is much more natural than wiring up configs.

methyl• 5 years ago

Native Provider means it no longer uses Terraform under the hood and it excites me a lot. We were having hard time with some more complex setups and it seems it can be remedy for issues that we faced. Stoked to check it out!

leg1004• 5 years ago

According to [1], "Native providers are built directly and automatically from the cloud provider’s API...", and are not hand-coded, unlike Terraform's providers.

I'm surprised this is possible. If it was, why didn't Terraform follow this approach a long time ago?

The cloud providers' APIs provide the endpoints for creating, updating, reading and deleting resources. Terraform and Pulumi's value is to provide an idempotent abstraction on top of that. And that abstraction is not straightforward to write, because it has to handle numerous nuances and anachronisms in the underlying APIs. For example, in the event of updating an existing storage bucket, the abstraction has to determine whether the bucket can be simply updated, or if it needs to be re-created (say if you were changing the name or location). And the underlying API will not necessarily reveal this kind of information.

Hence one would think completely avoiding handwritten code is incredibly difficult if not an insurmountable problem.

[1] https://www.pulumi.com/blog/pulumiup-native-providers/

k__2• 5 years ago

Wouldn't this mean that Pulumi has better coverage than CloudFormation?

Some things in the AWS API aren't (yet) available in CFN.

antoncohen2• 5 years ago

The Terraform provider for Google Cloud uses partial autogeneration, here is the repo that does the autogeneration for multiple automation tools:

https://github.com/GoogleCloudPlatform/magic-modules

joeduffy3• 5 years ago

[caveat, Pulumi co-founder here]

You are right that it's not easy. Thankfully the cloud providers themselves have moved in the direction of auto-generation for their own SDKs, documentation, etc., which has forced this to get better over time. This is motivated by much the same reason we've done it -- keeping one's own sanity, ensuring high quality and timeliness of updates, and ensuring good coverage and consistency, when supporting so many cloud resource types across many language SDKs.

Microsoft, for instance, created OpenAPI specifications very early on, and standardized on them (AFAIK, they are required for any new service). Those API specifications contain annotations to describe what properties are immutable (as you say, the need to "re-create" the resource rather than update it in place). Google Cloud's API specifications similarly contain such information but it's based on the presence of PATCH support for resources and their properties. Etc.

The good news is that we've partnered with the cloud providers themselves to build these and we expect this to be increasingly the direction things go.

pm903• 5 years ago

This is an excellent comment. I think the answer is just that the cloud provider API's are getting more "stable" and user friendly with time, making the additional abstractions of e.g. terraform not worth the overhead of learning that entire world. The response from the Pulumi co-founder seems to indicate as such, that they're relying on the simplicity of the API's to not get into a bad state (although I'm still skeptical if that's possible).

octopoc2• 5 years ago

The Pulumi Automation API lets you have an API to update infrastructure from your own process instead of via CLI commands that call your executable. I hope this makes it easy to implement multi-tenancy by spinning up separate infrastructure for different customers.

This would make it easier for software for dynamic schema built on top of a relational database. Dynamic schema is one area where most libraries don't help you out much, but it's an increasingly important feature, especially for businesses.

giovannibonetti• 5 years ago

Some relational databases like Postgres and MySQL support JSON columns. This is useful if, for example, you want to create an ecommerce application with multiple product variations stored in a single table, but with different fields - that is called Single Table Inheritance (STI). You can have some regular columns for the common fields, and a JSON column for the specific ones.

If your app is built with Rails, you can use this library to help you on that (I'm not affiliated with it): https://github.com/jrochkind/attr_json

vdfs• 5 years ago

Is search in large tables slow in Postgres JSON columns? can it be index?

mssundaram• 5 years ago

I am having trouble understand what Pulumi does, since I rarely work in web architecture. Could someone explain it?

adflux• 5 years ago

Define your cloud infrastructure in code (python, typescript, other languages) and deploy it with ease. I think it beats other formats.

tomca322• 5 years ago

It's a library around Terraform that lets you write infra stuff in Python, JavaScript, Go, and maybe some other langs.

For me the biggest benefit is the fact that I can write my infra with Types in TypeScript.

spooneybarger4• 5 years ago

It's no longer built around terraform. It uses the cloud provider APIs directly now.

See native provider.

qbasic_forever2• 5 years ago

It's infrastructure-as-code to the logical conclusion that you write real programming language code (Python, Javascript, Go, etc.) to build your infrastructure. Something like Terraform lets you declare infrastructure with HCL/JSON but doesn't give you a lot of flexibility to reason about or dynamically change things. If you want to spin up a frontend server for every backend for example, you have to write out the configs and tweak them to match the desired state. Pulumi and similar tools go the next step and let you write code to build your infrastructure--if you want your frontends and backends to be in sync, write some Python to make it happen instead of spelunking in a pile of configs.

It's super useful and compelling at scale, but there are some potential downsides. For one, it's a real program and IMHO you should treat it as such with the same rigor of documentation, testing, etc. as the production code you're deploying. This might actually make your processes a bit slower or more risky. Because if you don't do that then you just risk building unmaintainable, untrustworthy deployment code that nobody wants to use.

The second downside is you're taking a big bet on Pulumi to stick around. At this point, it's probably a safe bet. But the more you use their SDK and system the more tightly coupled your production deployment is to the continued existence of Pulumi. If they go bust and can't maintain the SDK you might have a production system that can't be deployed anymore.

jacobmischka• 5 years ago

Thank you for this concise summary. While not a dedicated cloud services administrator, I keep up with new technologies and follow tech news fairly religiously, and this press release made my head hurt with all of the cloud-specific buzzwords and proper nouns.

Sounds like a neat and useful project for complex setups.

aetherspawn• 5 years ago

Thanks, I wanted to know the main difference between Terraform and this (ie why would I pay $50/mo), but this sums it up nicely.

Still, Terraform has my needs at this stage and looks a lot more mature. But if I was a small web host or something this could be a really interesting integration.

satya712• 5 years ago

It lets you provision and configure cloud infrastracture using code. Think about all the things you could do via cloud provider consoles, but now via code. So you can automate deployment, testing, migration. It's all well documented in one place, shareable, repeatable.

andrewSC• 5 years ago

"Self-hosting is only available with the Pulumi Enterprise Edition. Contact us if you would like to evaluate the Self-Hosted Enterprise Edition." [0]

[0] https://www.pulumi.com/docs/guides/self-hosted/

Jgrubb• 5 years ago

Just curious, why'd you post that bit?

all2• 5 years ago

A big appeal of "open source" is control.

Not allowing self-hosting for non-paid tiers is cutting control off from the user and moving it to the producer.

nine_k• 5 years ago

Because it says: «the next major version of the Pulumi open source project». Apparently, either not very complete, or not very open-source, if self-hosting requires a license.

spooneybarger4• 5 years ago

You don't need to use the service, although it solves a lot of problems. You can use only the Pulumi SDK and then solve the various state management issues that their service provides in another fashion.

I think its a reasonable business model. The SDK is completely open and free. The service is how they make money .

delnaught2• 5 years ago

In my quick read, I didn't see a way not to use the service.

"If this is your first time running pulumi new or most other pulumi commands, you will be prompted to log in to the Pulumi service. The Pulumi CLI works in tandem with the Pulumi service in order to deliver a reliable experience."[0]

[0] https://www.pulumi.com/docs/get-started/kubernetes/create-pr...

daralthus2• 5 years ago

Just to elaborate:

State is a snapshot of what is deployed at a given time, it is used at the next run to compare if there were any drifts since, eg.: "Did anyone delete a server manually?" etc.

By default it is stored with Pulumi, but if you are on a budget you can just use S3 with a few tradeoffs (concurrent deployments would need a lock that you implement yourself):

https://www.pulumi.com/docs/intro/concepts/state/

antoncohen2• 5 years ago

Open Source SDK
The SDK is a CLI and collection of libraries for defining and deploying cloud apps and infrastructure in code.

https://www.pulumi.com/pricing/

With Terraform the open source CLI and libraries for defining and deploying infrastructure are fully usable, even for large systems.

What I would like to know is, how usable Pulumi is without a paid subscription?

I thought when I first looked at Pulumi, the only non-local state backend was the paid Pulumi Service. But looking at it now, they seem to support the normal object store backends that Terraform does (https://www.pulumi.com/docs/intro/concepts/state/). Though the talk at lack of concurrency control seems to imply they don't support locking like Terraform does.

joeduffy3• 5 years ago

[caveat, Pulumi co-founder here]

Pulumi is fully functional in open source form. The analogy I like to draw is Git and GitHub. You can use Git fully independent of GitHub, or you can choose to use them together, for a seamless experience within a team. (Not a perfect analogy since we built both Pulumi open source and the Pulumi SaaS, which causes this very confusion!) We don't hold anything back, if it's in the SDK, it's open.

We recently added concurrency control to the alternative backends. I'm sorry the docs are confusing on this matter -- we will get that fixed up. We also have many large customers in production on the open source alone. It's easier with the SaaS just because we handle security, reliability, and sharing with your team along with access controls, auditing, etc. But if you prefer to roll your own there we are entirely happy to have you in the community and help out. Admittedly our marketing materials aren't super clear here and we are working to fix this.

Hope this helps to clear things up and again apologies for the confusion.

fock2• 5 years ago

just because I don't get it from their marketing: their API is a cloud-service wrapping other cloudservices. Or is it a classical library with wrappers?

spooneybarger4• 5 years ago

The Pulumi cloud offering provides state management.

With it, you can bring up some infrastructure on machine A and tear it down on another machine.

Without the service, you can only tear down on the same machine you brought it up. Ditto for adding to existing infrastructure.

You don't need their service but you have to solve problems like knowing what is running, replicating the needed state etc.

The service is very nice and makes it easy to know what is running, how long it has been running, who created it etc- across clouds. But, it is not strictly needed. As an individual user, you could get away without using. For a team, it fills a very vital gap.

bogomipz3• 5 years ago

Is the difference the same as using Terraform Cloud vs just keeping your state in your own S3 bucket then?

pm903• 5 years ago

This seems like a huge con. Terraform at least allows you to store state yourself if you want to on multiple backends. With Pulumi you have to pay to store state, and you have to store it with Pulumi Inc. ? Why?

aChrisSmith2• 5 years ago

It’s a bit more involved than that…

Suppose you wrote a program to go and create your cloud infrastructure. Using AWS’ APIs you write an app to spin up a VM, setup a load balancer, and provision a database. You run your program, it works like a charm. Now you have all the infrastructure setup just the way you want it!

The problem is what happens when you want to _update_ that infrastructure? The program you wrote, using the aws.CreateVM(..) and aws.CreateLoadBalancer(..) API calls is no longer applicable. Instead, you probably want to use a different set of APIs, like aws.UpdateVM(…), to update your existing cloud resources. For example, change some port settings on your load balancer. So your app needs to be smart enough to check if the resources already exist, and if so, update them. Otherwise, create them fresh.

And it gets even worse. What if you want to create some new resources, such as attach an SSL certificate to your load balancer… but still keep all of your existing infrastructure as-is. Or what if you want to update an IAM usage policy that is already in-use by several other resources… Somehow your app needs to know the impact of that change, and how it will ripple out across other cloud resources.

Does that start to make sense? You don’t really want a “wrapper” for cloud APIs. You really want something that allows you to effectively describe your cloud infrastructure, and “make it happen”. And leave the specifics of “how” as an implementation detail… accomplished by a cloud provider’s APIs.

That is what Pulumi does — and other Infrastructure as Code tools, like Terraform. It provides you a way to describe your cloud infrastructure in a programming language, so that every time you run your app it will make the cloud reflect that target state. It will:

- Create resources if they don’t exist. - Update existing resources if they do. - Delete any resources that you no longer need.

I work at Pulumi and am happy to go into details about the joys of not dealing with cloud APIs directly, and just using Pulumi :)

pgroves• 5 years ago

Thanks for saying this ^^, it's the best explanation of what this thing actually is (sorry but I didn't really get it from the homepage). Define a state of infrastructure using a real sdk (not yaml files) and it can figure out and apply the migrations from the current state to the new state. (right?)

bogomipz3• 5 years ago

Hi. I was curious if you could talk about how "native providers" differs from how Terraform does things. Also how come no AWS yet but there is native provider but GCP and Azure. I would have thought that would have been a "must have" for release. Is it just the number of resources AWS has that made it more time consuming or something else? Thanks.

fock2• 5 years ago

That still wraps access to this cloud-services, albeit adding some more functionality. My main question was: is it a cloud service itself or can you use it standalone (as a classic API)?

wpietri2• 5 years ago

Do folks have opinions on how Pulumi compares to Terraform CDK?

I'm the only ops-y person in my group. I've done things so far with Terraform HCL, but a) it's not a great language, and b) I can't really ask other folks to learn it. But they all know Python, so my theory is that if I can wrap our ops stuff up in familiar-looking code, they'll be able to work with it effectively.

Is one of these toolkits better than the other? I'm inclined to go with Terraform CDK just because they company's further along, but I'd love other people's takes.

cmclaughlin4• 5 years ago

Do folks have opinions on how Pulumi compares to Terraform CDK?

I last tried Pulumi around the 2.0 release and it flat-out didn't work for me. After working through a few bugs with their tech sales I gave up on it. I'm guessing their TypeScript support is better than their Python - for me it just wasn't ready for production.

On the other hand I've been using Terraform CDK in production since the early alpha releases. Had to work around a couple of bugs here and there, but no blockers at all. It's been a game changer for my team and we're really enjoying it!

wpietri2• 5 years ago

Thanks! Very helpful.

nsonha• 5 years ago

just looking from far away I don't see any point of Pulumi while Terraform now has a CDK, for now Pulumi has play to catch up with Terraform on providers

chrischen• 5 years ago

Any thoughts on Pulumi vs Terraform?

whydoineedthis• 5 years ago

imho, if you think typecasting the AWS api in your local code repository is useful, than Pulumi is the way to go. Maybe one day someone will find an api that AWS didn't properly typecast before releasing and then the hundreds of hours of typecasting in your local repo will be worth it.

Other than that, it's really annoying way to keep infrastructure. I prefer the Terraform DSL myself, but I'm certain there is an audience that disagrees with me.

cies• 5 years ago

I dont understand your 2 opening statements:

if you think typecasting the AWS api in your local code repository is useful, then Pulumi is the way to go.

What do you mean by typecasting here? Isn't providing the types what Pulumi (or TF CDK for that sake) does by the libs it provides? I dot see how I, the consumer of those libs, needs to "type cast" anything.

someone will find an api that AWS didn't properly typecast before releasing and then the hundreds of hours of typecasting in your local repo will be worth it.

cmclaughlin4• 5 years ago

Any thoughts on Pulumi vs Terraform?

If you're just getting started with Terraform be sure to check out CDK for Terraform. It allows you to drive Terraform from a variety of high-level programming languages instead of it's DSL.

https://github.com/hashicorp/terraform-cdk/

StavrosK2• 5 years ago

Getting Started with Python. Prerequisites: Terraform >= v0.12, Node.js >= v12.16

What? :(

lucidone• 5 years ago

It's like Gradle versus Maven in Javaland.

1_player• 5 years ago

I found Terraform buggy, and really disliked its language. Upgrade paths are a pain. The fact that Pulumi runs on classic, Turing-complete languages is a huge plus to me.

Unless your infrastructure is so large and managed by so many departments you need the least powerful declarative language to stay sane, my suggestion is to go with Pulumi.

It's not perfect, the whole infrastructure-as-code ecosystem is frankly pretty bad, but if Pulumi is a 6/10, Terraform is a 4/10. YMMV

ncmncm• 5 years ago

Imagine, a release announcement containing a sentence that bothers to say what the thing being announced is.

They will surely lose points for reduced brevity.

dith3r• 5 years ago

Using Pulumi for about 1 year to manage gcp and AWS environments (5+) - each deployment is around 500 resources. At the begining it was buggy but when they started to add typing i see a lot of improvment.

daralthus2• 5 years ago

<3 Pulumi. It's really nice defining what you want to exist at a cloud provider or as a k8s service, and it will make sure that it is there and nothing else. All in typescript.

catern2• 5 years ago

I hope Pulumi eventually develops a "bare-metal" mode. I'm very much in favor of using a real programming language to set up systems[0], but I don't like to use cloud services beyond basic hosting. It would be nicer if Pulumi provided, as a library, code to launch and maintain self-hosted databases, proxies, etc, rather than just providing clients for cloud APIs.

[0] http://catern.com/progsys.html

octopoc2• 5 years ago

IIUC it already does this. Many of the libraries Pulumi exposes are not specific to any cloud platform. Examples:

- Docker: https://www.pulumi.com/docs/reference/pkg/docker/

- Postgresql: https://www.pulumi.com/docs/intro/cloud-providers/postgresql...

- Keycloak: https://www.pulumi.com/docs/reference/pkg/keycloak/

catern2• 5 years ago

I believe those are libraries for using these services (e.g., connecting to them and allocating resources inside them), not launching them and setting them up.

adevx• 5 years ago

A 100 times this. I would drop Ansible in heartbeat for Pulumi and have a strictly typed IaC base. Hope this use case gets more traction.

throwaway8238825• 5 years ago

Dear Pulumi: Do you have an HCL to Pulumi converter thingy? The first big barrier to using Pulumi in my org is that they basically banned anything other than Terraform "because standardization". I would love to get rid of TF, but we need a migration path.

The next big stumbling block is your license...

jaxxstorm• 5 years ago

Yes we do!

https://www.pulumi.com/tf2pulumi/ https://github.com/pulumi/tf2pulumi

kodeninja• 5 years ago

Pulumi looks super interesting!

Is there a tutorial out there that demonstrates using _just_ the Pulumi SDK, perhaps in a CI/CD setting, to deploy a simple service to Kubernetes?

Many thanks!

haolez• 5 years ago

I've been really interested in the code inlining feature[0], even though it's just available for JavaScript/TypeScript.

What's holding me back right now is that supported for Azure Functions is still in its infancy for code inlining. It's a shame, since I'm stuck in Azure for now.

[0] https://www.pulumi.com/docs/intro/concepts/function-serializ...

TruthWillHurt• 5 years ago

I'm sorry, I really liked Pulumi, but it became too complex after a short while.

Since all infra resources are defined as code you end up defining them in a procedural manner (Python), and struggle to arrange/resolve dependencies.

happyrock2• 5 years ago

I agree, I enjoyed using it and found it very powerful, but it was tricky to organize the code in a way that kept things simple, and there weren't many great examples to draw from in the documentation or elsewhere. I'd love to see what a large multi-provider Pulumi deployment looks like when it's done the right way.

ynouri• 5 years ago

Glad to see all the innovation happening in the IaC space.

I like the multi-language premise of Pulumi but I have the feeling that over time one of the languages will gain a significant advantage over the others (better supported, more features, native...) and thus will become the primary language for Pulumi package authors and users (similar to HCL for Terraform and TypeScript for CDK).

Meaning that eventually all the "serious" Pulumi users will end up converging on one single language (similar to this comment of a CDK user migrating from Python to TypeScript). Feedback loops will only help amplify this effect (e.g. more Pulumi users on language A -> more examples online -> more package authors working in this language -> better established best practices -> more submitted issue tickets for language A -> language better supported by Pulumi -> more users...)

Curious about how Pulumi multi-language components [0] work under the hood. Isn't writing a "Pulumi schema" describing resources the same as writing declarative HCL at the end of the day?

[0] https://www.pulumi.com/blog/pulumiup-pulumi-packages-multi-l...

joeduffy3• 5 years ago

Excellent topic. [caveat, Pulumi co-founder here]

Indeed what you say is true of many other "multi-language" platforms. I was an early engineer on .NET at Microsoft, and although it was multi-language from the outset (COBAL.NET was a thing!), the reality is most folks write in C# these days. And yet, you still see a lot of excitement for PowerShell, Visual Basic, and F#, each of which has a rich community, but uses that same common core. A similar phenomenon has happened in the JVM ecosystem with Java dominating most usage until the late 2000s, at which point my impression is that Groovy, Scala, and now Kotlin won significant mindshare.

I have reasons to be optimistic the infrastructure language domain will play out similarly. Especially as we are fundamentally serving multiple audiences -- we see that infrastructure teams often elect Python, while developers often go with TypeScript or Go, because they are more familiar with it. For those scenarios, the new multi-language package support is essential, since many companies have both sorts of engineers working together.

A "default language for IaC" may emerge with time, but I suspect that is more likely to be Python than, say, HCL. (And even then, I'm not so sure it will happen.) One of the things I'm ridiculously excited about, by the way, is bringing IaC to new audiences -- many folks learn Python at school, not so much for other infrastructure languages. Again, I'm biased. But, even if a default emerges, I guarantee there will be reasons for the others to exist. I for one am a big functional language fan and especially for simple serverless apps, I love seeing that written in F#. And we've had a ton of interest in PowerShell support since many folks working with infrastructure for the Microsoft stack know it. And Ruby due to the Chef and Puppet journeys.

I also won't discount the idea of us introducing a cloud infrastructure-specific language ;-). But it would be more general purpose than not. I worked a lot on parallel computing in the mid-2000s and that temptation was always there, but I'm glad we resisted it and instead just added tasks/promises and await to existing languages.

As to the Pulumi schema, you're right, that's a step we aim to remove soon. For TypeScript, we'll generate it off the d.ts files; for Go we'll use struct tags; and so on. Now that the basic runtime is in place, we are now going to focus there. This issue tracks it: https://github.com/pulumi/pulumi/issues/6804. Our goal is to make this as ridiculously easy as just writing a type/class in your language of choice.

mtalantikite• 5 years ago

There might be "a default language" but I suspect that is more likely to be Python than, say, HCL. One of the things I'm ridiculously excited about is bringing IaC to new audiences -- many folks learn Python at school, not so much for other infrastructure languages.

As someone that has worked largely on backend systems and infra for a long time, are colleges actually training these infra skills? Some app engineers at a place I consult for was just having this conversation last week, interested in how to train infra skills and pretty much everyone had sort of fallen into infra roles over their careers, with no formal training in it. Most of us were just *nix hackers as kids, learned to program at some point, and now we’re here. Working on infra is quite a lot more than just knowing the language.

I’m not mad at having something other than HCL — years ago when I worked at Engine Yard we developed a cross platform cloud library that let us write things in Ruby which was nice. But when thinking of solving infra problems I’ve never once thought “you know, if I could just write this in Python these problems would go away”. Actually I personally hate Python as a language, I’d much prefer to write Go or Rust or TypeScript, and it does feel like a bonus that everyone touching infra sort of just has to use HCL which removes a lot of bike shedding.

Totally open to improvements! More isn’t always better though.

JimDabell• 5 years ago

When I first started using Pulumi, I was using Python and it seemed like the Pulumi team focused on JavaScript and the rest of the languages were afterthoughts. But they’ve improved a lot on this front, so if anything the trend is in the opposite direction.

neopointer• 5 years ago

I couldn't agree more. In my company we've decided that everyone will use Typescript with Pulumi because we believe that the developer experience (IDE, code autocompletion, etc) will be the best + it's a language that arguably is more used here as frontenders will know it too + providing examples, etc.

mbStavola• 5 years ago

I'm in love with Pulumi. Simply one of the best tools I have in my stack.

After about a year of use, I simply cannot go back to editing YAML files or clicking things in a web UI. It's ruined me completely.

Kunigaikstis3• 5 years ago

I've been using Pulumi for the past few months as a person who's never touched infrastructure-as-code before. I love it. It has its downsides in the documentation. Mostly around limitations in resource name lengths.

Apart from that, it's an amazing tool to work with. The company I'm working at right now uses it extensively. All of our microservices have Pulumi in the CI/CD pipeline. It's an extremely powerful tool and I'd even use it for personal projects over provisioning resources through a vendor like AWS.

kjuulh• 5 years ago

Yeah, it has been a while since I used it, but I remember having to fall back on Terraform docs more than once. But it really is nice being able to ditch Terraform DSL with your language of choice.

And when I used it they also had quite permissive pricing for personal use, where you could store your tfstate or equivalent on their servers which is nice as well for solo devs, who doesn't want to build or shell our for a storage platform or CI for terraform.

The only thing I had some problems with was errors, which wasn't the easiest to decode. But it wasn't too bad.

Kunigaikstis3• 5 years ago

Agree with the errors but I believe they've gotten better/you get used to their true meaning after a while. It's a complex piece of technology so I don't bash them for it too much

lloydatkinson2• 5 years ago

How do you add it to CICD? Does it pull down and create infra every new push to the master branch? What does the workflow look like?

muxator2• 5 years ago

Last time I checked (admittedly, not so hard) I discovered that Pulumi only works by calling in to the mothership: i.e.: you cannot build an airgapped infrastructure with it.

This was very off-putting for me, and I had to give up on Pulumi. Is something changed on this regard?

hzia• 5 years ago

You totally can. They support storing pulumi state in any storage platform (like S3 or self hosted minio service)

The cloud offering is just there to streamline the storage + fancy dashboard.

Identical to how terraform works.

muxator2• 5 years ago

Thank you.

So this is a case of me not looking well enough while exploring many different solutions.

I have to say that reading this passage on one of the getting started guides (https://www.pulumi.com/docs/get-started/azure/create-project...):

If this is your first time running pulumi new or most other pulumi commands, you will be prompted to log in to the Pulumi service. The Pulumi CLI works in tandem with the Pulumi service in order to deliver a reliable experience. It is free for individual use, with features available for teams.

did not help. At least mentioning that the Pulumi service could be self hosted would have helped.

Thanks again.

lloydatkinson2• 5 years ago

I’m just getting into infrastructure stuff like this, how does Pulumi work with CI/CD? Does it check what infrastructure it needs to create every deploy? What assurances are there that it won’t delete the whole production database every push?

Kunigaikstis3• 5 years ago

We use it exclusively in CI/CD to provision/delete resources. It keeps track of everything created via Pulumi (or imported from outside of it) in a config on the cloud. Haven't experienced random resource deletion apart from genuine user error.

rswail• 5 years ago

A couple of years ago I started down the Pulumi path, but what was unclear at the time is that there are two phases of a pulumi run.

The first is on constructing the DAG of resources, the second is using that DAG to orchestrate the changes from the previous state.

The problem I had is that values that are only available in the second phase (eg a subnet ID) can't be accessed in the first phase directly. The subnet's ID is accessed via "my_subnet.id" but it's actually effectively a future.

You can't write something like "if my_subnet.id == "1234" (this is an arbitrary example), but effectively a reference to a resource's attributes is only available on the "right hand side" of an expression, and pretty much as a simple assignment only.

To use the value of an attribute, you essentially need to access the value in the second phase of the pulumi process, and to do that (at the time) involved essentially pushing through source code to be "eval"ed on the "other side".

I'll be interested to see what's changed :)

bionhoward• 5 years ago

Twice I've tried to love Pulumi but wind up going back to Terraform because the Terraform provider docs put clear usage examples front and center

supernovae• 5 years ago

I do believe the constraints of TF and similar DSLs are what makes them great.

Pulumi 3.0

Related Stories