lukehoban

I do agree there is a balance here, and that the ideal point in the spectrum is likely in between the two product experiences that are currently being offered here. There are a lot of benefits to using PRs for the review and iteration - familiar diff UX, great comment/review feedback mechanisms, ability to run CI, visibility and auth tracked natively within GitHub, etc. But Draft PRs are also a little too visible by default in GitHub today, and there are times when you want a shareable PR link that isn't showing up by default on the Pull Requests list in GitHub for your repo. (I frankly want this even for human-authored Draft PRs, but its even more compelling for agent authored PRs).

We are looking into paths where we can support this more personal/private kind of PR, which would provide the foundation within GitHub to support the best of both worlds here.

(Disclaimer: I work on coding agents at GitHub)

This data is great, and it is exciting to see the rapid growth of autonomous coding agents across GitHub.

One thing to keep in mind regarding merge rates is that each of these products creates the PR at a different phase of the work. So just tracking PR create to PR merge tells a different story for each product.

In some cases, the work to iterate on the AI generated code (and potentially abandon it if not sufficiently good) is done in private, and only pushed to a GitHub PR once the user decides they are ready to share/merge. This is the case for Codex for example. The merge rates for product experiences like this will look good in the stats presented here, even if many AI generated code changes are being abandoned privately.

For other product experiences, the Draft PR is generated immediately when a task is assigned, and users can iterate on this “in the open” with the coding agent. This creates more transparency into both the success and failure cases (including logs of the agent sessions for both). This is the case for GitHub Copilot coding agent for example. We believe this “learning in the open” is valuable for individuals, teams, and the industry. But it does lead to the merge rates reported here appearing worse - even if logically they are the same as “task assignment to merged PR” success rates for other tools.

We’re looking forward to continuing to evolve the notion of Draft PR to be even more natural for these use cases. And to enabling all of these coding agents to benefit from open collaboration on GitHub.

Yes. This is a really key part of why Copilot coding agent feels very different to use than Copilot agent mode in VS Code.

In coding agent, we encourage the agent to be very thorough in its work, and to take time to think deeply about the problem. It builds and tests code regularly to ensure it understands the impact of changes as it makes them, and stops and thinks regularly before taking action.

These choices would feel too “slow” in a synchronous IDE based experience, but feel natural in a “assign to a peer collaborator” UX. We lean into this to provide as rich of a problem solving agentic experience as possible.

(I’m working on Copilot coding agent)

As peer commenters have noted, coding agent can be really good at improving test coverage when needed.

But also as a slightly deeper observation - agentic coding tools really do benefit significantly from good test coverage. Tests are a way to “box in” the agent and allow it to check its work regularly. While they aren’t necessary for these tools to work, they can enable coding agents to accomplish a lot more on your behalf.

(I work on Copilot coding agent)

I played around with the idea of a language motivated by this same thought process last year: https://github.com/lukehoban/ten.

* Succinct syntax and operators tailored to AI model definition

* Fully statically typed tensors, including generic functions over tensor dimension and batch dimensions (...)

* First-class hyper-parameters, model parameters and model arguments for explicit model specification

* Einops-style reshaping and reductions - tensor dimensions are explicit not implicit

Ten Language

https://github.com/lukehoban/ten

Ten is a statically typed tensor programming language for defining AI models.

Ten has the following features: (1) Succint syntax and operators tailored to AI model definition (2) Fully statically typed tensors, including generic functions over tensor dimension and batch dimensions (...) (3) First-class hyper-parameters, model parameters and model arguments for explicit model specification (4) EinOps-style reshaping and reductions - tensor dimensions are explicit not implicit

Example (a functional GPT2 implementation):

    Gelu(x: {...}) -> {...}:
        return 0.5 * x * (1 + Tanh(0.7978845608 * x + 0.044715 * x**3))    

    SoftMax[N](x: {...,N}) -> {...,N}:
        exp_x = Exp(x - Max(x))
        return exp_x / Sum(exp_x)    

    LayerNorm[S,E]|g:{E},b:{E}|(x:{S,E}) -> {S,E}:
        mean = Mean(x)
        variance = Var(x)
        return g * (x - mean) / Sqrt(variance + 1e-5) + b    

    Linear[N,K]|w:{N,K},b:{K}|(x:{...,N}) -> {...K}:
        return x@w + b    

    FFN[S,E]|c_fc, c_proj|(x:{S,E}) -> {S,E}:
        a = Gelu(Linear[E,E*4]|c_fc|(x))
        return Linear[E*4,E]|c_proj|(a)    

    Attention[Q,K,N,V](q:{...,Q,K}, k:{...,N,K}, v:{...,N,V}, mask:{Q,N}) -> {...,Q,V}:
        return Softmax[N](q @ Transpose[N,K](k) / Sqrt(K) + mask) @ v    

    MHA[H,S,E,K]|c_attn, c_proj|(x:{S,E}) -> {S,E}:
        q, k, v = Linear[E,E*3]|c_attn|(x) {S,(3,H,K) -> 3,H,S,K}
        causal_mask = (Tri[S]() - 1) * 1e10
        out = Attention[S,K,S,K](q, k, v, causal_mask) {H,S,K -> S,(H,K)}   
        return Linear[E,E]|c_proj|(out)    

    Transformer[H,S,E]|mlp, attn, ln_1, ln_2|(x:{S,E}) -> {S, E}:
        y = x + MHA[H,S,E,E/H]|attn|(LayerNorm[S,E]|ln_1|(x))
        return y + FFN[S,E]|mlp|(LayerNorm[S,E]|ln_2|(y))    

    GPT2[H,S,E,B,V]|wte, wpe, blocks|(inputs:{S}) -> {S,V}:
        x = wte.[inputs] + wpe.[Range[S]()]
        z = for i in 0...B: x, y -> Transformer[H,S,E]|blocks.[i]|(y)
        return LayerNorm[S,E]|ln_f|(z) @ Transpose[V,E](wte)

Great suggestion! We haven't yet integrated ESC into Pulumi AI, but it's something we'll be looking into. Great opportunity to really make it easy to get started with ESC.

Agreed that YAML (and JSON) can be difficult to manage at large scale. This is actually a big part of why Pulumi ESC exists, to be able to decompose large YAML/JSON configuration files into smaller logical and composable units.

As you note, intellisense and error squiggles can also help a lot here - both for ensuring references to other environments are correct, and to get checking for your dynamic secrets providers. We’ve added the Monaco editor (from VS Code) into the Pulumi Cloud console to make it easier to offer these features in the very near future.

We also offer a preview pane to make it easy to play interactively validate your environments documents while working on them directly in the console.

Lots more coming for providing an even richer experience working with environments in Pulumi ESC.

Closure compiler was actually one of the biggest influences on the design of TypeScript, and even the early motivation for the approach that TypeScript took.

From https://medium.com/hackernoon/the-first-typescript-demo-905e...:

There were many options already available, but none seemed to be resonating well with a broad enough section of the market. Internally at Microsoft, Script# was being used by some large teams. It let them use C# directly instead of JavaScript, but as a result, suffered from the kind of impedance mismatch you get when trying to stand at arms length from the runtime model you are really programming against. And there was Google’s Closure Compiler, which offered a rich type system embedded in comments inside JavaScript code to guide some advanced minification processes (and along the way, caught and reported type-related errors). And finally, this was the timeframe of a rapid ascendancy of CoffeeScript within the JavaScript ecosystem — becoming the first heavily used transpiled-to-JavaScript language and paving the way for transpilers in the JavaScript development workflow. (Aside — I often explained TypeScript in the early days using an analogy “CoffeeScript : TypeScript :: Ruby : C#/Java/C++”, often adding — “and there are 50x more C#/Java/C++ developers than Ruby developers :-)”)

What we quickly discovered we wanted to offer was a “best of all worlds” at the intersection of these three — a language as close as possible to JavaScript semantics (like CoffeeScript) and syntax (like Closure Compiler) but able to offer typechecking and rich tooling (like Script#).

The self hosted backend does have plenty of wringles though like not supporting the same stack name across different projects with the same backend URL contrary to how the hosted backend works.

Just a note that we did add support for projects to the self-hosted backend a few months ago, which does bring the self-hosted and managed backends into alignment here.

https://www.pulumi.com/blog/project-scoped-stacks-in-self-ma...

It’s funny - when we were first designing TypeScript - I often described it as "TypeScript is to CoffeeScript as C#/C++/Java is to Ruby" often adding "and there are 50x more of the former developers than the latter" [0]. And CoffeeScript’s approach of transpiling down to clean JavaScript was a big inspiration for TypeScript. In the 10 years since then, some of the Ruby/CoffeeScript aesthetic has become more mainstream in other programming languages (Swift, Rust), and gradual type systems have become more of an expectation even in dynamic languages like Ruby (Sorbet), Python (mypy) and PHP (Hack). So it does seem very natural to bring these back together now like Civet is doing.

[0] https://medium.com/hackernoon/the-first-typescript-demo-905e...

Fun! Reminds me of a little tangent I went down when I realized the untyped lambda calculus could be embedded within the single Go type `type V func(v V) V`, including a Y Combinator, church numerals and factorial function with no values other than instances of type V.

https://medium.com/@lukeh/untyped-lambda-calculus-church-num...

https://gist.github.com/lukehoban/0ec2a3dbbb9a13338d338a3fbb...

There have been some good questions in a few of the threads here about why Pulumi added YAML support in the first place. I thought I’d share a few thoughts on that.

Certainly it wasn’t only about being able to compile all sorts of esoteric languages down into a YAML-based data representation - though it’s fun to see the results of that here :-).

Our goal at Pulumi has really been to offer the best tools for infrastructure as code, and for defining and managing cloud infrastructure more generally, for any developer working with the cloud.

We started with a focus on the high-end - teams managing significant complexity of cloud infrastructure to get the most they can out of the managed services their cloud providers are making available as building blocks. We’ve grown with this part of the market, with great adoption and usage across many of the most advanced cloud engineering teams.

In order to grow with these users, we’ve invested in many layers of the Infrastructure as Code stack. Some of those improvements have been related to the software engineering benefits of using traditional general purpose programming languages to manage cloud infrastructure - IDEs, types, abstraction and reuse, test frameworks, packaging and versioning and so much more. But we’ve also been making improvements at many other layers of the stack. Our cloud deployment orchestration engine is now, I believe, the richest option in the market - with multi-cloud support, built-in secrets management, refactoring support with aliases, rich controls over replacement behaviour, built-in support for components, and much more. And our native providers for Azure, Kubernetes, Google Cloud and AWS are the most complete and most up-to-date providers for managing those cloud platforms.

We wanted to be able to offer all these benefits to the broadest possible range of developers working in the cloud. Since we already have very rich programming language options, we wanted to add an option at the other end of the spectrum - the simplest possible interface we could to the Pulumi platform. And that is what Pulumi YAML is. It is very simple, and designed for small scale use cases (a few to a dozen resources). It composes with the rest of the Pulumi ecosystem, so it’s easy to push complexity into components built in other Pulumi languages, to reference outputs of stacks deployed by other Pulumi languages, or even to "eject" into another Pulumi language if the complexity gets too high in YAML. So unlike many other IaC ecosystems where YAML or a DSL is the only option, in Pulumi, YAML can be a nice solution for simple use cases, without having to be abused for the complex use cases that are already well served by Pulumi’s existing alternative programming language choices.

There’s more details on all of these points at https://www.pulumi.com/blog/pulumi-yaml/ for those interested in learning more!

CloudFormation is really three separate things rolled up into one: (1) a resource model for AWS, (2) a deployment orchestration engine and (3) a syntax for specifying desired state as CloudFormation templates.

Tools that compile to CloudFormation templates offer a way to access (2) directly - to use alternative front-end syntaxes but to still deploy via the CloudFormation orchestration engine.

Tools like Pulumi (and Terraform) have their own deployment orchestration engines, which we believe offer many benefits - performance, secrets, components, transformations, aliases/refactoring, multi-cloud provisioning, and a lot more.

Cloud Control API lets us (and others like us) access (1) directly, without having to use (2) or (3), and thus being able to still offer the full set of benefits of provisioning via Pulumi, along with the full set of benefits of a well-defined AWS resource model.

Indeed! And we just released a new AWS Native provider which builds on top of this new AWS Cloud Control API: https://www.pulumi.com/blog/announcing-aws-native/.

I'm really excited about what Cloud Control API enables more broadly as well. I see this as a new high bar for cloud provider resource provisioning APIs, and it's an approach I hope to see more of across the industry going forward.

Previously, to provision and manage a resource, you would need to find the REST APIs for that specific resource, and define your own model for what set of REST API calls constitute a logical Create, Read, Update or Delete operation. This work was done and re-done in dozens or hundreds of cloud provisioning tools.

But AWS themselves had already built a model for doing this, and service teams at AWS were already defining their own well-defined resource model - the CloudFormation resource model for their services/features - it was just locked up behind CloudFormation templates as the only way to author and deploy resources using this resource model.

With Cloud Control API, all of the value of that resource model, and the work by teams at AWS to cleanly define that resource model, is now unlocked to be used by any sort of provisioning tool or service - not just CloudFormation templates.

At Pulumi, that has enabled us to build our new AWS Native provider on top of Cloud Control API, but it opens up tons of additional scenarios as well. Kudos to the AWS team for their work on this!

What I want: Use Terraform programmatically, i.e. call "cdktf deploy" or similar FROM node or python and give users some scripts they can use where I can abstract away some of the difficulties of learning to use Terraform natively for simple use cases (i.e. deploy an S3-based frontend host). Ideally, I had intended to distribute some npm-installable packages which would run this stuff.

This is actually exactly the use case we’ve designed the Pulumi Automation API (https://www.pulumi.com/blog/automation-api/) to support.

Allowing modern IaC technology (like Pulumi or Terraform) to be easily embedded into custom software solutions, instead of just being something humans work with directly, is a huge potential enabler for the next wave of cloud infrastructure management tooling.

Pulumi itself (https://www.pulumi.com/) is a Modern Infrastructure as Code tool much like Terraform - but focused on modern cloud workloads like containers, serverless and Kubernetes - and using programming languages like Python, TypeScript, Go and .NET instead of constrained DSLs to bring improved software engineering capabilities into cloud infrastructure projects.

The Pulumi Kubernetes Operator (https://github.com/pulumi/pulumi-kubernetes-operator) is a new project that provides a Kubernetes-native way to continuously deliver cloud infrastructure defined using Pulumi into AWS, Azure, GCP or a Kubernetes cluster.

There is certainly some additional complexity here over the basic Lambda serverless setup. But I think the console-driven configuration as outlined in these posts often makes it harder to see what the core concepts really are.

With infrastructure-as-code tools, this can be a little clearer. At Pulumi we wrote a post earlier today on configuring the infrastructure needed to use EFS with Lambda, and it boils down to just a few concepts and a couple dozen lines of infrastructure code.

https://www.pulumi.com/blog/aws-lambda-efs/

Some of the complexity here also comes from the fact that EFS is a general purpose managed NFS service, instead of a fully-abstracted Lambda-specific feature. That does add a little additional up-front complexity, but means you can use EFS across all sorts of different compute in AWS - not just Lambda.

We’ve worked with several Pulumi users who have migrated thousands of resources (including in production) from being managed by Terraform to being managed by Pulumi - so this is definitely possible.

We’re actively working on an overhaul of the tf2pulumi tool to support more Pulumi languages and more breath of Terraform project structures. But already it should support the majority of adoption use cases.

At-scale adoption can be done in Pulumi by combining transformations [1] and imports [2]. Together, these allow programmatically importing existing resources without the need for manual steps.

See the section on adopting in the docs here for details: https://github.com/pulumi/tf2pulumi/blob/master/README.md#us....

If you have more detailed questions on this - feel free to reach out on the Pulumi Community Slack (I’m @luke) for a deeper discussion: https://slack.pulumi.com/

[1] https://www.pulumi.com/docs/intro/concepts/programming-model... [2] https://www.pulumi.com/docs/intro/concepts/programming-model...

This post has some thoughts on ksonnet compared to Pulumi (and generally applies to comparison with the variety of other similar tools in that space): https://www.pulumi.com/blog/if-you-liked-ksonnet-youll-love-...

CTO at Pulumi here. Sorry you didn't get the support you needed! Please do drop me a DM in the Pulumi slack or email me at luke@.

[disclaimer: CTO at Pulumi so clearly biased :-)]

One of the things I really believe is that you can have the best of both worlds here. Pulumi uses imperative programming languages, but is still "declarative". The imperative programs are executed to build up the desired state, which can then be reliably diff'd and previewed, and can be used to enforce manual or automatic checks for correctness. So you get the expressiveness of imperative programs (loops, conditionals, components, packages, versioning, IDE tooling, testing, error checking, etc.), but still the safeguards and reliability of declarative infrastructure-as-code (preview, gated deployments, policy enforcement, etc.).

I also tend to view the perceived benefits of JSON/YAML/HCL "simplicity" as somewhat comparing apples to oranges on a complexity specturm. If you are only managing a dozen resources, it may be that JSON/YAML/HCL are fundamentally simpler. But when you've copy/pasted tens of thousands of lines of YAML around all over your codebase to manage hundreds or thousands of resources, the value of abstraction, reuse, well defined interfaces, and tooling to manage that complexity feels to me essential to the scale of the problem. And that degree of complexity is no longer just something large organizations are dealing with. Modern cloud technologies (serverless, containers, Kubernetes, etc.) are leading to significant increases in the number of cloud resources being managed, and the pace at which those resource are deployed and updated.

Assembly is a "simpler" way to think about programming, but didn't scale as complexity of application software increases. I believe the same is true about JSON/YAML/HCL and cloud infrastructure.

What I really want is a typed programming language, where using normal code we describe how to produce all these yaml-described k8s resources using whatever abstractions make sense in our business.

This is more or less exactly what we’re working on at Pulumi. Using a language like TypeScript to describe Kubernetes infrastructure provides a nice mix of simple JSON-like resource definition, strong typing, and the ability to use variables, loops, functions, classes, packages, test frameworks, and more from an expressive general purpose programming language. You still have access to the full fidelity of the raw Kubernetes API, but also all the tools you need to apply proper software engineering on top of it.

https://www.pulumi.com/kubernetes/

We've had customers and users of Pulumi at many different sizes and stages looking at or migrating to Kubernetes - and I'd say there are a few consistent reasons we hear for making the choice to adopt Kubernetes.

First, Kubernetes offers a rich, standards-based API for compute. For teams who want to deliver containerized applications, using Kubernetes allows targeting a standards API instead of a cloud-provider specific API. This up-levels the same benefit folks get with cloud IaaS solutions where they can write standard Linux applications. Now the standardized and "portable" part of their applications extends to some key parts of the compute infrastructure as well.

Second, we see a ton of Cloud + Kubernetes - teams who are still taking advantage of a lot of managed services and operational benefits from AWS/Azure/GCP/etc., but want to combine that with the container orchestration and rich API of Kubernetes. This means leaving stateful workloads in managed services like Aurora or Cloud SQL, delegating auth to managed IAM services, etc. Pulumi happens to be a really nice fit for doing these together.

And lastly, there's just a really active ecosystem that's building a lot of value around Kubernetes. Some of those pieces aren't yet fully mature, but directionally many teams are betting on this being where the next wave of mature cloud platform services will become available.

Here at Pulumi, this is something we are working hard on!

For all the many benefits we've seen with Infrastructure as Code to date, the tools are still fairly primitive - copy/paste is the norm for reuse, testing is rare or non-existant, productivity during infrastructure development is low, continuous integration and delivery are largely ad-hoc, and there are very few higher-level libraries available to abstract away the complex details of today's modern cloud platforms. Net - it feels like we're still programming cloud infrastructure at the assembly-language level.

I'm really excited about the opportunity to bring more software engineering rigor into the infrastructure as code space. At Pulumi we believe using existing programming languages is a key enabler of this. Pulumi is still a desired state model like other Infrastructure as Code offerings (so you can still preview changes and make minimal deltas to existing infrastructure) - but you can write code to construct that desired state. As a result, you get for loops and conditionals, you get types and error checking, you get IDE productivity tooling, you can create abstractions and interfaces around components, you can write tests, you can confidently refactor you infrastructure code, you can deliver and version components via robust package managers, and you can integrate naturally into CI/CD workflows.

Pulumi isn't the only tool in this space - we're seeing things like Atomist bringing this same model to delivery pipelines, and AWS CDK bringing this model to the CloudFormation space. I'm excited about where these tools will take the Infrastructure as Code ecosystem in the coming years.

[disclaimer - CTO at https://pulumi.io so clearly biased on this topic :-)]

In Go test files, there's now a codelens annotation above every Test function with "run test" and "debug test" commands. Just click "debug test" to launch a debugger against that test.

See https://twitter.com/ramyanexus/status/860190148239081472

If you want to configure F5 to launch tests instead of your main entry point, you can do that too in your launch.json file.

Yes - the vscode-go extension will show you function signatures (and type definitions and other symbol definitions) on hover.

If you hold down `cmd`/`ctrl` during hover you additionally get a preview of the definition source code (signature and implementation).

I'd love to get more details on this. Open an issue on GitHub?

VS 2015 also has a cool new feature that shows you references to each function right above the function declaration. Clicking on the reference count will open a quick jump contextual menu.

This feature is actually also available in VS Code[0], though only for C# currently. We should be able to add support for this in Go as well[1].

[0] https://code.visualstudio.com/Docs/editor/editingevolved#_re...

[1] https://github.com/Microsoft/vscode-go/pull/75

There is a Python plugin for VS Code available already [0]. I believe the PTVS folks are also working on a plugin for VS Code as well [1].

[0] https://marketplace.visualstudio.com/items?itemName=donjayam... [1] https://news.ycombinator.com/item?id=10589451