How Dark Deploys Code in 50ms

https://medium.com/darklang/how-dark-deploys-code-in-50ms-771c6dd60671

by craigkerstiens • 7 years ago

269 167 7 years ago

medium.com

pbiggar33• 7 years ago

Author here - happy to answer any questions!

conorgriffin2• 7 years ago

What is Dark’s biggest weakness (excluding the current lack of adoption)?

pbiggar33• 7 years ago

Wow, what a start :) Probably the biggest one is that you have to leave your editor. If you've spent 20 years carefully configuring your vim or emacs setup, you're not going to be happy to give that up. Don't have a good solution to this right now either.

djm_• 7 years ago

Other languages have large ecosystems of packages that allow users to do common tasks quickly. I can imagine not having something similar would be a sticking point to adoption; how does Dark picture the Dark ecosystem working and competing with other already well-entrenched ecosystems? Thanks!

pbiggar33• 7 years ago

There's a couple of ways of handling this. First is that of course we need to create a package manager and ecosystem, and that will take time (we can accelerate it by making a great experience, but haven't gotten there yet).

Dark can talk to other HTTP things, so many of the functions that are done by libraries can be done over HTTP. So you can call other services, even if we don't have an SDK for that yet. And from there it should be easy to package it up so there is an SDK.

Finally, it's a question of what you're doing. There's many tasks where the disadvantage of not having an ecosystem is outweighed by Dark's other advantages. And as we grow the ecosystem, the disadvantages go away, unlike the status quo tooling.

manoji• 7 years ago

I am not able to signup for beta with an organization email address. is this intended?

pbiggar33• 7 years ago

No it's not! Sorry about that. Email me direct (see my profile) and I'll try to fix it.

the_duke• 7 years ago

Dark sounds like an interesting concept.

I seriously hope though that you will open source the compiler.

I can't see Dark going anywhere without a guarantee that you can use it without using the provided infrastructure and that your code will work fine even if Dark goes away or you decide to host it yourself.

I certainly won't touch Dark with a 10 foot pole otherwise. And would advise anyone against it strongly.

ToJans• 7 years ago

I see quite some similarities with low-code systems, 4GLs, smalltalk etc, so AFAIK the market is there... Please take into account that your target audience probably is not on HN. If I would be you, I would target web agencies, that can use the dark platform as a competitive advantage, if your system allows you to build stuff faster... IMO it does not make sense to learn all these new paradigms for a small project, you really need scale for this kind of investment from your customers... Right now it feels like you are building a solution that does not know which specific problem it is solving...

alexashka2• 7 years ago

Hi Paul,

How did the decision to create a whole new language take place? Was using existing languages a serious consideration and if so, what were the trade-offs that made you go in this direction?

I'm familiar with the basics of back-ends. Let's say you had a language that compiles each back-end point into a binary executable (each api point is a separate program). All you'd need is versioning via directories, symlinks to the active most up to date version and a router that looks up the executables in a dir and routes to them.

Making changes would involve changing the code, pressing "deploy" which pushes to github and gets the backend to pull, re-compile the binaries and update symlinks.

Given this setup - what does your approach do that's more than what I've just described? I'm assuming there's some fundamental problem my naive approach does not handle that made you go in the direction that you've just described.

pbiggar33• 7 years ago

We did of course consider it. I don't think it's possible to use existing languages to do this.

Presumably, we'd have to use Javascript as our basis cause everyone knows Javascript. But then we'd have people wanting "Dark for python" and "Dark for Ruby" and "Dark for rust", etc, which making a new language neatly sidesteps.

So if we built it for javascript, we'd need to decide how to support all of JS's features. How do we support eval? How do we support Typescript and Flow and ReasonML?

Then we'd be stuck with all the JS mistakes: promises and futures and callbacks and object model and null/undefined/0/"" comparison tables.

This list keeps going: it would have to use a parser. It would need to support npm, etc, etc.

I expect once people see Dark that they'll try to build similar features for existing languages, and I hope they succeed! There's some companies that are doing similar things, and I think the scale of their vision is much lower, partially because they are limited by language choice.

leonaves• 7 years ago

Is it possible to write code in an offline environment (e.g. on a plane)?

pbiggar33• 7 years ago

Yes, we'll have support for that, including handling sync conflicts when you get back online. It's not gonna be a great experience since Dark is really really designed for working on production, but it should work ok.

sddhhanover2• 7 years ago

Can I take Dark out for a spin?

pbiggar33• 7 years ago

Right now, we're in private alpha. If you have a product or service that you want to build, fill out https://darklang.com/subscribe and our CEO Ellen will get back to you to discuss and schedule an onboarding. We're not doing tire kicking yet though :)

sddhhanover2• 7 years ago

Entered my email awhile ago... just updated the mailchimp profile (daniel@orthly.com) to include my use-case. Let me know! Would love to provide feedback - what you're working on sounds awesome

devcriollo• 7 years ago

They show very little, I'm anxious to be able to see.

pbiggar33• 7 years ago

We're talking about the design of Dark in these posts. We plan to show what Dark looks like in our launch in September.

davejstl• 7 years ago

What are the server/OS/cloud prereqs for Dark?

pbiggar33• 7 years ago

There are none! We run the infrastructure in the cloud (we run it on GCP using Postgres and k8s, but will likely go multicloud at some point).

Most of the advantages we discuss in this post come from the fact that you're using the Dark editor, language and infra. I don't believe this would be possible with a self-hosted solution.

Deimorz2• 7 years ago

So anyone using Dark is completely locked-in, to the point that they can't even take their code into an editor other than yours?

It's a proprietary language, that can only be edited in a proprietary cloud-based editor, that only runs on proprietary infrastructure?

heavenlyblue2• 7 years ago

> Most of the advantages we discuss in this post come from the fact that you're using the Dark editor, language and infra. I don't believe this would be possible with a self-hosted solution.

What makes it impossible specifically?

voidr2• 7 years ago

When you write new code in a function or HTTP/event handler, we send a diff of the Abstract Syntax Tree (the representation of your code that our editor and servers use under the hood) to our servers, and then we run that code when requests come in.

You can achieve the same result with PHP and FTP, however once you start adding unit tests and branching, that's when things start to take time.

Long builds aren't necessarily a problem if they give you enough in return.

pbiggar33• 7 years ago

You're exactly right that this style of development would be completely unsafe using PHP+FTP, which is why we had to include all the other stuff: the feature flags, function versioning, DB migrations.

voidr2• 7 years ago

What I was pointing out that hitting save and getting a result on the live server was achievable in the 90s, there is nothing new about it.

The reason we stopped doing that is because we wanted version control, we wanted to work in teams and we wanted unit and integration tests, which I doubt that you could run within 50ms.

the feature flags, function versioning

I can easily replicate that with at least early 2000s PHP(would probably work with 90s PHP as well).

DB migrations

We do have DB migrations in all web frameworks that are worth using.

jrockway2• 7 years ago

I am not sure people are getting these results with PHP and FTP. Sure, the build time is zero seconds. But each page load takes 100ms because it's doing the same work over and over for every request. Yes, even if you cache opcodes. We have a legacy PHP app and an "echo hello world" request takes 100ms. An extremely complicated request then takes 102ms to process. It is shockingly slow, but it is not something you will notice unless you have a lot of instrumentation, so you might be lead to believe that your performance woes are caused by something else.

vrodic• 7 years ago

You must be doing something wrong. There's a lot of overhead for frameworks like Symfony, yes. But it's possible to write reasonably productive and modern website code where you can get sub 20ms responses (doing auth, fetching from db, processing and rendering html). I know because I've done that, on PHP 5.4.

nlawalker• 7 years ago

Sounds cool, seems like an interesting set of tradeoffs that nobody else has tried yet.

From what I've read so far it seems like Dark is a good fit for prototyping, experiments, and small products and teams that "move fast and break things". Would you consider it a good fit for large teams and applications, where different people own different components and reliability is the most important metric? A lot of the accidental complexity in the "standard issue" continuous deployment process detailed in the post is there to prevent accidents and enforce permissions.

pbiggar33• 7 years ago

Yeah, all this stuff is actually designed for large teams (though, it'll be a while before Dark is production-ready enough for them). CD matters because it allows you deploy in small chunks which is much less risk than deploying in big chunks.

The feature flag is a thing that you might not need if you're small, but it completely changes how large teams work. Same with DB migrations: if you have a small amount of data you can just do a schema migration and the locking wont affect you too badly.

We'll have a permissions model at some point that will take care of the ownership problem. We designed Dark to allow extremely fine grained ownership, for example allow a contractor access to a single HTTP route or function.

Overall, this is kinda similar to how we went about building CircleCI. We focused on good tooling for companies (eg parallelization and speed) instead of individual projects (eg build matrices).

lmilcin• 7 years ago

The way to go is to have high-trust culture and comprehensive automated testing system.

You want developers to behave responsibly and to put all development effort into writing good, reliable code in the first place. I personally prefer pair programming over code review. Code reviews have a lot of downsides but two most important are that you can't get from commit to prod automatically (as there is another inevitable manual step) and that reviewer typically doesn't have enough background/time to perform the review in meaningful way.

ricardobeat• 7 years ago

EDIT: Ha. My apologies as I just realized my stupidity - I was browsing on mobile and it turns out out I jumped into the 'What is Dark' post [1] and never finished the original! The mailing list links are in that one - I subscribed. This is a lot more interesting, cheers :)

[1] https://medium.com/darklang/the-design-of-dark-59f5d38e52d2

pbiggar33• 7 years ago

The section that describes how it does it in 50ms is called "#deployless". That describes the mechanics of the deployment. How we made deployless possible is addressed for the rest of the post after that.

(The mailing list thing is weird, I only see that link at the bottom. Can you point me at the ones you see in the text?)

adrienjt4• 7 years ago

How coupled is Dark with the Dark language? I understand that the purely functional nature of the language makes its integration with the rest of the platform easier, but the blog post doesn't go into much detail about that. What would it take to integrate Dark with: a) another (existing) purely functional language, and b) a general-purpose language, e.g., Go? Could a code intelligence tool like Sourcegraph help?

pbiggar33• 7 years ago

Extremely tightly coupled. That's kinda the point "what can we do if we don't have to couple dark to anything else" (eg editors, other languages). I can't imagine how you'd couple Dark to another language, but I also don't see the point?

adrienjt4• 7 years ago

Thank you. My point was risk mitigation, which I discussed in another reply (which you answered: https://news.ycombinator.com/item?id=20399267 )

iLemming• 7 years ago

I've been seeing a lot of talks about darkland and can't find a single example of code. Can someone show me at least how does it look like?

gspetr• 7 years ago

from darklang.com:

"Dark is currently in private alpha."

That's probably the primary reason why there's little to no info.

amelius• 7 years ago

They are leaving us in the dark, so to speak.

z3t43• 7 years ago

Syntax errors are the easiest errors to fix and almost never get into production.

Zero downtime deployments are very rare, and ads a lot of complexity, so it's a big trade-off, compared to just uploading the new code and restarting the service. And having the front-end deal with minor hiccups like service restarts, because users will have such issues all the time due to being on mobile networks, trains going through tunnels etc.

You can do as much testing as you want, both manually and automatically, but you still can't detect all issues as efficient as thousands of users in production. So just accept that there will be issues, and instead design your pipeline so that those issues can be fixed within minutes.

quickthrower23• 7 years ago

You can do as much testing as you want

With what Dark is promising, I believe testing would be easier. Deploy to a test environment in 50ms. Then run a suite of tests, then deploy to production in 50ms.

There could be some complementary tech that figures out what tests need to be run based on what you change to get the time down.

heavenlyblue2• 7 years ago

They're kind of lying.

You only need pip or any package install if your testing image doesn't pre-install the required packages before running the tests. Which I don't myself since I always want to see if my packages work with the latest versions of other packages - and Dark wouldn't be able to handle that anyway.

So my test runs actually complete in about 50 ms if I'm not testing any external API functionality. They will also never finish earlier than 5 minutes if I am, even on Dark - since they're based on external services.

snypox• 7 years ago

Zero downtime deployments are very rare

What do you mean? Rolling deployments are a very basic feature of container orchestrators.

nijave• 7 years ago

You don't even need containers--it's a feature of load balancers containers rely on. Start new container, wait for it to become healthy, add it to the load balancer, remove old container (with an optional, let connections to old container close first)

z3t43• 7 years ago

If you aim for zero downtime there will be a point where the complexity work against you. for example: load balancer goes down, corrupted state due to two different versions running simultaneous, a bottleneck causing congestion, etc.

My favorite "load balancing" for web-pages is using "DNS round robin" where you add several A records, and the browser automatically try another IP if the request fails.

namelosw• 7 years ago

I don't know what he refers to. But there is a specific kind of zero downtime deployments which can restart without losing in-memory state, like Erlang.

Erlang also provides "fix in minutes, in production" stories he refers. For most of languages, it's very hard to remotely get access to an object, let alone fix things directly.

With Erlang, you can inspect every object in the cluster to understand what's going on for any user, and fix the plane in the mid-air instead of rebuilding a plane with the same issue, then go through the process to fix it.

koliber• 7 years ago

It's an interesting concept that flips many things on its head. We're getting some real hard-to-achieve benefits out of the box, while sacrificing a lot of things we used to take for granted.

One thing that I notice is that Dark requires a completely walled in environment to be able to deliver all of its promises. In reality, any real-life project has to interact with external services, legacy systems, or custom peripherals which are unlikely to be based on Dark.

How will Dark be able to handle such cases? Current tooling and processes are able to cope with such things through custom, hand-crafted, and as the article stated, often fragile, systems and processes. The are a pain, but they get the job done. They are very versatile by their nature. While they break more often that we would like to, they can be patched up well enough to band-aid over rough patches and adjust to surprising externalities. Dark seems to have one way of doing things, and I am curious how compatible that way is when it comes to interfacing with external systems.

btown• 7 years ago

It seems that Dark is designed to allow you to wrap external APIs in easy-to-use interfaces, so it should be at home as part of a larger SOA or ecosystem, of course with the caveat that it's still being developed. See the footnote at https://medium.com/darklang/dark-residency-7dbc878a5f0a .

pbiggar33• 7 years ago

Yes, that's exactly right. Dark speaks HTTP (and at some point, will speak GRPC, Thrift, etc) so you can access 3rd party services over HTTP (and wrap them in SDKs in the Dark package manager, same as on npm).

We expect we'll have some built-in support for common things (resizing images and videos, or generating pdfs for instance; we already support static assets on a CDN).

stevefan1999• 7 years ago

hence the pun "going Dark", you're complete shut-in into a vendor-specific environment

quickthrower23• 7 years ago

I think it is fantastic that this kind of innovation is happening. I get sick of waiting for stuff at work, some kind of build or CI thing. npm i, docker build or whatever else. I hope the language experience is good, and it's a reasonable language (static typing!).

It sounds like taking the Go idea of quick builds to the next level.

davedx2• 7 years ago

How does it compare with hot swapping in other languages and platforms like Erlang, JVM/.NET hotswap technologies? I'm guessing the built-in safety functionality from the feature flagging is one thing?

pbiggar33• 7 years ago

It's much simpler. Those have complex in-process hot swapping. Dark's "hot swap" is a write to a DB, then new HTTP requests use the new code.

As we grow and start to look at compilation and optimization, I think we'll probably get much fancier, but for now we're very into keeping things super simple.

davedx2• 7 years ago

I'm pretty interested in this. Signed up to the mail list. Good luck!

bufferoverflow• 7 years ago

Code deployment is almost always trivial and fast. It's the databases that are slow and very tricky, especially in distributed systems.

unfunco• 7 years ago

It sounds like a convoluted version of Geocities.

GordonS• 7 years ago

As an aside, that big list of tasks and diagram at the end are perfect to show next time a client exclaims about how long it takes to get a simple change into production!

kccqzy• 7 years ago

Twenty years ago, deployment of a new version of a web app means FTP'ing a new php file to the server.

eloisant• 7 years ago

Well, you're free to deploy like this (but you probably know why it's a very bad idea).

somehnguy• 7 years ago

Feel free to explain why

pikzel• 7 years ago

Which wasn't a bad thing after all. I've had more incidents with deployment pipelines than with the actual code.

dharmab• 7 years ago

Mostly because people forget that pipelines are code and need documentation, tests and software engineering practices just like any code.

blumomo• 7 years ago

That's quite a dark name for such a bright idea.

oftenwrong2• 7 years ago

Dark sounds like it would satisfy my preferences well.

- statically typed, functional language with no null pointers

- safe deploys and rollbacks

- structured editing / version control

- minimal infrastructure management

...etc. A tool that makes writing a correct program, and deploying it without catastrophe or fuss, easy. I look forward to reading more.

bullen• 7 years ago

I added the same feature to my Java app server: http://github.com/tinspin/rupy it's a simpler design which compiles, zips and hotdeploys over HTTP to a classloader.

But it's fast enough as long as you keep each project small.

juandazapata• 7 years ago

Given how tightly coupled is Dark with its IDE and deployment environment, I'm interested in knowing how much VC money have they taken? What if they get acquired or simply shut down?

It looks like there won't be a migration path, and you'll need to rewrite your app from scratch using a different language. Again, it would be nice some transparency around how much VC money they've taken.

andy_ppp• 7 years ago

For there to be 9 of them and 4 open heads I'm guessing it's probably $10-20 million at this stage.

I think the only way I'd use this is if I had the option to host it myself i.e. the code was all open sauce.

pbiggar33• 7 years ago

The sustainability of Dark is super important to use. We're not fucking around with customers' livelihoods.

For our early customers, we have a handshake agreement that we won't leave them hung out to dry. If Dark was to shut down, we would 100% make sure they had the time and ability to port their app off. There would be a time cost to them of course, so it's not perfect.

As we get closer to opening up to the public, we'll be doing a bunch of work on the sustainability question so we have good answers to the "acquired or shutdown" problem. We imagine a legal framework to protect customers, but haven't got the details worked out.

londons_explore2• 7 years ago

A legal framework doesn't help much in the "bankruptcy" case. Unless that legal framework means all IP becomes open source and people can self-host I guess.

sciurus• 7 years ago

How does Dark handle code review?

Even if I was willing to accept the other tradeoffs that come with Dark, I'd want another human to approve my changes before that 50ms deploy to production kicks off.

jmull• 7 years ago

I don't think they've detailed that yet, at least not that I've seen.

But it is one of the few steps the Dark deployment process retains (see the diagram near the end of the article), so it seems they will support it (hopefully reasonably well).

londons_explore2• 7 years ago

Seems easy enough... Write whatever code you like, directly in production, but feature flags will be forced to zero (or to just your own test users) until code review passes...

It requires the language to have a bulletproof feature flag system which can't be bypassed, which it seems is the goal.

pbiggar33• 7 years ago

The model is quite different to how people write code today. Instead of a process that takes code from your machine and sends it to prod (and thus has a lot of risk), you code in a sorta sandbox _in prod_. That is then enabled for users via changing the feature flag setting.

So to do code review, you write the code behind the feature flag, then ask someone for code review. (There's google-docs style collaboration, so they can see your code if you permit it). After code review, you can "merge" by toggling the feature flag, or changing the setting to "let 10% of users see this" or whatever roll-out strategy you want.

just safely. So you're writing code behind a feature flag (but "in prod"), but no users can see it.

dkordik• 7 years ago

cool- I guess there is some concept of a snapshot/diff to show you the before and after?

theamk• 7 years ago

So what protects you from accidentally doing “DELETE TABLE users” on prod?

In traditional design, there will be pre-commit code reviews and staging env with limited access. But looks like Dark get rids of all of this?

keymone2• 7 years ago

The problem is granularity of deployment - the smallest piece you can rollout is, these days, a container with code and dependencies, that has to be build and threaded through your CI/CD stack before it hits prod.

I like to imagine what would need to happen to reduce(increase?) that granularity to single function - you change the code, byte-compile it, send that blob to your backend’s hot code swap port and voila - you’re done. A lot has to happen behind the scenes though - a change in dependencies would still require a larger deployment or much smarter code swapping procedure, caller/callee tree would need to be considered, functions need to be pragmatically pure (talking to dB is fine, being part of an “object” - not really), functions need to be extensively specced, etc.

But that would be my backend holy grail.

icebraining3• 7 years ago

What are you looking for, that isn't provided by platforms with hot code-swap (e.g. Erlang) or FaaS systems (e.g. AWS Lambda, OpenFaaS) ?

keymone2• 7 years ago

you're thinking of only surface functions/endpoints, those that are invoked directly from your api route dispatcher. i'm talking about any function in your source code of a large backend with many endpoints.

think of function `+` - i want to be able to deploy a change to that function such that no worker process gets respawned.

also imagine every time you call `+` in your application - it goes throught the whole FaaS shenanigans with serializing, sending rpc over wire, receiving response, deserializing, dealing with throttling, errors, backoffs. all of that is ridiculous for `(+ 1 2)`.

and yes, erlang is a good example of an environment where this could work.

toolslive• 7 years ago

Back in the day (early 00s), you could do something similar with ZOPE: develop on a production server, in a private session and you could mock about with your version of the code, and when you were satisfied push it through for everybody else. This was because the database that was holding the users, permissions, ... was also holding the code. It was lightyears ahead of J2EE at the time. (Now I wonder what happened to it)

blunte• 7 years ago

I know that Plone, which is built on Zope, found its place in a lot of universities and governments. So I imagine Zope lives on with necessary support from its big users.

slinkp2• 7 years ago

Yup. Plone is still pretty active https://github.com/plone

slinkp2• 7 years ago

Former professional Zope guy here! (I made a living building things with it in the early-mid 2000s).

What happened to Zope? Well, Chris McDonough (creator of Pyramid, and Zope veteran) blogged about this in 2011 and from my perspective he got it exactly right. I still think this history is fascinating and I wish more people knew it.

https://web.archive.org/web/20161019153348/http://plope.com/...

After the history lesson, some of his "lessons learned" bullet points seem very apropos in the context of Dark, particularly the first two:

* "Most developers are very, very risk-averse. They like taking small steps, or no steps at all. You have to allow them to consume familiar technologies and allow them to disuse things that get in their way."

* "The allure of a completely integrated, monolithic system that effectively prevents the use of alternate development techniques and technologies eventually wears off. And when it does, it wears off with a vengeance."

pbiggar33• 7 years ago

Thanks for pointing to that, it's super interesting.

One of the things we designed Dark around was being able to safely and easily continuously deliver language and framework changes. The Zope 3 project was a disaster by the sound of it, and I would hope this will allow us avoid a similar fate. That's the goal.

Also, we're pretty pro-testing (though also type systems, which help too).

KirinDave8• 7 years ago

So... you're 50ms away from a total global outage and/or major feature fault?

Why would I want that? How do I control it for green-blue? For rational isolation? Is there native support for canary releases?

coldtea3• 7 years ago

Yeah, because speed of deployment prevents CI/tests/checks /s

(Seriously, how do people come with such facile counter-arguments?)

KirinDave8• 7 years ago

How does CI actually prevent these kinds of outages? Did CI save Google from the networking outage or tests prevent a global Cloudflare outage? Do you think those organizations don't spend enough energy for the average SV commute on test runs per release? Are you seriously going to pretend that those efforts "solve" reliability and stability in any non-trivial system?

It's actually not hard at all to write fast software deployment. Many people do it accidentally. But to tie it directly to editors and stovepipe the infrastructure around testing, that seems to me like a great business model that any rational engineer interested in reliability should immediately reject unless it's coupled with open source, statistically valid canary tooling at the bare minimum.

What you actually want at that degree of universal knife-switching is rollbacks.

toast0• 7 years ago

So... you're 50ms away from a total global outage and/or major feature fault?

And 50ms away from restoring service.

The degree of care that needs to be put into a release is related to the potential damage a bad release can cause and the time to repair.

Reducing time to deploy reduces time to repair for most errors (certainly not for errors than write bad states, if you write bad data to your database, that's most likely a long repair regardless of how fast to deploy it was).

If you're building NES cartridges, you should really get it right the first time, since there's no way to update. If you're making a low cost internet service, for small changes, it's not unreasonable to try things without a lot of testing before hand as long as you commit to detecting and rolling back quickly.

KirinDave8• 7 years ago

And 50ms away from restoring service.

Sure, but fast rollbacks are much easier than fast rollouts.

Reducing time to deploy reduces time to repair for most errors (certainly not for errors than write bad states, if you write bad data to your database, that's most likely a long repair regardless of how fast to deploy it was).

A uniform fast rollout is almost always worse, and a separate problem from data recovery or fast rollbacks. But I agree with this paragraph in principle.

If you're making a low cost internet service, for small changes, it's not unreasonable to try things without a lot of testing before hand as long as you commit to detecting and rolling back quickly.

When you're doing more than basic presentation of semi-dynamic content, most CSPs make multi-region deployment very affordable and easy. It's generally a failure of early engineering departments or an accumulation of technical debt that makes it otherwise in 2019. Too many folks assume it is something they should do later rather than get right to start.

Canaries and regional redundancy should be part of every product's basic launch checklist, imo. People love to say any kind of software engineering best practice is bad, because they love to pretend software engineering has no value. But for every flash UX trick or tracking feature someone ships providing value, that's completely overshadowed by being down when a user needs you. Reliability not only saves you time at the developer level by making features more predictable, but it means you need to worry slightly less about funnel optimization because your user attrition is lower.

kreetx• 7 years ago

Isn't this better than deploying for 15 minutes, having an outage then, and then rolling back?

KirinDave8• 7 years ago

Not if that 15 minute deploy rolled out over a canary that gives me much less of a chance to actually have outages, yeah.

Knife-switch rollbacks are good though, tbf. I just don't think that's particularly challenging.

pbiggar33• 7 years ago

Author here. Was this in response to the article or the title? Because the article goes into great detail about this.

To answer, the whole system is based on feature flags (which are equivalent in this case to green-blue/canary releases). So yes, there is a massive amount of control for this; in fact, that's the whole point of the article.

KirinDave8• 7 years ago

Feature flags are not canaries. I checked the article twice, even did a find on the text of the view source because sometimes I miss stuff.

You just didn't talk about the actual complexity of deployment. And that's your call, but as you can see from this thread there are people who think CI/CD to prod with no controls is a good call.

cbetti• 7 years ago

From what I gather, Dark's deployment "unit" is the entire AST covered by a function, and these deployment units are versioned and promotable via an independent criteria.

Perhaps one of the promotion strategies could involve a human controlled criteria.

I would like to understand their plans on this point a bit better as well.

I see immense power in switching a promotion strategy away from large units, like entire containers or versioned artifacts, to fine grained components, like versioned functions.

Do you see any advantages here for improving your platform's availability?

umvi• 7 years ago

Very interesting, but I don't like the idea of abandoning VSCode to learn some single-purpose IDE

vinceguidry• 7 years ago

I'm getting really tired of learning new languages / stacks every few months. I want one language to write things in. It has to be a pure-OO language and right now that's ruby. I want to edit that code with one code editor. Syntax awareness is a nice-to-have but not essential. At the moment my editor of choice is Sublime Text 3 but to extend it I need to write python which is unideal. When I start writing websites it'll be hyperstack.

Unless your new language can fix all my issues, be open source, and be my One True Human Interface to All Things Machine, I'm going to keep building on ruby, with occasional bash / crystal / rust excursions for speed.

Is your language pure-OO? Do you have gradual typing? Does it run on a common VM? Am I going to have to rewrite everything I'd ordinarily bring in a library for? I can accept every single one of your claims but still not be interested in learning a whole new stack because I can already deliver stuff quickly enough with the one I've been iterating on for a decade. Especially when it could end up orphaned in 3 years if it doesn't get enough adoption.

I'm not saying your holy grail can't be holy enough. But it has to be pretty darn holy.

taosx• 7 years ago

Marketing piece with grand plans...not really

I would argue with you that this is not such a priority for a software development organization, there are more important things to do.

You got so excited...so fast, I think this community may be doing more bad for me than good lately.

vincentmarle• 7 years ago

This sounds extremely exciting and it looks like a lot of thought has been put into this. I’m looking forward to try it out.

A lot of comments here are coming from an anxiety that is caused by many years of dealing with fragile deployment processes, but I like the boldness of Dark to just bypass all that and make the fundamentals safe and solid.

burlesona• 7 years ago

First I’ve heard of Dark. The blog post is cool, but reading it I felt two main things: (1) no pull reviews would be terrible for a team, and (2) everything being instantly live in prod and automatically versioned etc sounds like a nightmare once you move beyond a simple program.

Reading on Darklang.com I see they kind of address this. The ecosystem they’re building is training wheels for developers, a way to make “coding 100x easier.”

I can see that working in a sense, and being an entry point that a lot of people use to make neat toys. I could even see it being a gateway that people use to get interested in and learn about programming. But I can’t imagine wanting to build a business with more than one programmer, or any kind of scale on a completely black box system like that.

I will also say, there’s a problem when it comes to entry-level systems, trying to teach people to code:

The struggle with the complexity is actually important. Dark isn’t actually making all the work of building and running a web application go away, it’s abstracting it all into a platform such that you can’t see it.

Suppose a person gets into coding self-taught and learns to work this way. The knowledge isn’t going to be very transferrable - ie when they look at other languages or systems they’re likely to struggle with problems like “what does prod versus dev mean, I just want my program to run for the world...”

You usually move knowledge between ecosystems by translating “I did it this way in (toolset A), so what’s the parallel in (toolset b)?” The more you have an idea of the underlying principles the easier this is to figure out.

That said, Dark certainly looks neat, and I imagine the implementation is quite cool.

The only other nit I’d pick is the name. To me, its not really a language, but maybe a development environment, or framework. I suppose it had a language in it, which I’d probably call DarkScript or something.

pbiggar33• 7 years ago

Author and Dark CTO here.

Reading on Darklang.com I see they kind of address this. The ecosystem they’re building is training wheels for developers, a way to make “coding 100x easier.”

No no no no no. This is not what we're doing. We're building something for experienced software engineers. (New developers will be able to use it too and it should be much much easier for them than learning in other ways, but that's a secondary audience).

Dark isn’t actually making all the work of building and running a web application go away, it’s abstracting it all into a platform such that you can’t see it.

Some of it is being abstracted, but a lot of it being properly removed. Servers are abstracted, but almost all of actual deployment is actually being removed.

module0000• 7 years ago

Hrm...we don't deploy code in to prod in 50ms, but we certainly deploy to dev/staging/testing in <1s, and promote to prod once we are 100% sure we want to do so. We use the boring, industry-standard tooling and process everyone else uses (Jenkins, k8s, Ansible, nothing bleeding edge). I'm unsure what advantage their product has that makes it stand out among the battle tested tools already in use. Would someone elaborate?

disclaimer: working for a l2 network provider, and a mistake costs us infinitely more than an "Oh we're sorry!" can fix.

ukd1• 7 years ago

The point is all that stuff (jenkins, k8s, ansible) is accidental complexity that, I presume someone at your company has-to/had-to think about. I think Dark's angle is it's stuff you shouldn't need to think about in order to solve the actual problem, and removing as much of that as possible is better.

alexandercrohde• 7 years ago

Okay, I mean conceptually cool, but it's not like an existing startup gonna throw away all their code and rewrite it to save on dev-ops time.

I guess the value prop is that if you're starting from scratch, and want to write your whole app in Dark then you can avoid hiring a dev-ops engineer and iterate faster?

Seems like a stiff trade-off. If it's an interpreted language you're coming from, setting up envs can be < 30 secs anyways (and same with prod deploys), so that's faster than you need.

If it's a compiled language you're coming from (e.g. scala/c++) then it's likely Dark may not match your needs.

Plus I guess this assumes every team in your company is using only dark? Like what if it's a SOA and a full environment has a node component?

akhilcacharya2• 7 years ago

Is Dark intended to be a Lisp? I'm surprised by the fact that the team didn't include language examples (from what I can gather)

pbiggar33• 7 years ago

No, it's an ML. (You're right that we haven't published samples, we think people get distracted by syntax and we want to focus on the goals at the moment)

akhilcacharya2• 7 years ago

Oh cool! That makes me quite a bit more interested.

jonstaab• 7 years ago

Very curious about the data migration piece. It sounds like it's designed to handle mapping a row of type a to a row of type a prime. What about situations like a specific combo of type a and c to b and two different records of type c, and optionally delete a record of type d? We occasionally have migrations that are only representable as a function of several entire tables, and which may even leave gaps, e.g. for domain events that affected the system but weren't logged until a later revision.

dcchambers• 7 years ago

Very interesting and flips mostly everything about CI and CD on it's head. I am disappointed the blog post doesn't include any screenshots or actual examples of the Dark code/editor working. It's kind of hard for me to wrap my head around how this works at a technical level just reading about it...would like to see some concrete examples. I went over to the website to look for a demo/examples but there isn't anything.

jdtangney• 7 years ago

One question: Security

How Dark Deploys Code in 50ms

Related Stories