How Dark deploys code in 50ms - Darklang

By Paul Biggar

Speed of developer iteration is the single most important factor in how quickly a technology company can move.

Unfortunately, modern applications work against us: our systems are required to be live updated, silently and without downtime or maintenance windows. Deploying into these live systems is tough, requiring complex Continuous Delivery pipelines, even for small teams.

These pipelines are typically ad-hoc, brittle, and slow. They need engineering time to build and manage, and companies often hire entire Devops teams to maintain them.

The speed of these pipelines dictate the speed of developer iteration. The best teams manage deploy times around 5–10 minutes, but most take considerably longer, in many cases hours for a single deploy.

In Dark, deploys take 50ms. Fifty! Milliseconds! Dark is a holistic programming language, editor and infrastructure that’s designed from the ground up for Continuous Delivery, and every aspect of Dark, including the language itself, was built around safe, instant deploys.

Why are Continuous Delivery pipelines so long?

Make the changes

  • Create a new branch in git
  • Make the changes behind a feature flag
  • Run unit tests to validate your changes with the feature flag both on and off

Pull request

  • Commit the changes
  • Push the changes to a remote on github
  • Make a pull request
  • CI build runs automatically in the background
  • Code review
  • Repeat this step a few times perhaps
  • Merge the changes into git master

CI runs on master

  • Install frontend dependencies via npm
  • Build/optimize HTML+CSS+JS assets
  • Run frontend unit/functional tests
  • Install Python dependencies from PyPI
  • Run backend unit/functional tests
  • Run integration tests against both assets
  • Push frontend assets to a CDN
  • Build a container for the Python program
  • Push container to registry
  • Update kubernetes manifest

Replace old code with new code

  • Kubernetes spins up some instances of the new container
  • Kubernetes waits for those instances to become healthy
  • Kubernetes add those instances to the HTTP load balancer
  • Kubernetes waits for old instances to become unused
  • Kubernetes spins down old instances
  • Kubernetes repeats until all old instances have been replaced with new ones

Enable new feature flag

  • Enable the new code for just yourself, gain confidence
  • Enable the new code for 10% of your users, watch operational and business metrics
  • Enable the new code for 50% of your users, watch operational and business metrics
  • Enable the new code for 100% of your users, watch operational and business metrics
  • Finally, go through the entire process again to remove the old code and the feature flag

You’ll have a different setup depending on your tooling, language, and use of service-oriented architectures, but this feels representative. I’ve haven’t listed deploys with DB migrations since they typically require careful planning, but I discuss Dark’s solution to that below.

There are many moving parts here, and most of them either take a long time to run, can fail, introduce temporary race conditions, or may crash the running production system.

Since these pipelines are almost always ad-hoc, they are often brittle. Most of us will have had to deal with days when it’s not possible to deploy code because of some problem in a Dockerfile, one of the dozen services involved is having an outage, or the exact person who’s needed to understand a build failure is on vacation.

Worse, many of these steps provide no value. It used to be that deploying code enabled it for users, but we have separated those steps, and now code is enabled with feature flags. As a result, the code deploy step — replacing the old code with the new code — now just introduces risk.

Of course, this is an extremely advanced pipeline. The team that built this invested significant engineering time and money in this pipeline to get fast deploys. Most teams have significantly slower and more brittle deployment pipelines.

Dark’s Continuous Delivery design

#deployless

Jessie Frazelle coining “deployless” during the Future of Software Development conference in Reykjavík

The first and most important decision was to make Dark “deployless” (thanks to Jessie Frazelle for coining the term). Deployless means that anything you type is instantly deployed and immediately usable in production. That doesn’t mean we let you ship broken or incomplete code though (we discuss our designs for safety below).

When demoing Dark, we often get asked how we managed to make our deploys so fast, which always strikes me as an odd question. I suspect people think that we developed some incredibly advanced technology that can diff code, compile it, containerize it, spin up a VM, and cold-start the container, etc, etc, in 50ms, which is probably impossible. Rather, we built a customized deployment engine which doesn’t need to do any of those.

Dark runs interpreters in the cloud. When you write new code in a function or HTTP/event handler, we send a diff of the Abstract Syntax Tree (the representation of your code that our editor and servers use under the hood) to our servers, and then we run that code when requests come in. So deployment is just a small write to a database, which is instant and atomic. Our deploys are so fast because we trimmed a deploy to the smallest thing possible.

In the future, Dark aims to be an infrastructure compiler, creating and running the ideal infrastructure for the performance and reliability of your app. We will keep the instant nature of deployment when we move in that direction later.

Making deployment safe

Every state of incomplete code in Dark has a valid execution semantics, similar to typed holes in Hazel. For example, as you change a function call, we maintain the memory of the old function until the new one is valid.

Since every program in Dark has meaning, incomplete code doesn’t prevent complete code from working.

Editing modes

The second is code that is currently in use. Any code that has traffic flowing through it, whether functions, event handlers, databases, or types, needs to be safely editable. We handle this by “locking” all code that’s in use, and requiring you to use more structured tooling to edit it. The structural tooling is discussed below: feature flags for HTTP/event handlers, a powerful migration framework for DBs, and a novel versioning technique for functions and types.

Feature flags

Creating and deploying a feature flag is a single operation in our editor. It creates a blank space to add new code, and provides the knobs to control who sees the old and new code, as well as a button/command to atomically switch over to the new code or abandon it.

Feature flags are built into the Dark language, and even incomplete flags have meaning; a flag where the condition hasn’t been filled in will execute the old locked code.

Dev environment

Instead of trying to create a cloned local environment, feature flags in Dark create a new sandbox in production, which replaces the dev environment. In the future we also plan to sandbox other parts of your app — instant database clones for example — though we haven’t found these to be as important so far.

Branches and deploys

Feature flags are the most powerful of these (and arguably the simplest to understand and use), and enables us to entirely remove both other concepts. The deployment concept is particularly useful to remove: if feature flags are what we use to enable code anyway, then the “switch our servers to new code” step only introduces risk to your deployment.

While git is hard to use and beginner unfriendly — a true gatekeeping technology — branches are nice concept. Many of the downsides of git are reduced; Dark is live edited and uses Google-Docs-style real-time collaboration, removing the concept of pushes and reducing the need for rebases and merges.

Feature flags are the cornerstone of our safe deploys. By combining them with instant deploys, developers can quickly test concepts in small, low-risk chunks, never having a single large change that can take down a system.

Versioning

We also version types, for similar reasons. We discussed our type system more in our last post.

By versioning functions and types, we allow you to slowly make changes to your app. You can validate that each handler continues to work with the new version individually, and you don’t need to make wholesale changes to your app (we provide tooling to allow you make them very quickly, if you prefer).

This is much safer that the all-or-nothing deploys that happen today.

New versions of packages and standard library

Screenshot of part of Dark’s autocomplete, showing two versions of the Dict::get function. Dict::get_v0's return type was Any (which we are phasing out) while Dict::get_v1’s return type was Option.

We frequently provide new standard library function, and deprecate old versions. Users who are using specific old versions in their code continue to have access to them, while we hide old deprecated versions for users who do not. We expect to provide tooling to upgrade users from old versions to new versions in one step, again using feature flags.

Dark also has a unique ability: because we run your production code, we can test the new versions for you, comparing their output for both new requests and past requests, to tell you if anything changed. This turns package updates that are largely blind (or rely on extensive testing for safety) into something with significantly lower risk, which may be upgradable automatically.

New versions of Dark

In Dark, when we make small language changes, we create a new version of Dark. Old code stays on the old version of Dark, and new code starts using the new version. You can use feature flags or function versions to switch over to the new Dark version.

This is especially useful since Dark is so young. There are many changes to the language and library that may or may not be good ideas. Versioning the language in small increments allows us to make small updates, meaning we can delay many language decisions until we have more users, and hence more information, rather than solving them all now.

DB migrations

  • rewrite your code to support both the new format and the old one
  • convert all your data to the new format
  • remove the old way of accessing the data

This process turns DB migrations into long-running, expensive projects. The result is that many of us have outdated schemas, as fixing even simple things like table and column names isn’t worth the cost.

Dark has a powerful DB migration framework that we believe makes DB migrations so simple you will do them often. All datastores in Dark (which are key-value stores / persistant hashtables) have a type. To migrate a datastore, you just give it a new type, and a rollback and rollforward function to convert values back and forth between the two types.

Datastores in Dark are accessed via versioned variable names. For example, a Users datastore will initially be accessed as Users-v0. When a new version, with a different type, is created, it is accessible as Users-v1. Data saved via Users-v0 that is accessed via Users-v1 has the roll-forward function applied to it. Data saved via Users-v1 that is accessed via Users-v0 has the rollback function applied to it.

DB migration screen showing old DB field names, rollforward and rollback expressions, and instructions to activate the migration.

You use feature flags to switch DB calls to Users-v0 over to Users-v1. You can do this a single HTTP handler at a time to reduce the risk, and feature flags allow you to enable them for single users to ensure they work as planned. Once there are no uses of Users-v0 anymore, Dark will convert all remaining data in the background, from the old format to the new format. This should be invisible to you.

Testing

In Dark, the editor automatically runs unit tests in the background for code that’s being edited, and runs those tests across all feature flags by default. We also expect to use our static types to automatically fuzz your code, finding bugs for you.

Dark also runs your infrastructure in production, and this allows new abilities. We automatically save HTTP requests to Dark infrastructure (currently we save them all but we’ll move to sampling in the future). We test new code against those as well as unit tests, and make it easy to convert interesting requests to unit tests.

What have we removed?

Comparison of a standard continuous delivery pipeline (left) and how continuous delivery works in Dark (right). Dark’s delivery has 6 steps and one loop, while the traditional version has 35 steps and three loops.

A deployment in Dark has only 6 steps and one loop (steps that are repeated multiple times), while the modern Continuous Delivery pipeline has 35 steps and three loops. In Dark, tests run automatically behind the scenes; dependencies install automatically; anything involving git or Github is no longer necessary; building, testing and pushing Docker containers are not necessary; and no Kubernetes deploy steps are needed.

Even the steps which remain are simpler in Dark. Since feature flags can be completed or dismissed in a single atomic action, there is no need to go through the entire deploy pipeline a second time to remove the old code.

This is as close to the essential complexity of shipping code as we can imagine, significantly reducing both the time and the risk of continuous delivery. We’ve also significantly simplified upgrading packages, DB migrations, testing, version control, installing dependencies, dev/production parity, and upgrading quickly and safely between language versions.

To read more about the design of Dark, check out What is Dark, follow us on Twitter (or just me), or sign up to the beta list and to be notified of future posts. If you’re coming to StrangeLoop in September, come to our launch.

Thanks to Ben Beattie-Hood, Charity Majors, Chris Biscardi, Craig Kerstiens, Daniel Mangum, Domantas Mauruča, Eric Ries, Jennifer Wei, John Obelenus, Jorge Ortiz, K. Rainbolt-Greene, Kent Beck, Matthew Wilson, and Russell Smith for reading drafts of this post.