Releases

[josevalim]

This is a meta issue to track the progress of releases in Elixir. The goal is to include minimum support for releases, then add user conveniences, and then finally work on hot code upgrades with relups and appups.

Milestone 1

The goal here is to add releases to core without many bells and whistles:

• Add new options to Elixir CLI to trim the size of release executables (Revamp Elixir CLI #8595)
• Add basic mix release support to Elixir ([WIP] mix release support #8677)

Milestone 2

The goal here is implement user facing features:

• Add eval, version, start_iex, daemon_iex commands
• Custom steps
• Custom vm.args (and possibly rel/vm.args.eex) and bin/start (and possibly rel/start.eex) files (possibly with a generator?)
• Release configuration (via config providers) and a rewrite of the existing Mix.Config. The default config provider will look a config at config_path |> Path.dirname() |> Path.join("release.exs") and always log if a release config is being used or not. We also need to raise a good error message if Mix.Config is used instead of Config

[josevalim]

Hi folks,

I have spent most of the week studying hot code upgrades and building prototypes and I have decided to not include them as part of Elixir Core for now (at least for Elixir v1.9). The rationale is that they are still very complex, error prone and opinionated in a way that whoever is performing them needs to be aware of many of the decisions taken. For example, the distillery guides on appups (which is a part of hot code upgrades) covers many of those topics.

We are not saying they will never be part of Elixir but not for now. This can be a good opportunity for the community to build on top of what Elixir provides.

I wrote an extensive section on the docs about hot code upgrades which I reproduce below for convenience that explains this rationale and guides users.

---

Erlang and Elixir are sometimes known for the capability of upgrading
a node that is running in production without shutting down that node.
However, this feature is not supported out of the box by Elixir releases.

The reason we don't provide hot code upgrades is because they are very
complicated to perform in practice, as they require careful coding of
your processes and applications as well as extensive testing. Given most
teams can use other techniques that are language agnostic to upgrade
their systems, such as Blue/Green deployments, Canary deployments,
Rolling deployments, and others, hot upgrades are rarely a viable
option. Let's understand why.

In a hot code upgrade, you want to update a node from version A to
version B. To do so, the first step is to write recipes for every application
that changed between those two releases, telling exactly how the application
changed between versions, those recipes are called .appup files.
While some of the steps in building .appup files can be automated,
not all of them can. Furthermore, each process in the application needs
to be explicitly coded with hot code upgrades in mind. Let's see an example.
Imagine your application has a counter process as a GenServer:

defmodule Counter do
  use GenServer

  def start_link(_) do
    GenServer.start_link(__MODULE__, :ok, name: __MODULE__)
  end

  def bump do
    GenServer.call(__MODULE__, :bump)
  end

  ## Callbacks

  def init(:ok) do
    {:ok, 0}
  end

  def handle_call(:bump, counter) do
    {:reply, :ok, counter + 1}
  end
end

You add this process as part of your supervision tree and ship version
0.1.0 of your system. Now let's imagine that on version 0.2.0 you added
two changes: instead of bump/0, that always increments the counter by
one, you introduce bump/1 that passes the exact value to bump the
counter. You also change the state, because you want to store the maximum
bump value:

defmodule Counter do
  use GenServer

  def start_link(_) do
    GenServer.start_link(__MODULE__, :ok, name: __MODULE__)
  end

  def bump(by) do
    GenServer.call(__MODULE__, {:bump, by})
  end

  ## Callbacks

  def init(:ok) do
    {:ok, {0, 0}}
  end

  def handle_call({:bump, by}, {counter, max}) do
    {:reply, :ok, {counter + by, max(max, by)}}
  end
end

If you to perform a hot code upgrade in such application, it would
crash, because in the initial version the state was just a counter
but in the new version the state is a tuple. Furthermore, you changed
the format of the call message from :bump to {:bump, by} and
the process may have both old and new messages temporarily mixed, so
we need to handle both. The final version would be:

defmodule Counter do
  use GenServer

  def start_link(_) do
    GenServer.start_link(__MODULE__, :ok, name: __MODULE__)
  end

  def bump(by) do
    GenServer.call(__MODULE__, {:bump, by})
  end

  ## Callbacks

  def init(:ok) do
    {:ok, {0, 0}}
  end

  def handle_call(:bump, {counter, max}) do
    {:reply, :ok, {counter + 1, max(max, 1)}}
  end

  def handle_call({:bump, by}, {counter, max}) do
    {:reply, :ok, {counter + by, max(max, by)}}
  end

  def code_change(_, counter, _) do
    {:ok, {counter, 0}}
  end
end

Now you can proceed to list this process in the .appup file and
hot code upgrade it. This is one of the many steps one necessary
to perform hot code upgrades and it must be taken into account by
every process and application being upgraded in the system.
The .appup cookbook
provides a good reference and more examples.

Once .appups are created, the next step is to create a .relup
file with all instructions necessary to update the release itself.
Erlang documentation does provide a chapter on
Creating and Upgrading a Target System.
Learn You Some Erlang has a chapter on hot code upgrades.

Overall, there are many steps, complexities and assumptions made
during hot code upgrades, which is ultimately why they are not
provided by Elixir out of the box. However, hot code upgrades can
still be achieved by teams who desire to implement those steps
on top of mix release in their projects or as separate libraries.

[OvermindDL1]

Although not achieved out of the box now, I think it is important to not only support them 'soon' but also have an easy way to test (ExUnit addition?) to make sure a given application 'can' be hot upgraded safely and for this to become part of the standard templates. Even if the feature is not used by an end-project this still helps ensure that program has a sense of 'migrations' and code_change's defined properly to reason about the information.

As for a more Elixir'y style, what about a set of migrations that migrate from version to version that define code_change functions and things that appup/relup's need and thus can be generated from that Migration code (ala Ecto.Migration style).

[OvermindDL1]

The main issue I think would happen if its not supported 'soon' in the mix release lifespan is that then a lot of libraries won't support it properly, and thus end projects that do use the functionality will get unexpected breakages that they may not notice for a period of time in production, where trying to enforce a way to do it well 'soon', especially with a testing infrastructure for it (given a 'migration' file and so forth and pulling different releases from hex/git of a library as an example then try to upgrade it it while performing tests) will help ensure that libraries handle it properly as soon as possible, thus making the ecosystem more hardy to the design.

[josevalim]

Those are very good points @OvermindDL1. I particularly like the ExUnit idea.

I think it reinforces the point that it needs a lot of work and exploration before it gets to fully be part of core. We can also do it in steps too. For example, we can include a way to define .appups and test them before we support relups in core per se.

But I also think we need to consider that, even if those functionalities are in Elixir, some people simply won't define the .appups, so tracking if appups are available or not is also necessary.

[bitwalker]

It is quite easy to automate a lot of the .appup work (as Distillery does), but it is still the case that code_change implementations are entirely on users, including for dependencies, but in theory it should be possible to identify all the places where the state structure of a process has changed without a corresponding code_change implementation; with a combination of comparing whether a module has changed, and analyzing the AST of the changed module (there are a few obvious places in the source to check, and if in doubt, be pessimistic and notify the user they need to check that module manually).

There are a number of people that have used hot upgrades with Distillery, and aside from some issues that were bugs in Distillery, I think the foundation it provided was better than having to do it all by hand.

I don't think hot upgrades are important to have right now, but I do think that there is a path towards supporting them in a way that simply takes away a lot of the manual work one has to do. In the near term, tools which build on core can provide hot upgrades (i.e. Distillery would likely continue to provide that capability as an additional feature).

[josevalim]

The last PRs are here: #8966 #8967 #8957. With them merged, this is effectively done! Beta testers welcome! ❤️