Revamp Elixir CLI

[josevalim]

This is a proposal to add five new options to Elixir's CLI with the goal of having releases into Elixir in the future.

In order to start an Elixir application in a release, there is a lot of work that has to be done by the underlying script: setting up the HEART_COMMAND, configuring boot files and boot vars, defining run_erl and making sure both pipe and log directories exist.

Because we want the scripts in an Elixir release to be minimal, we will move the complexity of managing those steps directly to the elixir/iex executables. Besides making releases more manageable, it means features like heart and run_erl become easily accessible to Elixir developers even outside of releases.

The options

This PR adds 5 new options to elixir CLI:

• --pipe-to PIPEDIR LOGDIR - invokes run_erl with daemon pipe and logs. This option is only for UNIX-like systems. Releases on Windows use a separate schema.
• --rpc-eval NODE COMMAND - evaluates the given expression on the given node
• --boot FILE, --boot-var VAR VALUE and --vm-args FILE - which are equivalent to Erlang's -boot, -boot_var and -args_file

By adding those options, we can build a script that would be part of a release and allows you to start, enable a remote console, perform rpcs (which includes stop, restart, pids, etc) in 70 LOC:

#!/bin/sh
set -e

SELF=$(readlink "$0" || true)
if [ -z "$SELF" ]; then SELF="$0"; fi
REL_ROOT="$(cd "$(dirname "$SELF")/.." && pwd -P)"
REL_NAME="demo"
REL_VSN="$(cut -d' ' -f2 "$REL_ROOT/releases/start_erl.data")"
export RELEASE_DIR="${RELEASE_DIR:-"$REL_ROOT/releases/$REL_VSN"}"
export COOKIE=${COOKIE:-$(cat "$REL_ROOT/releases/COOKIE")}

gen_id () {
  od -t x -N 4 /dev/urandom | head -n1 | awk '{print $2}'
}

rpc () {
  exec "$RELEASE_DIR/elixir" \
       --hidden --name "rpc-$(gen_id)@127.0.0.1" --cookie "$COOKIE" \
       --boot "${RELEASE_DIR}/start" \
       --boot-var RELEASE_ROOT "$REL_ROOT" \
       --rpc-eval "$REL_NAME@127.0.0.1" "$1"
}

start () {
  exec "$RELEASE_DIR/$1" --no-halt \
       --werl --cookie "$COOKIE" \
       --boot "${RELEASE_DIR}/${REL_NAME}" \
       --boot-var RELEASE_ROOT "$REL_ROOT" \
       --vm-args "${RELEASE_DIR}/vm.args" "${@:2}"
}

case $1 in
  start)
    start ${2:-elixir}
    ;;

  daemon)
    export RELEASE_TMP="${RELEASE_TMP:-"$REL_ROOT/tmp"}"
    start ${2:-elixir} --pipe-to "${RELEASE_TMP}/pipe" "${RELEASE_TMP}/log"
    ;;

  remote)
    exec "$RELEASE_DIR/iex" \
         --werl --hidden --name "remsh-$(gen_id)@127.0.0.1" --cookie "$COOKIE" \
         --boot "${RELEASE_DIR}/start" \
         --boot-var RELEASE_ROOT "$REL_ROOT" \
         --remsh "$REL_NAME@127.0.0.1"
    ;;

  rpc)
    if [ -z "$2" ]; then
      echo "BAD RPC" >&2
      exit 1
    fi
    rpc "$2"
    ;;

  restart|stop)
    rpc "System.$1"
    ;;

  pid)
    rpc "IO.puts System.pid"
    ;;

  *)
    echo "BAD COMMAND" >&2
    exit 1
    ;;
esac

This PR also reorganizes the CLI help instructions to make understanding all of those options slightly more manageable. It also trims down the iex CLI and makes it point to elixir -h. I am looking for feedback on the new options as well as on the new help structure.

Summary

We are not quite sure yet how Elixir releases will look like but this is the first step. Given we have 6 months until the next Elixir release, we have plenty of time to remove or add any new options if necessary. In fact, we know we will need at least one extra option for Windows support, but that will be tackled in another PR.

[josevalim]

I have implemented --rpc-eval and also added these features to Windows. This should be good to go (although I am waiting for CI on Windows too).

[ferd]

Be careful -- run_erl basically calls fsync on each line of output (https://github.com/erlang/otp/blob/f145ef83ea706ecb560be4f619b3c21c01c5f5b1/erts/etc/unix/run_erl.c) -- if you run it with lots of stdout logging or data, you can basically force your node to crawl to a halt.

For relx/rebar3/erlang-in-anger, we've basically decided to advise against using it for this reason.

[josevalim]

Thanks for the heads up @ferd. @tsloughter also told me to be careful with it so we will move it to a separate mode start mode called daemon. The sample start script has been updated.

[josevalim]

@ferd btw, was the fsync behaviour ever brought to discussion with the OTP team? Any chance we can put it behind a flag and/or disable it by default instead?

[josevalim]

@fhunleth @mobileoverlord @ConnorRigby I have some Nerves related questions:

1. Do you ever use -detached / —detached flag?

2. Are you using run_erl? run_erl requires you to have a mutable dir, which is where your logs go to, so what does Nerves do in this case?

3. Do you start the release using elixir or iex? I remember connecting to the device in the Nerves workshop. Do you always connect to the device remotely or is it also possible to connect it via the shell or some pipe?

Thanks!

[tsloughter]

Just to have it here, adding on what Fred said, aside from performance issues with run_erl it also doesn't seem to have much use today. Even if you aren't running in a container environment (like kubernetes or heroku where you output logs to stdout) you likely have systemd or similar for handling log rotation.

[ConnorRigby]

Do you ever use -detached / —detached flag?

Not that i know of. I'll let someone else answer for sure though.

Are you using run_erl? run_erl requires you to have a mutable dir, which is where your logs go to, so what does Nerves do in this case?

Frank just told me about run_erl pretty recently. I don't think we are using it quite yet, but i at least would like to use it in the future. Logs would likely go to either /root (the read/write data partition) or to /tmp.

Do you start the release using elixir or iex?

Nerves doesn't use Elixir at all to start a release on the device itself. erlinit starts the OTP release similarly to how one would manually start a distillery release. You can see how it is started here

Do you always connect to the device remotely ...

In Nerves we will have a UART console on the device itself, or a console on HDMI on devices that support that just like it were a regular machine. We also allow connecting via ssh and vie erlang distribution with nerves_init_gadget. Both of those mechanisms work more or less the same as on say a cloud server. You connect with
ssh nerves.local or ssh 192.168.x.x for ssh or with iex --sname console --cookie democookie --remsh app@nerves.local

[josevalim]

Nerves doesn't use Elixir at all to start a release on the device itself. erlinit starts the OTP release similarly to how one would manually start a distillery release. You can see how it is started here

So if I understand this correctly... you don't even use shell scripts to start it, which means that whatever we are doing here at the CLI level does not make a difference to Nerves. Is this correct?

Thanks @ConnorRigby!

[ConnorRigby]

@josevalim Correct. Nerves starts an OTP release with erlexec on the device. On the host side (on your development PC for example), a Nerves app is started like any other Elixir application.
ie iex -S mix

[fhunleth]

I think Connor answered most of the questions, and the most important one, that we don't use the shell scripts. For completeness, here's info on the parts he skipped.

1. Does Nerves ever start --detached?

No. Nerves' erlinit only supports starting the OTP release in the foreground. It waits for exits or crashes to restart the device or take some other action.

2. run_erl

Very few people use run_erl with Nerves. Our console frequently runs at 115,200 baud and loses data, so we advise people to avoid using it for logging. I recently caught a NIF outputting useful debug data to the startup console and that's the only reason I use run_erl. run_erl logs are stored under /tmp which is writable on Nerves.

[ferd]

@ferd btw, was the fsync behaviour ever brought to discussion with the OTP team? Any chance we can put it behind a flag and/or disable it by default instead?

Never discussed it with them. Mostly the disterl solution was always available (including on windows) and faster to us so we just made the switch. Plus in some scenarios, we'd start before the root disk partition was mounted in write mode.

[josevalim]

Hi everyone, this is ready to go and I plan to fully merge it tomorrow.

I have an application with barebones releases using those features here: https://github.com/josevalim/potato. Note you will need to use this branch. Run mix release and everything should work flawlessly. run_erl is not used by default, unless you run in daemon mode. You can see the latest version of the script here.

The next steps are:

1. Provide minimal releases to core (i.e. without relups and without configuration). I should have a pull request that achieves this still this week. Most of the work is already done in the potato project but there are many TODOs pending (including docs and tests)

2. Investigate relups and configuration

[eksperimental]

let's separate words with an underscore

Suggested change

	--pipe-to \"PIPEDIR\" \"LOGDIR\" Starts the Erlang VM as a named PIPEDIR and LOGDIR
	--pipe-to \"PIPE_DIR\" \"LOG_DIR\" Starts the Erlang VM as a named PIPE_DIR and LOG_DIR

[eksperimental]

Suggested change

	echo "--pipe-to : PIPEDIR cannot be a switch" >&2
	echo "--pipe-to : PIPE_DIR cannot be a switch" >&2

[eksperimental]

Suggested change

	It will attempt to create PIPEDIR and LOGDIR if they don't exist.
	It will attempt to create PIPE_DIR and LOG_DIR if they don't exist.

[eksperimental]

Suggested change

	See run_erl to learn more. To reattach, run: to_erl PIPEDIR.
	See run_erl to learn more. To reattach, run: to_erl PIPE_DIR.

[eksperimental]

Suggested change

	echo --pipe-to "PIPEDIR" "LOGDIR" Starts the Erlang VM as a named PIPEDIR and LOGDIR
	echo --pipe-to "PIPE_DIR" "LOG_DIR" Starts the Erlang VM as a named PIPE_DIR and LOG_DIR

[eksperimental]

Suggested change

	echo "--pipe-to : LOGDIR cannot be a switch" >&2
	echo "--pipe-to : LOG_DIR cannot be a switch" >&2

[josevalim]

Erlang refers to those and other environment variables without snake case so I am going with their convention for now.

[eksperimental]

All these options in this sections can be sorted alphabetically

[fertapric]

Should we just exit? I think at least elixir --help, or elixir -h should return 0.

[josevalim]

I believe it was because if Elixir is accidentally called without argument or something of sorts. So we probably need to check why we are invoking this. Let's do that in a separate occasion.