Run multiple versions of ArangoDB in parallel using the .tar.gz distribution

This post uses the new `.tar.gz` binary distribution of ArangoDB to run multiple versions of ArangoDB alongside each other on the same machines. We will do a production-ready deployment on 3 cloud instances with authentication, TLS encryption, (self-signed) certificates and `systemd` service. In the end, we show how to perform a rolling upgrade for one of the clusters to a new version.

Interested in trying out ArangoDB? Fire up your cluster in just a few clicks with ArangoDB ArangoGraph: the Cloud Service for ArangoDB. Start your free 14-day trial here

The new `.tar.gz` binary archive

Starting with Version 3.4.0 we have changed the way how we build our binary distribution. We are now using completely static binaries (see here for another interesting story) which makes it possible to run the same binary on any Linux distribution and version. As a consequence, we can publish a `.tar.gz` binary archive, which can be installed as any user at any location in the filesystem by extracting the archive. So the first step I did on my three instances was to download and extract the archive:

cd
wget https://download.arangodb.com/arangodb34/Community/Linux/arangodb3-linux-3.4.1.tar.gz
tar xzvf arangodb3-linux-3.4.1.tar.gz

which produced a directory `arangodb3-3.4.1` in my home directory `/home/neunhoef`. The idea is now that the ArangoDB starter `arangodb` can simply be run from the `bin` directory in that directory and will automatically find the rest of the distribution.

Setting up a CA and certificates

Since we want to do things properly, we want to protect all network traffic between instances and from the client to the database by TLS. Therefore we need certificates, more precisely, we want to use a self-signed CA and use this to produce a server certificate and key. Since we sign the server certificate with the CA, we can then give the CA certificate to our browser and thereby silence the nasty security warnings because the browser cannot establish a complete chain of trust from a root certificate it knows.

Fortunately, this is all very simple using the ArangoDB starter executable:

/home/neunhoef/arangodb3-3.4.1/bin/arangodb create tls ca
/home/neunhoef/arangodb3-3.4.1/bin/arangodb create tls keyfile \
    --host 34.73.36.211 --host 34.73.162.104 --host 34.73.115.255 \
    --host 10.142.0.3 --host 10.142.0.4 --host 10.142.0.2 \
    --host a.9hoeffer.de --host b.9hoeffer.de --host c.9hoeffer.de

Note that I simply had to use the full path to the starter executable to get it to work.

The first of these commands creates the CA with the two files `tls-ca.key` (private key) and `tls-ca.crt` (certificate). The second command produces a server certificate, for convenience I include all three public and private IP addresses as host names as well as the three DNS names I will use in my domain `9hoeffer.de`. This makes the server certificates valid for all three hosts. The internal IP addresses are needed for cluster-internal communication, the DNS names are needed for external access. I could have dropped the external IP addresses here, but never mind.

After this, I quickly stuff away `tls-ca.key` in a secure place, import `tls-ca.crt` as a trusted certificate agency certificate to verify server certificates into my browser, and copy `tls.keyfile` (containing both the private key and the certificate) to all three machines as `/home/neunhoef/arangodb.keyfile` with permissions `400`.

Preparations with volumes

If you wanted to use special volumes for the database, now would be the time. You could have locally attached SSDs for particularly fast performance, or create network volumes. All you would have to do here would be to mount the volume in a directory and make it available to the user account under which you want to run the database.

I am lazy, I simply use my home directory and create a directory `d` there by doing

cd
mkdir d

on all three machines.

Setting up the first cluster using ArangoDB 3.4.1

The only piece that is now missing is to create a shared secret such that the ArangoDB instances on the three machines can authenticate each other. To this end I just create a file `/home/neunhoef/arangodb.secret` with permissions `400` with these commands on all three machines:

cd
echo -n "this is a very secret sentence" > arangodb.secret
chmod 400 arangodb.secret

Note that we have not used `sudo` or the root account in any way so far.

We could now fire up the cluster with a single command, but we want to do things properly, therefore we want to set up a `systemd` service for our cluster. This needs root privileges, though. Furthermore, this now needs a Linux variant which uses `systemd`, I used Ubuntu 18.04, for example.

We simply use

sudo vi /etc/systemd/system/arangodb.service

to create the following file in `/etc/systemd/system/arangodb.service`:

[Unit]
Description=Run the ArangoDB Starter
After=network.target

[Service]
# system limits
LimitNOFILE=131072
LimitNPROC=131072
TasksMax=131072
User=neunhoef
Group=neunhoef
Restart=on-failure
KillMode=process
ExecStart=/home/neunhoef/arangodb3-3.4.1/bin/arangodb \
    --starter.address=10.142.0.3 \
    --starter.data-dir=/home/neunhoef/d \
    --starter.join=10.142.0.3:8528,10.142.0.4:8528,10.142.0.2:8528 \
    --auth.jwt-secret=/home/neunhoef/arangodb.secret \
    --ssl.keyfile=/home/neunhoef/arangodb.keyfile
TimeoutStopSec=60

[Install]
WantedBy=multi-user.target

Note that since I am using the same directory layout on all three machines, the file can look exactly the same on all three machines, with a single exception. The option `–starter.address` of course must be the internal IP address of the particular machine on which we deploy this file. So this has to be adjusted for each file. Note that the `–starter.join` option allows to use a totally symmetric setup with the very same information for all three machines, namely a comma-separated list of all internal IP addresses and ports that the starters use.

After that we can immediately fire up the cluster with these commands (on all three machines):

sudo systemctl daemon-reload
sudo systemctl start arangodb

The first of these commands makes `systemd` discover the new service file, the second starts the service with the name `arangodb`.

You can use these commands to diagnose any problems or see that it works:

systemctl status arangodb
journalctl -u arangodb

Since I have put the public IP addresses of my three machines in the DNS server for my domain and have opened port 8529 in the firewall on all three machines (I used the Google Cloud Console for this), I could now immediately point my browser to `https://a.9hoeffer.de:8529` and get the graphical UI of ArangoDB.

If you do this at home, do not forget to immediately change the password of the `root` user from empty to something sensible!

Setting up the second cluster using ArangoDB 3.4.0

One of the main purposes of this article is to demonstrate how to run multiple versions of ArangoDB on the same machine(s). Therefore I pick another such version, namely 3.4.0 (the only other 3.4 version which is available to me at the time of this writing). I can reuse the certificates and the secret, and therefore I essentially only do this on all machines to deploy the second cluster:

cd
wget https://download.arangodb.com/arangodb34/Community/Linux/arangodb3-linux-3.4.0.tar.gz
tar xzvf arangodb3-linux-3.4.0.tar.gz 
mkdir d2
sudo cp /etc/systemd/system/arangodb.service /etc/systemd/system/arangodb2.service
sudo vi /etc/systemd/system/arangodb2.service

and edit `/etc/systemd/system/arangodb2.service` to look like this:

[Unit]
Description=Run the ArangoDB Starter second instance 
After=network.target

[Service]
User=neunhoef
Group=neunhoef
Restart=on-failure
KillMode=process
ExecStart=/home/neunhoef/arangodb3-3.4.0/bin/arangodb \
    --starter.port=9528 \
    --starter.address=10.142.0.3 \
    --starter.data-dir=/home/neunhoef/d2 \
    --starter.join=10.142.0.3:9528,10.142.0.4:9528,10.142.0.2:9528 \
    --auth.jwt-secret=/home/neunhoef/arangodb.secret \
    --ssl.keyfile=/home/neunhoef/arangodb.keyfile
TimeoutStopSec=60

[Install]
WantedBy=multi-user.target

All I had to change was the Description, the path to the `arangodb` executable, the database directory `/home/neunhoef/d2` as well as the ports used by the starters, which is now 9528. Therefore I had to add the option `–starter.port=9528`. Of course, I needed to open the port 9529 in the firewall as well. The port used by the database coordinator is always one greater than the port the starter is using.

After that,

sudo systemctl daemon-reload
sudo systemctl start arangodb2

fired up the second cluster.

Performing a rolling upgrade

We will conclude this post by showing how we can upgrade the second cluster smoothly to Version 3.4.1 as well. We are using the rolling upgrade feature which is built into the starter `arangodb`. We follow the procedure as documented in this manual section.

The first step is to install the new version. With the `.tar.gz` binary distribution this can simply be done by extracting the archive in some directory. In our particular case here, we have already installed Version 3.4.1, so nothing has to be done.

The second step is to stop the three old starters and restart them with the new executable. Note that only the starter processes `arangodb` should be restarted, not the `arangod` processes. This is important to avoid any service interruption.

We achieve this by editing the `systemd` service file in `/etc/systemd/system/arangodb2.service` and adjusting the path to the `arangodb` executable to be run from

ExecStart=/home/neunhoef/arangodb3-3.4.0/bin/arangodb \

to

ExecStart=/home/neunhoef/arangodb3-3.4.1/bin/arangodb \

Then, after doing

sudo systemctl daemon-reload

to tell `systemd` about the change, we can simply send

kill -9 PID

with `PID` replaced by the process ID of the running `arangodb` process.
Use

systemctl status arangodb2

to find this process ID under `Main ID`. Do not use

sudo systemctl kill -s SIGKILL arangodb2

since this would kill the `arangod` subprocesses, too! Furthermore, note that we intentionally added the line

KillMode=process

to all `systemd` service files, otherwise `systemd` would kill all subprocesses alongside with the `arangodb` process.

Once the kill has happened, `systemd` will subsequently automatically restart the new `arangodb` executable which will pick up the running `arangod` processes.

Once this has been performed on all three machines, we are in a position to run the rolling upgrade. To this end, issue the following command on exactly one of the machines only:

/home/neunhoef/arangodb3-3.4.1/bin/arangodb upgrade \
      --starter.endpoint=https://localhost:9528

This will contact the new starter process running on the same machine and all three starter processes on the three machines will work together to perform a smooth rolling upgrade to the new version. On the console you will see progress being logged as follows:

2019-01-10T21:10:43Z |INFO| Database automatic upgrade from version 3.4.0 to version 3.4.1 has been started component=arangodb
2019-01-10T21:10:43Z |INFO| Servers upgraded: none, remaining servers: 3 agents, 3 coordinators, 3 dbservers component=arangodb
2019-01-10T21:10:51Z |INFO| Servers upgraded: 1 agent, remaining servers: 2 agents, 3 coordinators, 3 dbservers component=arangodb
2019-01-10T21:11:01Z |INFO| Servers upgraded: 2 agents, remaining servers: 1 agent, 3 coordinators, 3 dbservers component=arangodb
2019-01-10T21:11:05Z |INFO| Servers upgraded: 3 agents, remaining servers: 3 coordinators, 3 dbservers component=arangodb
2019-01-10T21:11:13Z |INFO| Servers upgraded: 3 agents, 1 dbserver, remaining servers: 3 coordinators, 2 dbservers component=arangodb
2019-01-10T21:11:22Z |INFO| Servers upgraded: 3 agents, 2 dbservers, remaining servers: 3 coordinators, 1 dbserver component=arangodb
2019-01-10T21:11:30Z |INFO| Servers upgraded: 3 agents, 3 dbservers, remaining servers: 3 coordinators component=arangodb
2019-01-10T21:11:37Z |INFO| Servers upgraded: 3 agents, 1 coordinator, 3 dbservers, remaining servers: 2 coordinators component=arangodb
2019-01-10T21:11:44Z |INFO| Servers upgraded: 3 agents, 2 coordinators, 3 dbservers, remaining servers: 1 coordinator component=arangodb
2019-01-10T21:11:51Z |INFO| Database upgrade has finished component=arangodb

The mission has been accomplished.