Run multiple versions of ArangoDB in parallel using the .tar.gz distribution
This post uses the new `.tar.gz` binary distribution of ArangoDB to run multiple versions of ArangoDB alongside each other on the same machines. We will do a production-ready deployment on 3 cloud instances with authentication, TLS encryption, (self-signed) certificates and `systemd` service. In the end, we show how to perform a rolling upgrade for one of the clusters to a new version.
Interested in trying out ArangoDB? Fire up your cluster in just a few clicks with ArangoDB ArangoGraph: the Cloud Service for ArangoDB. Start your free 14-day trial here
The new `.tar.gz` binary archive
Starting with Version 3.4.0 we have changed the way how we build our binary distribution. We are now using completely static binaries (see here for another interesting story) which makes it possible to run the same binary on any Linux distribution and version. As a consequence, we can publish a `.tar.gz` binary archive, which can be installed as any user at any location in the filesystem by extracting the archive. So the first step I did on my three instances was to download and extract the archive:
cd wget https://download.arangodb.com/arangodb34/Community/Linux/arangodb3-linux-3.4.1.tar.gz tar xzvf arangodb3-linux-3.4.1.tar.gz
which produced a directory `arangodb3-3.4.1` in my home directory `/home/neunhoef`. The idea is now that the ArangoDB starter `arangodb` can simply be run from the `bin` directory in that directory and will automatically find the rest of the distribution.
Setting up a CA and certificates
Since we want to do things properly, we want to protect all network traffic between instances and from the client to the database by TLS. Therefore we need certificates, more precisely, we want to use a self-signed CA and use this to produce a server certificate and key. Since we sign the server certificate with the CA, we can then give the CA certificate to our browser and thereby silence the nasty security warnings because the browser cannot establish a complete chain of trust from a root certificate it knows.
Fortunately, this is all very simple using the ArangoDB starter executable:
/home/neunhoef/arangodb3-3.4.1/bin/arangodb create tls ca /home/neunhoef/arangodb3-3.4.1/bin/arangodb create tls keyfile \ --host 188.8.131.52 --host 184.108.40.206 --host 220.127.116.11 \ --host 10.142.0.3 --host 10.142.0.4 --host 10.142.0.2 \ --host a.9hoeffer.de --host b.9hoeffer.de --host c.9hoeffer.de
Note that I simply had to use the full path to the starter executable to get it to work.
The first of these commands creates the CA with the two files `tls-ca.key` (private key) and `tls-ca.crt` (certificate). The second command produces a server certificate, for convenience I include all three public and private IP addresses as host names as well as the three DNS names I will use in my domain `9hoeffer.de`. This makes the server certificates valid for all three hosts. The internal IP addresses are needed for cluster-internal communication, the DNS names are needed for external access. I could have dropped the external IP addresses here, but never mind.
After this, I quickly stuff away `tls-ca.key` in a secure place, import `tls-ca.crt` as a trusted certificate agency certificate to verify server certificates into my browser, and copy `tls.keyfile` (containing both the private key and the certificate) to all three machines as `/home/neunhoef/arangodb.keyfile` with permissions `400`.
Preparations with volumes
If you wanted to use special volumes for the database, now would be the time. You could have locally attached SSDs for particularly fast performance, or create network volumes. All you would have to do here would be to mount the volume in a directory and make it available to the user account under which you want to run the database.
I am lazy, I simply use my home directory and create a directory `d` there by doing
cd mkdir d
on all three machines.
Setting up the first cluster using ArangoDB 3.4.1
The only piece that is now missing is to create a shared secret such that the ArangoDB instances on the three machines can authenticate each other. To this end I just create a file `/home/neunhoef/arangodb.secret` with permissions `400` with these commands on all three machines:
cd echo -n "this is a very secret sentence" > arangodb.secret chmod 400 arangodb.secret
Note that we have not used `sudo` or the root account in any way so far.
We could now fire up the cluster with a single command, but we want to do things properly, therefore we want to set up a `systemd` service for our cluster. This needs root privileges, though. Furthermore, this now needs a Linux variant which uses `systemd`, I used Ubuntu 18.04, for example.
We simply use
sudo vi /etc/systemd/system/arangodb.service
to create the following file in `/etc/systemd/system/arangodb.service`:
[Unit] Description=Run the ArangoDB Starter After=network.target [Service] # system limits LimitNOFILE=131072 LimitNPROC=131072 TasksMax=131072 User=neunhoef Group=neunhoef Restart=on-failure KillMode=process ExecStart=/home/neunhoef/arangodb3-3.4.1/bin/arangodb \ --starter.address=10.142.0.3 \ --starter.data-dir=/home/neunhoef/d \ --starter.join=10.142.0.3:8528,10.142.0.4:8528,10.142.0.2:8528 \ --auth.jwt-secret=/home/neunhoef/arangodb.secret \ --ssl.keyfile=/home/neunhoef/arangodb.keyfile TimeoutStopSec=60 [Install] WantedBy=multi-user.target
Note that since I am using the same directory layout on all three machines, the file can look exactly the same on all three machines, with a single exception. The option `–starter.address` of course must be the internal IP address of the particular machine on which we deploy this file. So this has to be adjusted for each file. Note that the `–starter.join` option allows to use a totally symmetric setup with the very same information for all three machines, namely a comma-separated list of all internal IP addresses and ports that the starters use.
After that we can immediately fire up the cluster with these commands (on all three machines):
sudo systemctl daemon-reload sudo systemctl start arangodb
The first of these commands makes `systemd` discover the new service file, the second starts the service with the name `arangodb`.
You can use these commands to diagnose any problems or see that it works:
systemctl status arangodb journalctl -u arangodb
Since I have put the public IP addresses of my three machines in the DNS server for my domain and have opened port 8529 in the firewall on all three machines (I used the Google Cloud Console for this), I could now immediately point my browser to `https://a.9hoeffer.de:8529` and get the graphical UI of ArangoDB.
If you do this at home, do not forget to immediately change the password of the `root` user from empty to something sensible!
Setting up the second cluster using ArangoDB 3.4.0
One of the main purposes of this article is to demonstrate how to run multiple versions of ArangoDB on the same machine(s). Therefore I pick another such version, namely 3.4.0 (the only other 3.4 version which is available to me at the time of this writing). I can reuse the certificates and the secret, and therefore I essentially only do this on all machines to deploy the second cluster:
cd wget https://download.arangodb.com/arangodb34/Community/Linux/arangodb3-linux-3.4.0.tar.gz tar xzvf arangodb3-linux-3.4.0.tar.gz mkdir d2 sudo cp /etc/systemd/system/arangodb.service /etc/systemd/system/arangodb2.service sudo vi /etc/systemd/system/arangodb2.service
and edit `/etc/systemd/system/arangodb2.service` to look like this:
[Unit] Description=Run the ArangoDB Starter second instance After=network.target [Service] User=neunhoef Group=neunhoef Restart=on-failure KillMode=process ExecStart=/home/neunhoef/arangodb3-3.4.0/bin/arangodb \ --starter.port=9528 \ --starter.address=10.142.0.3 \ --starter.data-dir=/home/neunhoef/d2 \ --starter.join=10.142.0.3:9528,10.142.0.4:9528,10.142.0.2:9528 \ --auth.jwt-secret=/home/neunhoef/arangodb.secret \ --ssl.keyfile=/home/neunhoef/arangodb.keyfile TimeoutStopSec=60 [Install] WantedBy=multi-user.target
All I had to change was the Description, the path to the `arangodb` executable, the database directory `/home/neunhoef/d2` as well as the ports used by the starters, which is now 9528. Therefore I had to add the option `–starter.port=9528`. Of course, I needed to open the port 9529 in the firewall as well. The port used by the database coordinator is always one greater than the port the starter is using.
sudo systemctl daemon-reload sudo systemctl start arangodb2
fired up the second cluster.
Performing a rolling upgrade
We will conclude this post by showing how we can upgrade the second cluster smoothly to Version 3.4.1 as well. We are using the rolling upgrade feature which is built into the starter `arangodb`. We follow the procedure as documented in this manual section.
The first step is to install the new version. With the `.tar.gz` binary distribution this can simply be done by extracting the archive in some directory. In our particular case here, we have already installed Version 3.4.1, so nothing has to be done.
The second step is to stop the three old starters and restart them with the new executable. Note that only the starter processes `arangodb` should be restarted, not the `arangod` processes. This is important to avoid any service interruption.
We achieve this by editing the `systemd` service file in `/etc/systemd/system/arangodb2.service` and adjusting the path to the `arangodb` executable to be run from
Then, after doing
sudo systemctl daemon-reload
to tell `systemd` about the change, we can simply send
kill -9 PID
with `PID` replaced by the process ID of the running `arangodb` process.
systemctl status arangodb2
to find this process ID under `Main ID`. Do not use
sudo systemctl kill -s SIGKILL arangodb2
since this would kill the `arangod` subprocesses, too! Furthermore, note that we intentionally added the line
to all `systemd` service files, otherwise `systemd` would kill all subprocesses alongside with the `arangodb` process.
Once the kill has happened, `systemd` will subsequently automatically restart the new `arangodb` executable which will pick up the running `arangod` processes.
Once this has been performed on all three machines, we are in a position to run the rolling upgrade. To this end, issue the following command on exactly one of the machines only:
/home/neunhoef/arangodb3-3.4.1/bin/arangodb upgrade \ --starter.endpoint=https://localhost:9528
This will contact the new starter process running on the same machine and all three starter processes on the three machines will work together to perform a smooth rolling upgrade to the new version. On the console you will see progress being logged as follows:
2019-01-10T21:10:43Z |INFO| Database automatic upgrade from version 3.4.0 to version 3.4.1 has been started component=arangodb 2019-01-10T21:10:43Z |INFO| Servers upgraded: none, remaining servers: 3 agents, 3 coordinators, 3 dbservers component=arangodb 2019-01-10T21:10:51Z |INFO| Servers upgraded: 1 agent, remaining servers: 2 agents, 3 coordinators, 3 dbservers component=arangodb 2019-01-10T21:11:01Z |INFO| Servers upgraded: 2 agents, remaining servers: 1 agent, 3 coordinators, 3 dbservers component=arangodb 2019-01-10T21:11:05Z |INFO| Servers upgraded: 3 agents, remaining servers: 3 coordinators, 3 dbservers component=arangodb 2019-01-10T21:11:13Z |INFO| Servers upgraded: 3 agents, 1 dbserver, remaining servers: 3 coordinators, 2 dbservers component=arangodb 2019-01-10T21:11:22Z |INFO| Servers upgraded: 3 agents, 2 dbservers, remaining servers: 3 coordinators, 1 dbserver component=arangodb 2019-01-10T21:11:30Z |INFO| Servers upgraded: 3 agents, 3 dbservers, remaining servers: 3 coordinators component=arangodb 2019-01-10T21:11:37Z |INFO| Servers upgraded: 3 agents, 1 coordinator, 3 dbservers, remaining servers: 2 coordinators component=arangodb 2019-01-10T21:11:44Z |INFO| Servers upgraded: 3 agents, 2 coordinators, 3 dbservers, remaining servers: 1 coordinator component=arangodb 2019-01-10T21:11:51Z |INFO| Database upgrade has finished component=arangodb
The mission has been accomplished.
Get the latest tutorials, blog posts and news: