MERII Hummingbird A80 Optimus Cluster: ArangoDB Deployment

For running ArangoDB in clusters doing performance tests we wanted to have a non virtualized set of descent hardware with fast ethernet connection, enough RAM (since thats what Arango needs) and multicore CPU. Since you need a bunch of them, cheap ARM devel boards come to mind. The original Raspberry PI (we have those) is out of the game due to V8 is not supporting it anymore. The now available PI 2 doesn’t cut it, since its ethernet NIC is connected via USB (as on the original PI). The Odroid series only have one of both: Fast ethernet or enough RAM. The Cubieboard 4 wasn’t available yet, but its Allwinner A80 SOC seemed a good choice. Then we met the Merii Optimus board, which seems to be almost the same as the PCDuino (now renamed to Arches) with the A80. While we got a bunch of them for a descent price over at Pollin, the upstream support wasn’t that good.

However, with some help of the SunXi-Linux Project we started flashing OS images to replace the preloaded Android image with the Merii Linux image. Since the userland of the Merii image is pretty sparse, we wanted something more useable. There is already a how-to on running Ubuntu which requires running a Windows host. We prefer a Linux host and want to run a Debian. Since the new Pi2 is also able to run regular Debian with ArmV7, we pick the root fs from sjoerd.

Building the Merii SDK

The works of Merii have collected a little dust on their way from china, so it doesn’t work out of the box on a recent Debian. Our buildhost is an Amd64 Jessie using multiarch to also be able to run i386 binaries from the SDK. Verify that multiarch is working properly:

a80_MerriiLinux_optimus_v1$ ldd ./tools/pack/pctools/linux/eDragonEx/dragon
    linux-gate.so.1 (0xf77d8000)
    libdl.so.2 => /lib/i386-linux-gnu/i686/cmov/libdl.so.2 (0xf77a5000)
    libstdc++.so.6 => /usr/lib/i386-linux-gnu/libstdc++.so.6 (0xf76b3000)
    libm.so.6 => /lib/i386-linux-gnu/i686/cmov/libm.so.6 (0xf766c000)
    libgcc_s.so.1 => /lib/i386-linux-gnu/libgcc_s.so.1 (0xf764f000)
    libc.so.6 => /lib/i386-linux-gnu/i686/cmov/libc.so.6 (0xf74a5000)
    /lib/ld-linux.so.2 (0xf77d9000)

Since we want to run a more recent userland with systemd, we need to enable CONFIG_FANOTIFY – else /etc/fstab entries for the SD-Card will fail to mount during boot. We edit linux-3.4/.config and change the line:

# CONFIG_FANOTIFY is not set

To enabled:

CONFIG_FANOTIFY=y

After some fiddling, bending download URLs to places where the software still can be found, picking up dangling symlinks, and waiting for the Compiler to do its job, we get:

./build.sh -p sun9iw1p1
./build.sh pack
ls -l tools/pack/sun9iw1p1_dragonboard_optimus.img
  -rwxr-xr-x 1 willi willi 61M May 27 11:47 tools/pack/sun9iw1p1_dragonboard_optimus.img*

This is an image intended to be flashed to the board using the sunxi Livesuite. As mentioned above, it only brings some busybox userland, which we want to pimp now. Use the output of dmesg to find the SD-Card device. Then flash the Raspberry pi disk image to an SD-Card:

bmaptool copy --nobmap jessie-rpi2-20150202.img /dev/sdd
  bmaptool: info: no bmap given, copy entire image to '/dev/sdd'
  bmaptool: info: 100% copied
  bmaptool: info: synchronizing '/dev/sdd'
  bmaptool: info: copying time: 6m 11.0s, copying speed 8.1 MiB/sec

and mount its root filesystem:

mkdir /pi2root
mount /dev/sdd2 /pi2root

Since we need to copy some things from Merii image, and we can’t mount it directly, we locate the raw root filesystem in the buildroot and mount it using the loopback device:

modprobe loop # load the loopback module, just in case...
mkdir orig_loopimage
mount out/sun9iw1p1/linux/common/rootfs.ext4 ./orig_loopimage -t ext4 -o loop=/dev/loop1 

Now we can access the files from the Merii root, and copy it (i.e. the kernel modules) so we have it available later on:

mkdir /pi2root/hummingroot
rm -rf /pi2root/lib/modules/3.18.0-trunk-rpi2 # remove the original kernel modules, space is sparse...
cd orig_loopimage
tar -cO bin sbin lib opt root test usr var |tar -xC /pi2root/hummingroot
mv /pi2root/hummingroot/lib/modules/* /pi2root/lib/modules
cd ..
umount orig_loopimage
df -h
  Filesystem      Size  Used Avail Use% Mounted on
  /dev/sdd2       2.8G  445M  2.2G  18% /pi2root

Sadly the generated filesystem doesn’t allow to enter the UBOOT prompt. So we need to modify kernel boot parameters in tools/pack/chips/sun9iw1p1/configs/default/env.cfg before packing the image. I.e. we may want to make systemd offer a rescue console (which helped me a lot, since fiddling serial console to work wouldn’t have been possible otherwise):

setargs_nand=setenv bootargs console=${console} root=${nand_root} init=${init} vmalloc=384M loglevel=${loglevel} partitions=${partitions} systemd.unit=rescue.target

Now we see that the disk is 32G, but we only have 512MB on the board for the root image. So now we need to create a smaller image so it fits:

mv out/sun9iw1p1/linux/common/rootfs.ext4 out/sun9iw1p1/linux/common/rootfs.ext4.org
dd if=/dev/zero of=out/sun9iw1p1/linux/common/rootfs.ext4 bs=1024 count=$((1024*512))
mkfs.ext4 out/sun9iw1p1/linux/common/rootfs.ext4
mkdir new_loopimage
mount out/sun9iw1p1/linux/common/rootfs.ext4 ./new_loopimage -t ext4 -o loop=/dev/loop2
WHERE="`pwd`/new_loopimage"
cd /pi2root
ln -s lib/systemd/systemd init

As Lars Vogdt describes in his excellent blog post, we fix the serial console:

cp lib/systemd/system/getty\@.service etc/systemd/system/getty.target.wants/getty\@ttyS0.service

Then edit etc/systemd/system/getty.target.wants/getty@ttyS0.service and set the correct baud rate:

...
[Service]
ExecStart=-/sbin/agetty %I 115200
...
[Install]
Alias=getty.target.wants/getty@ttyS0.service

We configure the ethernet adding / editing etc/network/interfaces.d/eth0:

iface eth0 inet static
        address 192.168.1.100
        gateway 192.168.1.1

Edit the etc/fstab and replace the pi specific stuff:

# <file system> <mount pt>     <type>   <options>         <dump> <pass>

/dev/mmcblk0p7  / ext4 relatime,errors=remount-ro,discard 0 1
# we want to chroot to the SD-Card later:
/dev/mmcblk1p1  /mnt ext4 relatime,errors=remount-ro 0 1

proc            /proc          proc     defaults          0      0
proc            /mnt/proc      proc     defaults          0      1
devpts          /dev/pts       devpts   defaults,gid=5,mode=620   0      0
devpts          /mnt/dev/pts   devpts   defaults,gid=5,mode=620   0     1 
tmpfs           /tmp           tmpfs    defaults          0      0
tmpfs           /mnt/tmp       tmpfs    defaults          0     1 
sysfs           /sys           sysfs    defaults          0      0
sysfs           /mnt/sys       sysfs    defaults          0     1 

Finally, we have to add some fix up to etc/rc.local:

cd /lib/modules/3.4.39

/sbin/insmod nand.ko
# power up the PHY for the NIC:
echo 3000000 > /sys/devices/platform/reg-15-cs-cldo1/max_microvolts
echo 3000000 > /sys/devices/platform/reg-15-cs-cldo1/min_microvolts

# some display driver stuff (if you need it ;-)
/sbin/insmod ./disp.ko
/sbin/insmod lcd.ko 
/sbin/insmod hdcp.ko
/sbin/insmod ./bcmdhd.ko

/sbin/insmod gt9xx_ts.ko
/sbin/insmod sunxi-ir-rx.ko

# now load the ethernet module:
/sbin/insmod sunxi_gmac.ko
/sbin/ifup eth0

Now we can copy everything into the root filesystem image:

tar -cO * |tar -xC ${WHERE}
cd ${WHERE}/..
umount new_loopimage
build.sh pack
ls tools/pack/sun9iw1p1_dragonboard_optimus.img -lh
  -rwxr-xr-x 1 root root 532M May 28 14:29 tools/pack/sun9iw1p1_dragonboard_optimus.img*

We have our beautiful Image which we can flash to the board. We first start the livesuite mentioned above, and select the image. Then we need to put the board into FEL-mode using minicom and the serial to USB cable that came with the board. If you’re able to access the UBOOT prompt, the efex command can do this. Otherwise holding down the 2 key during power up will do the job.

Some more links that helped me a lot to figure everything out:

• The very active linux sunxi community (who also helped me on IRC)
• How to Upgrade Allwinner A80 OptimusBoard Firmware (cnx-software)

Conclusion

It will take a while for the verry powerfull A80 SOC to gain ground. Part of this slow uptake is the very sparse communication policy of Allwinner and Merii. However, that seems to change a bit. With the Cubieboard 4 and the LinkSprite Arches (PCDuino) nice boards with a very active community will become available, I guess Olimex will follow soon. Expect them to arrive at exp-tech, Pollin or Watterott. Also devices with better WAF to look nice in your livingroom are about to arive.

In the next blog post we will install ArangoDB on this cluster and show some performance results. Therefore we will use fast micro-SD cards to provide additional disk-space.