Performance Impact of Meltdown and Spectre V1 Patches on ArangoDB

To investigate the impact of the Meltdown and Spectre patches on the performance of ArangoDB, we ran benchmark tests with the two storage engines available in ArangoDB (MMFiles & RocksDB). We used the arangobench benchmark and test tool for these tests.

The tests include 10 different test cases with changing test parameters like concurrency, batch requests and asynchronous execution.

Test setup

We used the arangobench tool to execute a few different performance tests. arangobench is an easy-to-use tool to measure performance, and it comes bundled with every ArangoDB release.

The test we ran measure the performance (in terms of elapsed time) for mass document insertion, because that is one of the most common use cases. For these document insertion tests, we varied some of the test invocation parameters to see if the impact varies depending on the setup.

Here are a few of the options we varied in the tests:

• serial insertion (one client thread) vs. parallel insertion (multiple client threads)
• using one HTTP request per insert vs. batching multiple operations in a single HTTP request (batching)
• synchronous (blocking) requests vs. asynchronous (non-blocking) fire-and-forget requests

We combined these options sensibly to end up with 10 different test cases. The test cases were inspired/referenced from here

The hardware used for testing was a MSi CX62 7QL laptop. It had 8GB RAM and 7th generation Intel i5 processors clocked at 2.5GHz. The benchmark was run on Ubuntu 16.04. The Spectre and Meltdown security patches were applied as per the instructions here. The unpatched kernel version we used was 4.13.0-31 and the patched kernel was 4.13.0-32. The patched kernel version fixes the Meltdown bug and the Spectre V1 bug, but not the Spectre V2 bug. At the time the tests were run and this article was written, a patch for Spectre V2 was not available in an official Ubuntu release (we did not want to install firmware patches either, because issues were reported that the firmware updates can cause issues).

ArangoDB 3.3.2 was installed as per instructions here.

Results

The test results show uniform and almost symmetric execution times. We measured the arithmetic mean, median and standard deviations of the 20 test runs we did in each test case. Most importantly we examined the standard deviations of the execution times as it gives a quality rating of a central tendency and tells us how much the numbers deviate from the average. Standard deviation helped us to take a firm conclusion on performances before and after the Meltdown patch.

The following is one detailed test case of Concurrency-8 in MMFiles engine. The table shows all the 20 runs having uniform results.

Here are the standard deviations of all the test cases on MMFiles engine before and after installing the Meltdown patch. The standard deviations of RocksDB engine were also similar.

Both engines show a different performance impact behavior after installing the Meltdown patch. In general, the impact on both engines is lower than expected but still significant for MMfiles engine for some test cases. As the results are uniform and almost symmetric, the mean and median are almost the same so we decided to use the arithmetic mean to decide on performance impact and plot graphs.

MMFiles Engine

The results for the MMFiles engine show a varying performance impact between 0 and 17% across the test cases with the Spectre and Meltdown patches applied.

Here is a detailed graph showing the execution time for the different test cases before and after the patches were applied:

RocksDB Engine

After applying the security patches, performance decrease is small in comparison to the standard deviation of the measured results. Hence, there is no significant performance impact on RocksDB engine.

Here is a more detailed graph of the RocksDB results before and after the patches were applied:

Conclusion

In summary, the results of this benchmark show some negative performance impact of the Meltdown and Spectre V1 patches for the document insertion test cases we ran. The impact is notable for the MMFiles engine (varying performance impact between 0 and 17% across the test cases), but little to negligible for the RocksDB engine. We have started an internal investigation as to the root cause of these observed performance effects. Preliminary results seem to indicate that the mmfiles engine does a lot more system calls for R/W locks, which could explain the difference. We will investigate further.

Test Setup Details

In more detail, the arangobench test invocation parameters used were:

• --test-case -document
• --requests -500000
• --batch-size -0, -2, -4 or -128 (Number of operations to send per batch. Use 0 to disable batching
• --collection -test (Name of collection to use in test (only relevant for tests that invoke collections)
• --concurrency Number of parallel threads that will issue requests (default: 1)
• --async -true or -false
• --complexity 10: determines the number of attributes for the documents inserted