home shape
Key2Publish-logo

Impressive speed of ArangoDB-based Webshop & Product Information Management (PIM) System for 1 million products

right blob img min

Key2Publish helps organizations improve the way they create, collect, manage, enrich and publish product information.

by Johan Schulting, CEO and Founder of Key2Publish

The Challenge

Key2Publish was commissioned by ITK to develop a webshop and a Product Information Management (PIM) system. The products of the webshop are managed in and published from the PIM system. Furthermore, PDF-product sheets have to be generated through database publishing and product feed files exported from the PIM system.

Considering the large assortment of products, where each product contains another 100 attributes, allowing users fast and easy access to our product catalogue would be no small challenge.

ITK distinguishes the following product groups:

  • Pathology
  • Cell Biology
  • Flow Cytometry
  • Microbiology
  • Molecular Biology
  • Molecular Diagnostics
  • Consumables & Miscellaneous

ITK currently has an extensive range with around 200,000 products each with unique attributes. The intention is that this range will be expanded to 1,000,000 products within a few years. We needed a fast scalable solution to handle our projected growth and complex data requirements.

background img

Our Requirements

Beforehand we had a few demands:

  • Fast Querying/Searching
  • Flexible amount of columns
  • Needs to hold a lot of data without impacting speed
  • It needs to have some form of relational elements (graph)
  • JSON/REST API to communicate with the database
  • OpenSource
  • Available on all major platforms (Windows, Mac, Linux)

Why we chose ArangoDB

Before choosing ArangoDB we researched other database systems available on the market. When researching how PostgresSQL would perform with our dataset, we saw complicated JOIN queries would be needed, leading to slower query times. When comparing MongoDB to ArangoDB we noticed similar issues with query speed. Unlike ArangoDB’s graph features, with MongoDB relational queries became complicated to keep consistent, thus slowing down development. Finally we researched other multi-model database vendors and encountered too many unfixed bugs and a smaller, weaker support side in comparison to ArangoDB.

Our Solution

Speed is critical to our success. Fast querying and search capabilities are necessary attributes for our webshop and PIM system. ArangoDB was our best fit solution.

We decided to implement ArangoDB to store our large product catalogue. ArangoDB is an open-source multi-model NoSQL database with document and graph data store capabilities. ArangoDB’s native multi-model database system provided us the flexibility we needed by not limiting ourselves to just one type of database system. For our relational elements a graph model is the ideal solution, while for storing large volumes of data, document model is best. ArangoDB allows us to use both data models under the same roof with one AQL querying language.

Speed is critical to our success. Fast querying and search capabilities are necessary attributes for our webshop and PIM system. ArangoDB was our best fit solution.

We needed a system that can hold a lot of data without impacting speed. Both the webshop and PIM system must run trouble-free with over a million products. ArangoDB has the ability to handle data amounts exceeding 100GB and 300 million nodes. Furthermore, ArangoDB can deliver query-response-time of less than 20ms for time-critical recommendations. This combination of speed and delivery was the ideal solution for our needs.

right blob long

How the webshop works:

The webshop visitors can make use of an intelligent search and filter options. You can also try it yourself at https://labned.com/

At ITK, for example, this can look like this:

Starting with: 180.673 products/solutions.

image8

Entering a search text, the alternative “names” algorithm makes suggestions and in milliseconds the visitors find their way directly to the desired products. Auto-complete functionality allows suggestions to be presented concurrently with the user typing.

image10

Using ArangoDB, in just milliseconds the first results are presented:

image9

We already tested the new ArangoSearch in RC1 of ArangoDB 3.4 for our search needs and found it being flexible and even faster compared to the existing fulltext index. We are looking forward to the official production-ready release of ArangoDB 3.4.

Our Benefits

ArangoDB turned out to be the ideal database for this project. Despite the very large dataset, the webshop and the PIM system continue to work smoothly with impressive query response times using ArangoDB.

After researching other competitors, ArangoDB was the best solution providing the speed we needed when performing JOIN queries. ArangoDB is able to provide us with the flexibility necessary for our constantly changing data modeling requirements.

We especially enjoyed searching with the VIEW feature in v3.4 replacing FULLTEXT making querying even easier. These benefits paired with the JSON/REST API ability to communicate with the database made ArangoDB the perfect match.

Aside from speed and accuracy, ArangoDB provides support and documentation on par with no other. Not only is the open source documentation detailed allowing our team to get up and running quickly, but there is also an extensive and active community on channels such as StackOverflow and Github to assist with any questions or issues we came across.

In the future we are planning to implement more sophisticated sales reporting to the webshop leveraging the advanced calculating power of ArangoDB. Furthermore, we can utilize the graph model to gain a better understanding of our customers wants and preferences.

Thanks to the ArangoDB multi-model Database design, we are very flexible when it comes to expanding the data structures as our business continues grows to scale.

Big thanks to Johan Schulting for investing time to write this use case!

Also using ArangoDB? Write a few lines – post it to your blog or send it to us and we’ll publish it here.