This is an old version of this page. You can view the most recent version or browse the history.

benchmarks report

Soledad automated benchmarks

This is a report about work done to address the 2017 roadmap plan.

This document is still under construction.

After reading this, you should be informed about:

The general purpose of Soledad, LEAP's client-encrypted synchronization solution.
Work done previous to the roadmap plan.
The current status of the roadmap plan.

Soledad automated benchmarks

Mail delivery and client-encrypted data storage and sync

The LEAP project aims to develop server-side infrastructure and client-side applications for increasing the security of user's communications:

for service providers, tools to deploy and maintain VPN and Email servers.
for end users, a multi-platform application that works well with LEAP providers and does transparent key management (as much as possible), client-side encryption and data synchronization among user's devices.

The component of the LEAP infrastructure that addresses client-side encryption and synchronization is called Soledad. Until the beginning of 2017, we have developed a solution in python that works for Linux and serves as a base implementation for other platforms.

Challenges in data transfer and encryption pipeline

Many challenges arose during development. Some of them are performance issues related to the encryption and transfer pipeline both in server and client side, and had to be addressed in a careful way with the aid of some benchmarking / preformance measurement infrastructure:

on the server side, there are memory issues with the use of couchdb as a data backend for client-encrypted data, suggesting the use of backpressure techniques.
on the client-side, transfer and decryption / re-encryption of email data were causing both delivery delays and memory exhaustion until the sync and crypto machineries were analysed, re-evaluated and re-worked.

Benchmarking and resource consumption analysis

In the first semester of 2017, LEAP applied for a Mozilla grant with the aim to work specifically in creating a benchmarking infrastructure that could help fix existing problems and also help drive development decisions. The issues we wanted to address at that point were:

Creation of a live benchmarking infrastructure, that is, a public website where we could get live performance data directly from recently committed source code.
Modification of Soledad so it treats binary data as such, instead of storing and transferring payload data as JSON strings.
Improvement of server-side scalability, addressing concurrency and memory issues.

Roadmap and current stage

The proposed roadmap was split in 5 activities:

Creation of a live benchmark infrastructure to expose resource consumption statistics directly from the development repository tree.
A minimal treatment of BLOBs/payloads as such instead of encoding them as strings as was previously done.
Client-side transfer/crypto pipeline optimization.
Server-side backend storage optimization.
Server scalability.

Next, we address each item individually to give an account of its current state of development.

Live benchmark statistics website

In July 2017 LEAP launched the https://benchmarks.leap.se/ website, where benchmarking statistics can be seen for every set of commits pushed to Soledad's repository. We currently have time, memory and cpu consumption for each test of a special benchmarking set of tests. The solution is based on elasticsearch to store results of benchmarks, kibana to plot graphs, and some custom scripts to glue everything together. Documentation on how to add more tests to the benchmarking suite and website is available.

As Soledad works with a reactor pattern, it is important to have feedback on possible reactor blocks and the overall reactor responsiveness. This is mostly done, but still not merged (#8908 (closed)).

On our todo list we still have measurement of speed o I/O, server scalability when used by concurrent clients, and adding some kind of warning or merge rejection when we introduce changes that cause performance to drop dramatically in some scenario.

BLOBs infrastructure

The pristine version of Soledad used to store all data as JSON documents, which means that binary data was also being treated as strings. This has many drawbacks, as increased memory usage and difficulties to do transfer and crypto in a proper binary pipeline.

Starting with 0.10.0, Soledad now has a proper BLOB infrastructure that decouples payloads from metadata both in storage and in the synchronization process. The first use case was mail delivery, and we implemented an API so the MX component now hands Soledad Server the responsibility of packing and storing incoming messages as public-key-encrypted data. Metadata is synchronized using the usual sync mechanism, while BLOBs are synchronized in parallel and on-demand. This makes it feasible to have much quicker feedback of incoming messages and load their content and attachments on demand, as is done in most implementations of IMAP clients.

Client-side transfer/crypto pipeline optimization.

The following is a graph comparing the ratios of Inbox loading time having legacy JSON-based sync as baseline and blobs implementations with different features:

The incoming mail pipeline considered on those tests consists of the following:

receiving incoming messages from server.
decrypting them.
creating new local metadata documents and local content documents or blobs.
syncing everything back to server.

The implementations compared in the figure above are:

Legacy: uses the JSON-based synchronization to transfer all messages.
Blobs: uses the new BLOBs implemenation to transfer content of messages. Legacy sync is still used to transfer metadata.
Blobs + session cache: same as above, but also caches session in server-side, thus avoiding having to query couch for user authentication data on every request.
Blobs + session cache + persistent http: same as above, but also reuses HTTP connections on client side, thus avoiding having to open a new TLS/TCP connection for every request.

GitLab