|
|
# Soledad automated benchmarks
|
|
|
# benchmarks
|
|
|
|
|
|
This is a report about work done to address the [2017 roadmap plan](https://0xacab.org/leap/soledad/wikis/2017-roadmap).
|
|
|
|
... | ... | @@ -14,29 +14,31 @@ After reading this, you should be informed about: |
|
|
|
|
|
[[_TOC_]]
|
|
|
|
|
|
## Mail delivery and client-encrypted data storage and sync
|
|
|
## Client-encrypted data storage and sync
|
|
|
|
|
|
The LEAP project aims to develop server-side infrastructure and client-side applications for increasing the security of user's communications:
|
|
|
|
|
|
* for service providers, tools to deploy and maintain VPN and Email servers.
|
|
|
* for end users, a multi-platform application that works well with LEAP providers and does transparent key management (as much as possible), client-side encryption and data synchronization among user's devices.
|
|
|
* for end users, a multi-platform application that works well with LEAP providers and handles key management, client-side encryption and data synchronization among user's devices in a transparent way.
|
|
|
|
|
|
The component of the LEAP infrastructure that addresses client-side encryption and synchronization is called [Soledad](https://soledad.readthedocs.io/). Until the beginning of 2017, we have developed a solution in python that works for Linux and serves as a base implementation for other platforms.
|
|
|
The component of the LEAP infrastructure that addresses client-side encryption and synchronization is called [Soledad](https://soledad.readthedocs.io/). By the beginning of 2017, we had developed a solution in python that works for Linux and serves as a base implementation for other platforms.
|
|
|
|
|
|
## Challenges in data transfer and encryption pipeline
|
|
|
## Challenges in data transfer and client-side encryption pipeline
|
|
|
|
|
|
Many challenges arose during development. Some of them are performance issues related to the encryption and transfer pipeline both in server and client side, and had to be addressed in a careful way with the aid of some benchmarking / preformance measurement infrastructure:
|
|
|
As performance issues related to the encryption and transfer pipeline both in server and client side arose, we felt the need to analyse resource consumption in a careful way. In order to do that, we needed some benchmarking and performance measurement infrastructure:
|
|
|
|
|
|
* on the server side, there are memory issues with the use of couchdb as a data backend for client-encrypted data, suggesting the use of backpressure techniques.
|
|
|
* on the client-side, transfer and decryption / re-encryption of email data were causing both delivery delays and memory exhaustion until the sync and crypto machineries were analysed, re-evaluated and re-worked.
|
|
|
* on the server side, there were memory issues with the use of CouchDB as a data backend for client-encrypted data, suggesting the use of backpressure techniques.
|
|
|
* on the client-side, transfer and decryption / re-encryption of email data were experiencing both delivery delays and memory exhaustion.
|
|
|
|
|
|
## Benchmarking and resource consumption analysis
|
|
|
|
|
|
In the first semester of 2017, LEAP applied for a Mozilla grant with the aim to work specifically in creating a benchmarking infrastructure that could help fix existing problems and also help drive development decisions. The issues we wanted to address at that point were:
|
|
|
In the first semester of 2017, LEAP applied for a Mozilla grant with the aim to work specifically in creating a benchmarking infrastructure that could help fix existing problems and drive development decisions. The issues we wanted to address at that point were:
|
|
|
|
|
|
* Creation of a live benchmarking infrastructure, that is, a public website where we could get live performance data directly from recently committed source code.
|
|
|
* Modification of Soledad so it treats binary data as such, instead of storing and transferring payload data as JSON strings.
|
|
|
* Improvement of server-side scalability, addressing concurrency and memory issues.
|
|
|
* Gather information about resources consumed both in client and server (cpu, memory, and time).
|
|
|
* Evaluate responsiveness and scalability of the server and client applications.
|
|
|
* Create a live benchmarking infrastructure: a public website where we could get live performance data directly from recently committed source code.
|
|
|
* Work on one known bottleneck which was to modify Soledad to treat binary data as such, instead of storing and transferring payload data as strings.
|
|
|
* Improve server-side scalability, addressing concurrency and memory issues.
|
|
|
|
|
|
## Roadmap and current stage
|
|
|
|
... | ... | @@ -54,9 +56,9 @@ Next, we address each item individually to give an account of its current state |
|
|
|
|
|
In July 2017 LEAP launched the https://benchmarks.leap.se/ website, where benchmarking statistics can be seen for every set of commits pushed to Soledad's repository. We currently have time, memory and cpu consumption for each test of a special **benchmarking** set of tests. The solution is based on [elasticsearch](https://www.elastic.co/products/elasticsearch) to store results of benchmarks, [kibana](https://www.elastic.co/products/kibana) to plot graphs, and some custom scripts to glue everything together. Documentation on [how to add more tests to the benchmarking suite and website](http://soledad.readthedocs.io/en/latest/misc/benchmarks-website.html) is available.
|
|
|
|
|
|
As Soledad works with a reactor pattern, it is important to have feedback on possible reactor blocks and the overall reactor responsiveness. This is mostly done, but still not merged (#8908).
|
|
|
A small visual report was also added to the [LEAP dashboard](http://hare.leap.se:3030/default), so developers have a continuous feed of information regarding the state of the benchmarking. Comparison of results with the historic series is also being performed. It is possible to see outliers, get warned about them, and also fail the benchmarking pipeline in case a result is too much out of the expected.
|
|
|
|
|
|
On our todo list we still have measurement of speed o I/O, server scalability when used by concurrent clients, and adding some kind of warning or merge rejection when we introduce changes that cause performance to drop dramatically in some scenario.
|
|
|
As Soledad works with a reactor pattern, it is important to have feedback on possible reactor blocks and the overall reactor responsiveness. Server scalability is also of uttermost importance as the Soledad Server aims to serve for thousands of clients concurrently. We have added instrumentation to evaluate the responsiveness of the reactor and the scalability of the server during specific tasks. These are not plotted automatically because we didn't see the value of having those kinds of measurements repeated for every single test.
|
|
|
|
|
|
## BLOBs infrastructure
|
|
|
|
... | ... | @@ -88,9 +90,9 @@ As it can be seen, the BLOBs implementation (with all the performance features) |
|
|
|
|
|
## Server-side backend storage optimization.
|
|
|
|
|
|
On the server side, aside from authentication data caching as described in the last section, the main improvement so far has been to move away from the old WSGI-based implementation for the new BLOBs resource, by using twisted-based resources that can take full advantage of the asynchronous reactor. The old JSON documents synchronization code is still running using the WSGI machinery, but all of BLOBs has been implemented using Twisted's API.
|
|
|
On the server side, aside from authentication data caching as described in the last section, the main improvement so far has been to move away from the old WSGI-based implementation for the new BLOBs resource, by using twisted-based resources that can take full advantage of the asynchronous reactor and streamed transfer.
|
|
|
|
|
|
We should see more improvements once all the server is moved to Twisted async code, and when new endpoints are added to allow for batched upload/download of BLOBs.
|
|
|
The old JSON documents synchronization code is still running using the WSGI machinery, but all of BLOBs has been implemented using Twisted's API. We should see more improvements once all the server is completelly moved to Twisted async code, and when new endpoints are added to allow for streamed upload/download of BLOBs.
|
|
|
|
|
|
## Server scalability.
|
|
|
|
... | ... | |