This is meant to be a report to address the proposal in https://0xacab.org/leap/soledad/wikis/2017-roadmap
This is still under construction.
Poposal of topics to address:
- General purpose of soledad.
- Work done previous to the roadmap.
- For each proposed activity, what has been changed during development and what is the current status.
Mail delivery and client-encrypted data storage and sync
The LEAP project aims to develop server-side infrastructure and client-side applications for increasing the security of user's communications:
- for service providers, tools to deploy and maintain VPN and Email servers.
- for end users, a multi-platform application that works well with LEAP providers and does transparent key management (as much as possible), client-side encryption and data synchronization among user's devices.
The component of the LEAP infrastructure that addresses client-side encryption and synchronization is called Soledad. Until the beginning of 2017, we have developed a solution in python that works for Linux and serves as a base implementation for other platforms.
Challenges in data transfer and encryption pipeline
Many challenges arose during development. Some of them are performance issues related to the encryption and transfer pipeline both in server and client side, and had to be addressed in a careful way with the aid of some benchmarking / preformance measurement infrastructure:
- on the server side, there are memory issues with the use of couchdb as a data backend for client-encrypted data, suggesting the use of backpressure techniques.
- on the client-side, transfer and decryption / re-encryption of email data were causing both delivery delays and memory exhaustion until the sync and crypto machineries were analysed, re-evaluated and re-worked.
Benchmarking and resource consumption analysis
In the first semester of 2017, LEAP applied for a Mozilla grant with the aim to work specifically in creating a benchmarking infrastructure that could help fix existing problems and also help drive development decisions. The issues we wanted to address at that point were:
- Creation of a live benchmarking infrastructure, that is, a public website where we could get live performance data directly from recently committed source code.
- Modification of Soledad so it treats binary data as such, instead of storing and transferring payload data as JSON strings.
- Improvement of server-side scalability, addressing concurrency and memory issues.