ACTIVITY 1. LIVE BENCHMARK INFRASTRUCTURE.
Improve automation of benchmarks infrastructure. Collect output of current integrated benchmark runs and expose the historical data for easy comparison to assess immediately the impact of each incremental change.
1.1 Analyze options and decide on a data flow to gather, store, analyze and visualize performance data for soledad client and server.
The metrics to collect include memory, cpu usage, sync time and reactor responsiveness.
cpu consumption for several operations (encryption and decryption of payloads, sync operations) #8844
3.2. Optimization of the Encrypted Blobs Backend on the Client Side (sqlcipher based).
streaming of encryption and decryption (leveraging sqlite blob api) #8809#8810
detect incomplete upload/download (moved from 2.3) (#8824)
profile write and read speed for the current client sqlcipher blob backend
explore and compare performance of other backends (among other options, filesystem) (there are no other backends yet)
3.3. Implement on-demand blob transfer and priority queues / prioritizing certain types of content for the email use case (#8691)
ensure that key material, headers and attachments can be downloaded on demand with different priorities.
allow priority downloads to be completed before ongoing background download tasks are completed (drebs and shyba think that this is an important feature but it would be early optimization given the amount/complexity of code needed and the possibility of introducing new bugs, because of introduction of dependency between get/put and send/fetch_missing.)
DELIVERABLE 3 -- Jan2018
ACTIVITY 4. ITERATE OVER THE FILESYSTEM BLOBS BACKEND ON THE SERVER SIDE
Final analysis of security considerations (analysis of the padding, sanitization of input, etc) (#9005)
Address the most important security concerns
Benchmark filesystem backend and server (profile and identify bottlenecks). (#9006)
Create documentation for scalable backend for data storage (#9007)
Adapt filesystem backend and add support for another data storage backend according to specs (#9008)
ACTIVITY 5. IMPROVE SCALABILITY OF THE SOLEDAD SERVER
5.1. Enable multi-core usage for the Blobs Backend
5.2. Rewrite of the metadata sync engine to use asynchronous couchdb libraries, phasing out the current wsgi-based service (if profiling of a simple prototype show that it is worthy, as some preliminar experiments seems to indicate).