2017 Soledad Roadmap

LEAP BUDGET TIMELINE

• End of Feb we run out of money.

1+2: Nov 15 3: Dec 15 4+5: Jan 15

MOZILLA TIMELINE

• Start: August 17 - End April 17
• Deliverable 1 (Activity 1+2): 3mo Due Nov17
• Deliverable 2 (Activity 3): 3 months Due Jan 18
• Deliverable 3 (Activity 4+5): 2mo Due April 18

   DELIVERABLE 1 -- Nov2017

ACTIVITY 1. LIVE BENCHMARK INFRASTRUCTURE. Improve automation of benchmarks infrastructure. Collect output of current integrated benchmark runs and expose the historical data for easy comparison to assess immediately the impact of each incremental change.

• 1.1 Analyze options and decide on a data flow to gather, store, analyze and visualize performance data for soledad client and server. The metrics to collect include memory, cpu usage, sync time and reactor responsiveness.

  • cpu consumption for several operations (encryption and decryption of payloads, sync operations) #8844 (closed)
  • memory consumption #8904 (closed)
  • speed of io #8969 (closed)
  • responsiveness of application (responsiveness of the twisted reactor) #8908 (closed)
  • ensure statistical significance of every data point #8978 (closed)
  • ensure that the different runs cover real usage scenarios, regarding payload size #8884 (closed)
  • measure server scalability when used by concurrent clients #8906 (closed) #8979 (closed)

• 1.1.1. collect output of the benchmark runs and store it as a data point for a time series. #8974 (closed)

• 1.1.2. visualize the output of the different metrics across time and for different payloads. #8975 (closed)

• 1.1.3. expose the live output of benchmark information in a public place ("benchmarks.leap.se") that gets updated for every commit to soledad master. #8796 (closed)

ACTIVITY 2: MINIMAL VERSION FOR E2E BLOBS.

• 2.1. Complete and review the current encrypted blobs implementation

  • complete the minimal filesystem backend for encrypted blobs on the server side. #8815 (closed)
  • review the blobs API in soledad client. #8816 (closed)
  • finalize the proof of concept implementation. #8817 (closed)
  • integration with platform (update debian packages, ensure backups, monitoring etc). #8818 (closed)
  • implement functionality to handle saving of message drafts (as a special case). #8819 (closed) !124 (merged)

• 2.2. Implement API for incoming message pool in soledad server (needed by mx platform component)

  • temporary refactor moving writing of incoming messases out of mx. #8827 (closed), #8828 (closed)
  • implementation of incoming/ using the blobs filesystem backend. #8829 (closed)

• 2.3. Implement a minimally functional transfer retry strategy #8970 (closed) #8848 (closed)

  • unified download/upload manager (blobmanager)
  • Minimal crash recovery for the blobs network transfer manager #8932 (closed)
  • implement a retry strategy #8822 (closed)
  • ability to save transfer metadata locally #8823 (closed)
  • detect incomplete upload/download (depends on streaming) -> #8824 (closed) on !178 (merged)
  • properly notify a verification failure (should mark the downloaded blob as unusable) #8825 (closed)

   DELIVERABLE 2 -- Dec2017

ACTIVITY 3: FEATURE COMPLETION AND STABILITY.

Optimization of the Network Transfer and the Encrypted Blobs Backend on the Client Side.

2 months

• 3.1. Iterate over the blobs network transfer manager

  • bandwitdh throttling (#8931 (closed))
  • resumable downloads (#8989 (closed)) -> !175 (merged) (merged)
  • stability/robustness (#8959 (closed))

• 3.2. Optimization of the Encrypted Blobs Backend on the Client Side (sqlcipher based).

  • streaming of encryption and decryption (leveraging sqlite blob api) #8809 (closed) #8810 (closed) !166 (merged) and !172 (merged) - both merged
  • detect incomplete upload/download (moved from 2.3) (#8824 (closed)) on !178 (merged)
  • profile write and read speed for the current client sqlcipher blob backend
  • explore and compare performance of other backends (among other options, filesystem) (there are no other backends yet)

• 3.3. Implement on-demand blob transfer and priority queues / prioritizing certain types of content for the email use case (#8691 (closed))

  • ensure that key material, headers and attachments can be downloaded on demand with different priorities.
  • allow priority downloads to be completed before ongoing background download tasks are completed (drebs and shyba think that this is an important feature but it would be early optimization given the amount/complexity of code needed and the possibility of introducing new bugs, because of introduction of dependency between get/put and send/fetch_missing.)

   DELIVERABLE 3 -- Jan2018

ACTIVITY 4. ITERATE OVER THE FILESYSTEM BLOBS BACKEND ON THE SERVER SIDE

1 month

• 4.1. Final analysis of security considerations (analysis of the padding, sanitization of input, etc) (#9005) Address the most important security concerns
• 4.2. Benchmark filesystem backend and server (profile and identify bottlenecks). (#9006 (closed))
• 4.3. Create documentation for scalable backend for data storage (#9007 (closed))
• 4.4. Adapt filesystem backend and add support for another data storage backend according to specs (#9008)

ACTIVITY 5. IMPROVE SCALABILITY OF THE SOLEDAD SERVER

1 month

• 5.1. Enable multi-core usage for the Blobs Backend (#9018)
• 5.2. Rewrite of the metadata sync engine to use asynchronous couchdb libraries, phasing out the current wsgi-based service (if profiling of a simple prototype show that it is worthy, as some preliminar experiments seems to indicate). (#9019)

NOTES *Original Timeline

• Start: April 10th
• Activity 1+2: July 10th
• Activity 3: September 10
• Activity 4+5: November 10*