|
|
**LEAP BUDGET TIMELINE**
|
|
|
* End of Feb we run out of money.
|
|
|
|
|
|
1+2: Nov 15
|
|
|
3: Dec 15
|
|
|
4+5: Jan 15
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**MOZILLA TIMELINE**
|
|
|
* Start: August 17 - End April 17
|
|
|
* Deliverable 1 (Activity 1+2): 3mo Due Nov17
|
|
|
* Deliverable 2 (Activity 3): 3 months Due Jan 18
|
|
|
* Deliverable 3 (Activity 4+5): 2mo Due April 18
|
|
|
|
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
DELIVERABLE 1 -- Nov2017
|
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
|
|
**ACTIVITY 1. LIVE BENCHMARK INFRASTRUCTURE.**
|
|
|
Improve automation of benchmarks infrastructure. Collect output of current integrated benchmark runs and expose the historical data for easy comparison to assess immediately the impact of each incremental change.
|
|
|
|
|
|
- [x] 1.1 Analyze options and decide on a data flow to gather, store, analyze and visualize performance data for soledad client and server.
|
|
|
The metrics to collect include memory, cpu usage, sync time and reactor responsiveness.
|
|
|
|
|
|
* [x] cpu consumption for several operations (encryption and decryption of payloads, sync operations) #8844
|
|
|
* [x] memory consumption #8904
|
|
|
* [x] speed of io #8969
|
|
|
* [x] responsiveness of application (responsiveness of the twisted reactor) https://0xacab.org/leap/soledad/issues/8908
|
|
|
* [x] ensure statistical significance of every data point https://0xacab.org/leap/soledad/issues/8978
|
|
|
* [x] ensure that the different runs cover real usage scenarios, regarding payload size #8884
|
|
|
* [x] measure server scalability when used by concurrent clients #8906 #8979
|
|
|
|
|
|
- [x] 1.1.1. collect output of the benchmark runs and store it as a data point for a time series. #8974
|
|
|
|
|
|
- [x] 1.1.2. visualize the output of the different metrics across time and for different payloads. #8975
|
|
|
|
|
|
- [x] 1.1.3. expose the live output of benchmark information in a public place ("benchmarks.leap.se") that gets updated for every commit to soledad master. #8796
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**ACTIVITY 2: MINIMAL VERSION FOR E2E BLOBS.**
|
|
|
|
|
|
- [x] 2.1. Complete and review the current encrypted blobs implementation
|
|
|
- [x] complete the minimal filesystem backend for encrypted blobs on the server side. #8815
|
|
|
- [x] review the blobs API in soledad client. #8816
|
|
|
- [x] finalize the proof of concept implementation. #8817
|
|
|
- [x] integration with platform (update debian packages, ensure backups, monitoring etc). #8818
|
|
|
- [x] implement functionality to handle saving of message drafts (as a special case). #8819 !124
|
|
|
|
|
|
- [x] 2.2. Implement API for incoming message pool in soledad server (needed by mx platform component)
|
|
|
- [x] temporary refactor moving writing of incoming messases out of mx. #8827, #8828
|
|
|
- [x] implementation of incoming/ using the blobs filesystem backend. #8829
|
|
|
|
|
|
- [x] 2.3. Implement a minimally functional transfer retry strategy #8970 #8848
|
|
|
* [x] unified download/upload manager (blobmanager)
|
|
|
* [x] Minimal crash recovery for the blobs network transfer manager #8932
|
|
|
* [x] implement a retry strategy #8822
|
|
|
* [x] ability to save transfer metadata locally #8823
|
|
|
* [ ] ~~detect incomplete upload/download~~ (depends on streaming) -> #8824 on !178
|
|
|
* [x] properly notify a verification failure (should mark the downloaded blob as unusable) #8825
|
|
|
|
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
DELIVERABLE 2 -- Dec2017
|
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
|
|
**ACTIVITY 3: FEATURE COMPLETION AND STABILITY.**
|
|
|
|
|
|
Optimization of the Network Transfer and the Encrypted Blobs Backend on the Client Side.
|
|
|
|
|
|
2 months
|
|
|
|
|
|
- [x] 3.1. Iterate over the blobs network transfer manager
|
|
|
* [x] bandwitdh throttling (#8931)
|
|
|
* [x] resumable downloads (#8989) -> !175 (merged)
|
|
|
* [x] stability/robustness (#8959)
|
|
|
|
|
|
- [x] 3.2. Optimization of the Encrypted Blobs Backend on the Client Side (sqlcipher based).
|
|
|
* [x] streaming of encryption and decryption (leveraging sqlite blob api) #8809 #8810 !166 and !172 - both merged
|
|
|
* [x] detect incomplete upload/download (moved from 2.3) (#8824) on !178
|
|
|
* [x] profile write and read speed for the current client sqlcipher blob backend
|
|
|
* [ ] ~~explore and compare performance of other backends (among other options, filesystem)~~ (there are no other backends yet)
|
|
|
|
|
|
- [x] 3.3. Implement on-demand blob transfer and priority queues / prioritizing certain types of content for the email use case (#8691)
|
|
|
* [x] ensure that key material, headers and attachments can be downloaded on demand with different priorities.
|
|
|
* [ ] ~~allow priority downloads to be completed before ongoing background download tasks are completed~~ (drebs and shyba think that this is an important feature but it would be early optimization given the amount/complexity of code needed and the possibility of introducing new bugs, because of introduction of dependency between get/put and send/fetch_missing.)
|
|
|
|
|
|
|
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
DELIVERABLE 3 -- Jan2018
|
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
|
|
**ACTIVITY 4. ITERATE OVER THE FILESYSTEM BLOBS BACKEND ON THE SERVER SIDE**
|
|
|
|
|
|
1 month
|
|
|
|
|
|
- [ ] 4.1. Final analysis of security considerations (analysis of the padding, sanitization of input, etc) (#9005) Address the most important security concerns
|
|
|
- [x] 4.2. Benchmark filesystem backend and server (profile and identify bottlenecks). (#9006)
|
|
|
- [x] 4.3. Create documentation for scalable backend for data storage (#9007)
|
|
|
- [ ] 4.4. Adapt filesystem backend and add support for another data storage backend according to specs (#9008)
|
|
|
|
|
|
**ACTIVITY 5. IMPROVE SCALABILITY OF THE SOLEDAD SERVER**
|
|
|
|
|
|
1 month
|
|
|
|
|
|
- [ ] 5.1. Enable multi-core usage for the Blobs Backend (#9018)
|
|
|
- [ ] 5.2. Rewrite of the metadata sync engine to use asynchronous couchdb libraries, phasing out the current wsgi-based service (if profiling of a simple prototype show that it is worthy, as some preliminar experiments seems to indicate). (#9019)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**NOTES**
|
|
|
*Original Timeline
|
|
|
* Start: April 10th
|
|
|
* Activity 1+2: July 10th
|
|
|
* Activity 3: September 10
|
|
|
* Activity 4+5: November 10* |
|
|
\ No newline at end of file |