soledad issueshttps://0xacab.org/leap/soledad/-/issues2018-02-06T19:40:10Zhttps://0xacab.org/leap/soledad/-/issues/9032Create a roadmap for 20182018-02-06T19:40:10ZdrebsCreate a roadmap for 2018Things we know we want (in no specific order):
- Integration of blobs into bitmask.mail (on demand download of message bodies).
- Search index.
- Sync heuristics.
- Python 3 port.
- Improve standalone use, modularize auth, create standa...Things we know we want (in no specific order):
- Integration of blobs into bitmask.mail (on demand download of message bodies).
- Search index.
- Sync heuristics.
- Python 3 port.
- Improve standalone use, modularize auth, create standalone example app (password manager?).
- Improve Thunderbird integration.
- Research/implement mobile client app.
- Cleanup, define and apply code standards.
- Integration with the platform (https://0xacab.org/leap/platform/issues/8854).
- Add recovery code, eliminate shared db.
Here is a wiki page where we can come up with a roadmap for 2019: https://0xacab.org/leap/soledad/wikis/2018-roadmaphttps://0xacab.org/leap/soledad/-/issues/9031soledad server crashed2018-02-05T12:14:21Zkwadronautsoledad server crashedgot loads of test error mails from cdev, leap test told me this, I restarted soledad-server.
```
= [giraffe] FAIL: Soledad > Is Soledad running? [soledad.rb:13]:
= [giraffe] Service 'soledad-server' is not running:
= [giraffe] ● ...got loads of test error mails from cdev, leap test told me this, I restarted soledad-server.
```
= [giraffe] FAIL: Soledad > Is Soledad running? [soledad.rb:13]:
= [giraffe] Service 'soledad-server' is not running:
= [giraffe] ● soledad-server.service - Soledad Server
= [giraffe] Loaded: loaded (/lib/systemd/system/soledad-server.service; enabled)
= [giraffe] Active: inactive (dead) since Wed 2018-01-24 22:10:22 UTC; 1 weeks 3 days ago
= [giraffe] Process: 13203 ExecStart=/usr/bin/soledad-server (code=exited, status=0/SUCCESS)
= [giraffe] Main PID: 13203 (code=exited, status=0/SUCCESS)
= [giraffe] 12 tests: 8 passes, 0 skips, 0 warnings, 2 failures, 0 errors
```
In the logs: Jan 25 13:24:12 giraffe systemd[1]: Stopped Soledad Server.
I didn't investigate further, because I don't know the state of changes in those machines.https://0xacab.org/leap/soledad/-/issues/9030Improve the blobs server proxy performance2018-02-06T18:51:21ZdrebsImprove the blobs server proxy performanceIn #9029 we can see that the Twisted based proxy proposed in !180 introduces serious overhead. We have to better understand why and come up with an improved solution for now.In #9029 we can see that the Twisted based proxy proposed in !180 introduces serious overhead. We have to better understand why and come up with an improved solution for now.Cycle 10https://0xacab.org/leap/soledad/-/issues/9028Prototype a multiprocessing solution using a separate blobs server2018-01-23T18:35:56ZdrebsPrototype a multiprocessing solution using a separate blobs serverBecause we will not modify the platform now, we want an intermediate solution for the multiprocessing problem that will allow for usage with the current server and will also be usable when we migrate to platform-based routing.
This is t...Because we will not modify the platform now, we want an intermediate solution for the multiprocessing problem that will allow for usage with the current server and will also be usable when we migrate to platform-based routing.
This is the proposed structure for theis solution:
* business as usual, if not configured (everything runs in the same process).
* if soledad-blobs-detached-server configured, then it's responsibility of sysadmin to run `soledad-blobs-server`.
* if `/sync`, do as usual.
* if `/blobs` or `/incoming` or `/stream`, delegate request to local `soledad-blobs-server` port.
Obs: `soledad-blobs-server` is a new entrypoint, very lightweight (no weird dependencies, just Twisted). It is a router that spawns multi process and worries about task coordination.Cycle 10https://0xacab.org/leap/soledad/-/issues/9024Running tests with pypy fails with2017-12-27T17:49:41ZdrebsRunning tests with pypy fails withSteps to reproduce:
* Checkout current master (5edf23deefca71b632b2cdf2b686c478bc7dcce2)
* Set `envlist = py27, pypy` in `tox.ini`.
* Remove `basepython` from `[testenv]` in `tox.ini`.
* Run: `tox -e pypy`.
I get the following:
```
__...Steps to reproduce:
* Checkout current master (5edf23deefca71b632b2cdf2b686c478bc7dcce2)
* Set `envlist = py27, pypy` in `tox.ini`.
* Remove `basepython` from `[testenv]` in `tox.ini`.
* Run: `tox -e pypy`.
I get the following:
```
________________________________________ BlobManagerTestCase.test_get_doesnt_include_unavailable_blobs _________________________________________
NOTE: Incompatible Exception Representation, displaying natively:
FirstError: FirstError[#1, [Failure instance: Traceback: <class 'pysqlcipher.dbapi2.DatabaseError'>: file is encrypted or is not a database
/home/drebs/dev/leap/repos/soledad/.tox/pypy/site-packages/twisted/internet/defer.py:500:errback
/home/drebs/dev/leap/repos/soledad/.tox/pypy/site-packages/twisted/internet/defer.py:567:_startRunCallbacks
/home/drebs/dev/leap/repos/soledad/.tox/pypy/site-packages/twisted/internet/defer.py:653:_runCallbacks
/home/drebs/dev/leap/repos/soledad/.tox/pypy/site-packages/twisted/internet/defer.py:1442:gotResult
--- <exception caught here> ---
/home/drebs/dev/leap/repos/soledad/.tox/pypy/site-packages/twisted/internet/defer.py:1384:_inlineCallbacks
/home/drebs/dev/leap/repos/soledad/.tox/pypy/site-packages/twisted/python/failure.py:408:throwExceptionIntoGenerator
/home/drebs/dev/leap/repos/soledad/src/leap/soledad/client/_db/blobs/sql.py:165:update_sync_status
/home/drebs/dev/leap/repos/soledad/.tox/pypy/site-packages/twisted/python/threadpool.py:250:inContext
/home/drebs/dev/leap/repos/soledad/.tox/pypy/site-packages/twisted/python/threadpool.py:266:<lambda>
/home/drebs/dev/leap/repos/soledad/.tox/pypy/site-packages/twisted/python/context.py:122:callWithContext
/home/drebs/dev/leap/repos/soledad/.tox/pypy/site-packages/twisted/python/context.py:85:callWithContext
/home/drebs/dev/leap/repos/soledad/src/leap/soledad/client/_db/blobs/sql.py:365:_runInteraction
/home/drebs/dev/leap/repos/soledad/.tox/pypy/site-packages/twisted/enterprise/adbapi.py:464:_runInteraction
/home/drebs/dev/leap/repos/soledad/.tox/pypy/site-packages/twisted/enterprise/adbapi.py:36:__init__
/home/drebs/dev/leap/repos/soledad/.tox/pypy/site-packages/twisted/enterprise/adbapi.py:76:reconnect
/home/drebs/dev/leap/repos/soledad/.tox/pypy/site-packages/twisted/enterprise/adbapi.py:433:connect
/usr/lib/pypy/lib_pypy/_functools.py:45:__call__
/home/drebs/dev/leap/repos/soledad/src/leap/soledad/client/_db/pragmas.py:45:set_init_pragmas
/home/drebs/dev/leap/repos/soledad/src/leap/soledad/client/_db/pragmas.py:59:_set_init_pragmas
/home/drebs/dev/leap/repos/soledad/src/leap/soledad/client/_db/pragmas.py:368:set_write_ahead_logging
]]
```https://0xacab.org/leap/soledad/-/issues/9021Investigate pypy for multi-core Soledad Server2017-12-27T17:50:00ZdrebsInvestigate pypy for multi-core Soledad ServerCycle 10https://0xacab.org/leap/soledad/-/issues/9020Document ciphertext size estimation2017-12-26T10:49:32ZdrebsDocument ciphertext size estimationSoledad Backloghttps://0xacab.org/leap/soledad/-/issues/9019Rewrite of the metadata sync engine to use asynchronous couchdb libraries2018-01-27T19:56:23ZdrebsRewrite of the metadata sync engine to use asynchronous couchdb librariesCycle 10https://0xacab.org/leap/soledad/-/issues/9018Enable multi-core usage for the Blobs Backend2018-02-14T15:47:54ZdrebsEnable multi-core usage for the Blobs Backend- [ ] #9021 - Investigate pypy for multi-core Soledad Server
- [x] #9022 - Investigate possibilities of multiprocessing for Soledad Server
- [x] #9025 - Modify the blobs file system backend to account for concurrency
- [x] #9026 - Refact...- [ ] #9021 - Investigate pypy for multi-core Soledad Server
- [x] #9022 - Investigate possibilities of multiprocessing for Soledad Server
- [x] #9025 - Modify the blobs file system backend to account for concurrency
- [x] #9026 - Refactor Soledad Server entrypoint to start several processes
- [ ] #9028 - Prototype a multiprocessing solution using a separate blobs server
- [x] #9029 - Benchmark proxied soledad-blobs-serverCycle 10https://0xacab.org/leap/soledad/-/issues/9015CI packaging job fails with "ERROR: Job failed (system failure): aborted: ter...2017-12-18T11:49:23ZdrebsCI packaging job fails with "ERROR: Job failed (system failure): aborted: terminated"From: https://0xacab.org/drebs/soledad/-/jobs/29861
```
00:13:48 Unpacking linux-headers-4.13.0-1-common (4.13.13-1) over (4.13.4-2) ...
ERROR: Job failed (system failure): aborted: terminated
```From: https://0xacab.org/drebs/soledad/-/jobs/29861
```
00:13:48 Unpacking linux-headers-4.13.0-1-common (4.13.13-1) over (4.13.4-2) ...
ERROR: Job failed (system failure): aborted: terminated
```https://0xacab.org/leap/soledad/-/issues/9014Separate configuration of kibana/elasticsearch from benchmarks html website u...2017-12-22T17:20:16ZdrebsSeparate configuration of kibana/elasticsearch from benchmarks html website updateCurrently, there's one script configured in **moose.leap.se**'s cron for both configuring kibana/elasticsearch and updating the benchmarks html website. Because the process is lengthy and they are separate processes, they should be split...Currently, there's one script configured in **moose.leap.se**'s cron for both configuring kibana/elasticsearch and updating the benchmarks html website. Because the process is lengthy and they are separate processes, they should be split in 2 scripts and configured independently in cron (eg: configure kibana/elastic hourly and update the website every 5 minutes).Soledad Backloghttps://0xacab.org/leap/soledad/-/issues/9008Discuss scalable backends and priorities2017-12-22T17:15:07ZdrebsDiscuss scalable backends and prioritiesCycle 10https://0xacab.org/leap/soledad/-/issues/9005Discuss security of blobs and decide next steps2017-12-24T23:31:36ZdrebsDiscuss security of blobs and decide next stepsThis is about the 2 following items of Activity 4 of [the roadmap](https://0xacab.org/leap/soledad/wikis/2017-roadmap):
- Final analysis of security considerations (analysis of the padding, sanitization of input, etc)
- Address the most...This is about the 2 following items of Activity 4 of [the roadmap](https://0xacab.org/leap/soledad/wikis/2017-roadmap):
- Final analysis of security considerations (analysis of the padding, sanitization of input, etc)
- Address the most important security concerns
We have to send an email to the list to discuss and decide how to proceed.Cycle 10https://0xacab.org/leap/soledad/-/issues/9002Verify exceeded quota pipeline2017-12-22T17:20:35ZdrebsVerify exceeded quota pipelineI found the cdev's MX node stalled while trying to delivery mail to the e2e test user:
```
(...)
Nov 28 12:31:19 lynx leap-mx: [-] Waiting for the process of skipped mail to be done...
Nov 28 12:31:29 lynx leap-mx: [-] Waiting for the p...I found the cdev's MX node stalled while trying to delivery mail to the e2e test user:
```
(...)
Nov 28 12:31:19 lynx leap-mx: [-] Waiting for the process of skipped mail to be done...
Nov 28 12:31:29 lynx leap-mx: [-] Waiting for the process of skipped mail to be done...
Nov 28 12:31:29 lynx leap-mx: [-] Waiting for the process of skipped mail to be done...
```
Looking at Soledad Server, it seems that it was also stalling while trying to reply new blob delivery attempts using the Incoming API. After restarting soledad-server, i saw many delivery attempts and many quota-exceeded messages like this:
```
Nov 28 12:42:00 giraffe soledad-server: [leap.soledad.server._blobs#error] Error 507: Quota exceeded for user: a12ae9727e9ed3511e6464128c72aaa6
Nov 28 12:42:00 giraffe soledad-server: [-] 0.0.0.0 - - [28/Nov/2017:12:41:59 +0000] "PUT /incoming/a12ae9727e9ed3511e6464128c72aaa6/8deb3487-3828-4f4c-b792-3f673aedd249 HTTP/1.1" 507 32 "-" "-"
```
It felt like as if some endpoint was not releasing the connection correctly (don't know if on MX or Soledad Server side), but I am not sure if that is the problem.
We should:
- [ ] ensure that the delivered blobs to e2e are removed after the test runs.
- [ ] double check that the quota-exceeded responses are returning and releasing resources correctly (both in MX and Soledad Server sides).Soledad Backloghttps://0xacab.org/leap/soledad/-/issues/8998Review and standardize Soledad Server environment variables and configuration...2017-12-04T18:57:12ZdrebsReview and standardize Soledad Server environment variables and configuration optionsDifferent environment variables were added during the history of development of Soledad Server that affect the server in different ways. They are all currently documented in the `soledad-server` command man page and in the online documen...Different environment variables were added during the history of development of Soledad Server that affect the server in different ways. They are all currently documented in the `soledad-server` command man page and in the online documentation. Despite that, some variables are prefixed with `SOLEDAD_`, some with `SOLEDAD_SERVER` and some with nothing (i.e. `HTTPS_PORT`). Also, some of them replace specific configuration options that are available in the config file (i.e. `SOLEDAD_COUCH_URL`.
We have to review and standardize the environment variables naming, and maybe it's precedence over configuration options in the config file.https://0xacab.org/leap/soledad/-/issues/8994Remove pinning of elasticsearch-py when a new release fixes the TLS cert bug2017-12-04T18:46:35ZdrebsRemove pinning of elasticsearch-py when a new release fixes the TLS cert bugVersion 0.6.0 of elasticsearch-py [introduced a bug in our CI benchmarks pipeline](https://0xacab.org/leap/soledad/issues/8993). This issue is meant to remember us to remove that pinning when elasticsearch-py has a release that includes ...Version 0.6.0 of elasticsearch-py [introduced a bug in our CI benchmarks pipeline](https://0xacab.org/leap/soledad/issues/8993). This issue is meant to remember us to remove that pinning when elasticsearch-py has a release that includes the fix for the bug ([commit](https://github.com/elastic/elasticsearch-py/commit/f639f9b28e1d4692d73d39b3952e31d84333b0dc), [issue](https://github.com/elastic/elasticsearch-py/issues/669)).Soledad Backloghttps://0xacab.org/leap/soledad/-/issues/8990Add HTML documentation to debian package.2017-11-30T14:39:36ZdrebsAdd HTML documentation to debian package.https://0xacab.org/leap/soledad/-/issues/8985Automate responsiveness and scalability tests2017-12-04T18:53:31ZdrebsAutomate responsiveness and scalability testsThe machinery for [responsiveness](https://0xacab.org/leap/soledad/tree/master/tests/responsiveness) and [scalability](https://0xacab.org/leap/soledad/tree/master/scripts/scalability) tests are in place, but those are not being run and g...The machinery for [responsiveness](https://0xacab.org/leap/soledad/tree/master/tests/responsiveness) and [scalability](https://0xacab.org/leap/soledad/tree/master/scripts/scalability) tests are in place, but those are not being run and graphed automatically yet.
We have to define the priority for this work given the [current roadmap](https://0xacab.org/leap/soledad/wikis/2017-roadmap) and decide if and when to proceed with the automation of responsiveness and scalability tests.Soledad Backloghttps://0xacab.org/leap/soledad/-/issues/8984Enforce per-user concurrency limit on server side.2017-12-04T18:20:34ZVictorEnforce per-user concurrency limit on server side.Today we use a simple semaphore on server side to control the maximum number of file writes. It would be nice to deny requests over X simultaneous connections per uuid. This would help making sure each client gets a fair share of concurr...Today we use a simple semaphore on server side to control the maximum number of file writes. It would be nice to deny requests over X simultaneous connections per uuid. This would help making sure each client gets a fair share of concurrency under this global limit and prevent DDoS from misconfigured clients.
If done, we need to make sure that client can handle `HTTP 429 - (Too many requests)` response during retry, lowering it's own concurrency limit to match the server. Also expose the server limit on GET / to initialize properly, as we do with batching on traditional doc sync.Soledad Backloghttps://0xacab.org/leap/soledad/-/issues/8983Dashboard doesn't recognize failing master build2017-12-04T18:59:58ZVaracDashboard doesn't recognize failing master buildThe dashboard checks the last run pipeline for success. Now that we have scheduled benchmark jobs that run alone by themselves (see https://0xacab.org/leap/soledad/pipelines), the latest pipeline will most always succeed.
We need to che...The dashboard checks the last run pipeline for success. Now that we have scheduled benchmark jobs that run alone by themselves (see https://0xacab.org/leap/soledad/pipelines), the latest pipeline will most always succeed.
We need to check the last *unscheduled* pipeline results.
Atm, the last *unscheduled* pipeline [failed](https://0xacab.org/leap/soledad/pipelines/6070)!