soledad issueshttps://0xacab.org/leap/soledad/-/issues2017-12-04T18:49:06Zhttps://0xacab.org/leap/soledad/-/issues/8789Decide if soledad client needs to know about network connectivity / server re...2017-12-04T18:49:06ZdrebsDecide if soledad client needs to know about network connectivity / server reachabilityhttps://0xacab.org/leap/soledad/-/issues/8785Document secret bootstrap and exceptions in case of first run for a uuid.2017-12-04T18:49:06ZdrebsDocument secret bootstrap and exceptions in case of first run for a uuid.# Definition of done
* [ ] documentation page for secrets mechanism
* [ ] some level of behavior test for ensuring the mechanism works as expected
# Details
When soledad is instantiated for the first time for a given user in a device...# Definition of done
* [ ] documentation page for secrets mechanism
* [ ] some level of behavior test for ensuring the mechanism works as expected
# Details
When soledad is instantiated for the first time for a given user in a device, the following happens:
* it will not find secrets stored locally.
* it has then to determine whether there are already secrets stored for that user in the server.
* it has to refuse to start if it can't make that verification (otherwise, there's the risk of creating new local secrets and failing to decrypt data previously encrypted with another secret).
There are some cases in which soledad will not be able to determine if there's a secret in the server:
* if it can't reach the server (network error, connectivity problem, server is offline, etc).
* if it can reach the server but it doesn't have a token to authorize the request (client hasn't logged in yet, token has expired, etc).
We have to properly document this behaviour and ensure that Bitmask client works accordingly.
Related: #8721https://0xacab.org/leap/soledad/-/issues/8784Create e2e tests that instantiates soledad with all parameters, bootstraps, a...2017-12-04T18:49:06ZdrebsCreate e2e tests that instantiates soledad with all parameters, bootstraps, and don't failScenarios:
* secrets in local storage
* secrets in remote storage
* no secrets (first run)Scenarios:
* secrets in local storage
* secrets in remote storage
* no secrets (first run)https://0xacab.org/leap/soledad/-/issues/8781review interface for client secrets api2017-12-04T18:49:06ZKali Kanekoreview interface for client secrets apias part of b433c1ed736f5d4c, soledad is passed as a parameter for the client secrets api.
in the review I objected that passing the whole soledad object to the initialization seems like a bad design to me, since it couples the secret api...as part of b433c1ed736f5d4c, soledad is passed as a parameter for the client secrets api.
in the review I objected that passing the whole soledad object to the initialization seems like a bad design to me, since it couples the secret api to an object with too much responsibility.
I'd like to examine a couple of options:
- using a zope.proxy for initializing/updating the token/credentials instead, that would solve the problem of accessing the token instead.
- if that's not desirable for some reason, the secrets/storage objects should not depend on a soledad instance, but we could try to limit what interface those objects expect (making soledad implement those interfaces). this should make the contract more explicit and allow for better testability.https://0xacab.org/leap/soledad/-/issues/8759round up the blob size calculation2017-12-04T18:20:31ZKali Kanekoround up the blob size calculationwe should round up the record for the blob size, in order to minimize information leaking.
we can start with a simplistic ceiling-padding and substitute in the future for a more sophisticated algorithm
(note however that it might be good...we should round up the record for the blob size, in order to minimize information leaking.
we can start with a simplistic ceiling-padding and substitute in the future for a more sophisticated algorithm
(note however that it might be good to increase the block size with the blob size)
https://www.freehaven.net/anonbib/cache/traffic-padding-pets12.pdf
Remember to:
- [ ] update documentationSoledad Backloghttps://0xacab.org/leap/soledad/-/issues/4799Quota Support2017-12-04T18:49:06ZVaracQuota SupportNot urgent, but we need to start discussing it.... is there a plan for that on the soledad roadmap? There hasn't been any work on that in soledad yet about this.
webapp: set / change quota
leap_mx: query quota + bounce or deliver (start...Not urgent, but we need to start discussing it.... is there a plan for that on the soledad roadmap? There hasn't been any work on that in soledad yet about this.
webapp: set / change quota
leap_mx: query quota + bounce or deliver (start with couchdb disk_size)
imap daemon: query quota + rejct or save msg, tell mail client about disk usage
...
maybe there is a couch database quota?
"CouchDB does not support quotas natively, so you may need something custom.
You can enforce it yourself, either with a tiny fork of CouchDB or with your related hosting software, and use the usage information returned by CouchDB. Since version 1.2, CouchDB indicates not only the disk usage, but "data" size, not counting metadata and old data."
from http://stackoverflow.com/questions/10360145/multi-hosting-quota-on-couchdb
*(from redmine: created on 2013-12-18)*https://0xacab.org/leap/soledad/-/issues/7430signal invalidAuthTokenError2017-12-04T18:49:05ZKali Kanekosignal invalidAuthTokenErrorwhen you leave bitmask syncing for some time (+1h of inactivity?), the token used in soledad expires, and one gets the following error on the sync attempts:
<pre>
[2015-09-08 16:37:11] DEBUG - L#176 : leap.mail.incoming.service:fetc...when you leave bitmask syncing for some time (+1h of inactivity?), the token used in soledad expires, and one gets the following error on the sync attempts:
<pre>
[2015-09-08 16:37:11] DEBUG - L#176 : leap.mail.incoming.service:fetch - fetching mail for: 4bca6c5c82998cc1a3424c6cac66b75c test_docker_002@cdev.bitmask.net
[2015-09-08 16:37:11] INFO - L#109 : twisted.logger._stdlib:__call__ - FETCH: syncing soledad...
[2015-09-08 16:37:12] ERROR - L#717 : leap.soledad.client.api:_sync_errback - Soledad exception when syncing!
*--- Failure #5257 ---
Failure: leap.soledad.common.errors.InvalidAuthTokenError:
*--- End of Failure #5257 ---
Unhandled Error
Traceback (most recent call last):
Failure: leap.soledad.common.errors.InvalidAuthTokenError:
[2015-09-08 16:37:12] CRITICAL - L#109 : twisted.logger._stdlib:__call__ - Unhandled Error
Traceback (most recent call last):
Failure: leap.soledad.common.errors.InvalidAuthTokenError:
[2015-09-08 16:38:11] DEBUG - L#176 : leap.mail.incoming.service:fetch - fetching mail for: 4bca6c5c82998cc1a3424c6cac66b75c test_docker_002@cdev.bitmask.net
[2015-09-08 16:38:11] INFO - L#109 : twisted.logger._stdlib:__call__ - FETCH: syncing soledad...
[2015-09-08 16:38:12] ERROR - L#717 : leap.soledad.client.api:_sync_errback - Soledad exception when syncing!
*--- Failure #5263 ---
Failure: leap.soledad.common.errors.InvalidAuthTokenError:
*--- End of Failure #5263 ---
Unhandled Error
Traceback (most recent call last):
Failure: leap.soledad.common.errors.InvalidAuthTokenError:
[2015-09-08 16:38:12] CRITICAL - L#109 : twisted.logger._stdlib:__call__ - Unhandled Error
Traceback (most recent call last):
Failure: leap.soledad.common.errors.InvalidAuthTokenError:
</pre>
In the future (probably soledad 0.8 release), we'd like to have some integration between soledad and bonafide so that the session object can get a new token and make it available to the soledad object in a simple way. This probably won't happen before the bitmask 0.10 release cycle, so I think we could make a way to support refreshing of tokens in the bitmask 0.9.x and soledad 0.7.x branches.
What we could do for this interim phase is this:
1. signal invalidtokenauth
2. get the client (ugh) capture this signal
3. get a new token (probably by logging out and initiating a new srp-auth)
4. pass the new token to the soledad proxy object.
if this strategy is acceptable, we should implement points 1 and 4 in soledad client.
*(from redmine: created on 2015-09-08, closed on 2015-11-03)*
* Relations:
* relates #6776https://0xacab.org/leap/soledad/-/issues/7473add property to let client indicate proper initialization2017-12-04T18:49:05ZKali Kanekoadd property to let client indicate proper initializationIn the original context, it was suggested that the soledad.client should expose a simple property to let user know if it has been properly initialized or not (mainly, if shared secrets have been generated AND uploaded to the server). (ke...In the original context, it was suggested that the soledad.client should expose a simple property to let user know if it has been properly initialized or not (mainly, if shared secrets have been generated AND uploaded to the server). (keymanager probably should do something similar)
this would solve some issues like trying to generate key material again for an already-initialized account.
*(from redmine: created on 2015-09-18)*
* Relations:
* relates #7470
During the blobs iteration, I thought that we could use a generic initialization document to store the blob initialization.https://0xacab.org/leap/soledad/-/issues/7660move sync loop inside soledad2017-12-04T18:14:08ZKali Kanekomove sync loop inside soledadCurrently, the Incoming service is the only one using it.
It las a LoopingCall that syncs soledad.
We need a way of adding callbacks to the result.
*(from redmine: created on 2015-12-01)*Currently, the Incoming service is the only one using it.
It las a LoopingCall that syncs soledad.
We need a way of adding callbacks to the result.
*(from redmine: created on 2015-12-01)*https://0xacab.org/leap/soledad/-/issues/7665improve couchdb tests2017-12-04T18:49:05ZKali Kanekoimprove couchdb tests- Improve couchdb isolation
- Add Mocks on some tests to improve speed in running the testsuite
- separate unit/integration tests.
*(from redmine: created on 2015-12-01)*
* Relations:
* precedes #7664- Improve couchdb isolation
- Add Mocks on some tests to improve speed in running the testsuite
- separate unit/integration tests.
*(from redmine: created on 2015-12-01)*
* Relations:
* precedes #7664https://0xacab.org/leap/soledad/-/issues/7666client: batch put to sqlcipher2017-12-04T18:41:12ZKali Kanekoclient: batch put to sqlcipherneed to be able to insert multiple documents at once
*(from redmine: created on 2015-12-01)*need to be able to insert multiple documents at once
*(from redmine: created on 2015-12-01)*Soledad Backloghttps://0xacab.org/leap/soledad/-/issues/7668sensible defaults for conflict handling2017-12-04T18:49:05ZKali Kanekosensible defaults for conflict handlingIdeally, syncs should never raise error in conflicts unless there is no way to solve it.
This might involve the design of type-specific heuristics to solve conflicts on a SoledadDocument type basis, client side.
*(from redmine: created ...Ideally, syncs should never raise error in conflicts unless there is no way to solve it.
This might involve the design of type-specific heuristics to solve conflicts on a SoledadDocument type basis, client side.
*(from redmine: created on 2015-12-01)*https://0xacab.org/leap/soledad/-/issues/8038Investigate Soledad errors stored in nagios2017-12-04T18:49:05ZVaracInvestigate Soledad errors stored in nagiosToday i acknoleged the webapp log nagios warnings for this host:
- alpaca.mail
- aardwolf.unstable
check_mk logwatch gathered webapp logs in this spoolfiles:
<pre>
root@donkey:/var/lib/check_mk/logwatch# find |grep soledad
./aardwolf...Today i acknoleged the webapp log nagios warnings for this host:
- alpaca.mail
- aardwolf.unstable
check_mk logwatch gathered webapp logs in this spoolfiles:
<pre>
root@donkey:/var/lib/check_mk/logwatch# find |grep soledad
./aardwolf.unstable.bitmask.i/\var\log\soledad.log
./alpaca.mail.bitmask.i/\var\log\soledad.log
</pre>
Todo:
- acknoledge the nagios problem so others don't get bugged by it and know that you are taking care
- go through the spoolfiles, and create issues for those errors
- temporarily ignore these errors in leap_platform/puppet/modules/site_check_mk/files/agent/logwatch/soledad.cfg so they don't pile up again in the spoolfiles
- remove the spoolfiles
- repeat :D
Here's a very short doc about log monitoring: https://leap.se/en/docs/platform/troubleshooting/tests#log-monitoring (happy to recieve improvements).
*(from redmine: created on 2016-04-14)*https://0xacab.org/leap/soledad/-/issues/8685The id of documents may leak information2017-12-04T18:49:05ZdrebsThe id of documents may leak informationSoledad is a document store, and its documents have 3 fields: *id*, *revision*, and *content*. The fields *id* and *revision* are used by soledad sync protocol (both on client and server side) to compare documents and decide if versions ...Soledad is a document store, and its documents have 3 fields: *id*, *revision*, and *content*. The fields *id* and *revision* are used by soledad sync protocol (both on client and server side) to compare documents and decide if versions superseed and if there are conflicts.
The *content* field is client-encrypted before being sent from to the server. Currently, the *id* and the *revision* are sent unmodified.
Problem: if users of the soledad api decide to set the *id* to arbitrary values, they may be leaking information to the server about the *content* of the document. One current example bitmask's mail implementation: it sets the id of a document based on the type of its content (if it represents a header, message flags, content, etc). With knowledge of the document ids, the server is able to learn if an attachment is the hash of a known mime-part, for example.
There are 2 possible approaches to this:
# leave the implementation as is, and make it clear to the users of soledad api that it is their responsibility to avoid leaking information through the id. This would imply having to change the bitmask mail implementation, if we really want to avoid such a leakage there.
# modify soledad to hide the original id and revision in some way. Currently, there are 2 proposals for this:
## replace the id for a hash, and store the original id somewhere inside the encrypted document content. This has the downside of having to store content in "tombstone" (deleted) documents, which usually have their content set to @None@.
## replace the id for an encrypted version, something in the lines of "iv:encrypted_id". This has the downside of having to come up with a way to derive one encryption key for each document without knowledge of the original document id, so a fresh install will be able to decrypt all documents successfully.
We should first decide if we will care for this and then decide for a solution if it is the case.
*(from redmine: created on 2016-12-08)*https://0xacab.org/leap/soledad/-/issues/8690BlobsBackend (client): specs for blobs upload state2017-11-02T17:03:30ZKali KanekoBlobsBackend (client): specs for blobs upload state# Definition of Done
a blobs spec is added to the docs/ folder.
# Discussion
We need to model how much state we do want to maintain in the client side for the blobs uploads.
- has a given blob been uploaded yet? yes/no? why? network er...# Definition of Done
a blobs spec is added to the docs/ folder.
# Discussion
We need to model how much state we do want to maintain in the client side for the blobs uploads.
- has a given blob been uploaded yet? yes/no? why? network errors? retries needed?
- on the download side, what to do if we try to retrieve metadata for a document that the other replica didn't upload yet? can we use the related documents at all?
I think we need to settle on very simplistic assumptions at first, just in order to deploy an MVP. But we need to be aware of the problems that are going to come later on on a multidevice scenario, and have at least some simplistic failure mode available.
*(from redmine: created on 2016-12-15)*
* Relations:
* relates #8665
* relates #8731
* relates #8823Soledad Backlog