soledad issueshttps://0xacab.org/leap/soledad/-/issues2017-07-03T12:32:53Zhttps://0xacab.org/leap/soledad/-/issues/8748add blobs interface to pysqlcipher2017-07-03T12:32:53Zmeskioadd blobs interface to pysqlcipherSoledad 0.9.3meskiomeskiohttps://0xacab.org/leap/soledad/-/issues/8750make blobs feature configurable in the soledad server, via a config file2017-07-03T12:32:50ZVictormake blobs feature configurable in the soledad server, via a config fileSoledad 0.9.3https://0xacab.org/leap/soledad/-/issues/8745make blobs announced by the soledad server (capabilities in GET / )2017-03-22T16:06:37ZVictormake blobs announced by the soledad server (capabilities in GET / )Soledad 0.9.3https://0xacab.org/leap/soledad/-/issues/8749make client able to check blobs capability on a server2017-03-22T16:06:37ZVictormake client able to check blobs capability on a serverSoledad 0.9.3https://0xacab.org/leap/soledad/-/issues/8746write a function that checks authentication for a blob client request2017-07-03T12:32:55ZVictorwrite a function that checks authentication for a blob client requestSoledad 0.9.3https://0xacab.org/leap/soledad/-/issues/8751Implement a simple cooperative file backend for blobs-io server2017-03-22T16:06:37ZVictorImplement a simple cooperative file backend for blobs-io serverArchivehttps://0xacab.org/leap/soledad/-/issues/8752uploads: write standalone service that handles raw upload from a client and w...2017-03-22T16:06:37ZVictoruploads: write standalone service that handles raw upload from a client and writes the stream, cooperatively, to specified pathArchivehttps://0xacab.org/leap/soledad/-/issues/8753profile performance and responsiveness of blobs-io service.2017-09-27T18:38:12ZVictorprofile performance and responsiveness of blobs-io service.Need triaginghttps://0xacab.org/leap/soledad/-/issues/8754downloads: write a service that serves all the tree of downloadable blobs for...2017-03-22T16:06:37ZVictordownloads: write a service that serves all the tree of downloadable blobs for a given userArchivehttps://0xacab.org/leap/soledad/-/issues/8755plug authentication call to blob-uploads service2017-03-22T16:06:37ZVictorplug authentication call to blob-uploads serviceSoledad 0.9.3https://0xacab.org/leap/soledad/-/issues/8756plug authentication call to blob-downloads service2017-03-22T16:06:37ZVictorplug authentication call to blob-downloads serviceSoledad 0.9.3https://0xacab.org/leap/soledad/-/issues/8747add test to ensure the new format of secrets.json is not broken after secrets...2017-07-03T12:32:55Zdrebsadd test to ensure the new format of secrets.json is not broken after secrets refactorWe have recently [refactored secrets storage and crypto code](https://0xacab.org/leap/soledad/commit/96c2aa022e42788fedcb2753d04828583e6fba06). Pixelated had MAC auth failures when trying to use the refactored code, so there is probably ...We have recently [refactored secrets storage and crypto code](https://0xacab.org/leap/soledad/commit/96c2aa022e42788fedcb2753d04828583e6fba06). Pixelated had MAC auth failures when trying to use the refactored code, so there is probably something broken in the refactor and the old secrets are not being stored correctly after the refactor.
We have to add a test to ensure whatever is stored in the new format matches exactly what was stored in the old format.Soledad 0.9.3VictorVictorhttps://0xacab.org/leap/soledad/-/issues/8864soledad migration script improvements2017-12-04T18:40:26Zelijahsoledad migration script improvementsSome questions from sysdevs:
* Can the migration script be interrupted and resumed? or if it is interrupted, does it need to start over?
* What happens if the script is mid-way through converting a user db and the power is pulled?
* Can...Some questions from sysdevs:
* Can the migration script be interrupted and resumed? or if it is interrupted, does it need to start over?
* What happens if the script is mid-way through converting a user db and the power is pulled?
* Can the migrate.py script be made to output some nice progress output?
* Can migrate.py script print DONE when finished?
* Can we change migration/0.8.2 to migration/0.10, since it is migrating *to* 0.10, not 0.8.2?
*(from redmine: created on 2016-08-30)*Need triaginghttps://0xacab.org/leap/soledad/-/issues/8744Missing couch config document on soledad-server start2017-06-24T00:11:40ZVaracMissing couch config document on soledad-server startThis happened out of the blue on a pixelated provider:
<pre>
root@yyy:/var/run# systemctl start soledad-server.service
root@yyy:/var/run# grep soledad /var/log/messages | tail -15
Jan 6 12:00:06 yyy systemd[1]: Stopped LSB: Start sole...This happened out of the blue on a pixelated provider:
<pre>
root@yyy:/var/run# systemctl start soledad-server.service
root@yyy:/var/run# grep soledad /var/log/messages | tail -15
Jan 6 12:00:06 yyy systemd[1]: Stopped LSB: Start soledad daemon at boot time.
Jan 6 12:00:18 yyy systemd[1]: Starting LSB: Start soledad daemon at boot time...
Jan 6 12:00:19 yyy soledad-server: [-] twistd 16.2.0 (/usr/bin/python 2.7.9) starting up.
Jan 6 12:00:19 yyy soledad-server: [-] reactor class: twisted.internet.epollreactor.EPollReactor.
Jan 6 12:00:19 yyy soledad-server: [-] Site (TLS) starting on 2323
Jan 6 12:00:19 yyy soledad-server: [-] Starting factory <twisted.web.server.Site instance at 0x7f568b1dbe18>
Jan 6 12:00:19 yyy soledad-server: [-] set uid/gid 999/999
Jan 6 12:00:19 yyy soledad-server[7139]: Starting soledad: twistd.
Jan 6 12:00:19 yyy systemd[1]: Started LSB: Start soledad daemon at boot time.
Jan 6 12:00:19 yyy soledad-server: [leap.soledad.common.couch.state#error] Missing couch config document in database user-001b9e841714714618fde47daa1e2df8
Jan 6 12:00:19 yyy soledad-server: [-] (TLS Port 2323 Closed)
Jan 6 12:00:19 yyy soledad-server: [-] Stopping factory <twisted.web.server.Site instance at 0x7f568b1dbe18>
Jan 6 12:00:19 yyy soledad-server: [-] Main loop terminated.
Jan 6 12:00:19 yyy soledad-server: [-] Warning: No permission to delete pid file
Jan 6 12:00:19 yyy soledad-server: [-] Server Shut Down.
</pre>
At this time, soledad-* 0.8.1~271.gbp57447e was installed from the pixelated repo, which is based on 0.9.1:
<pre>
--- git/soledad ‹pixelated_develop* ⁇› » git lg |head
* 57447e6 - (HEAD -> pixelated_develop, pixelated/develop) Fix pep8 (6 weeks ago) <Tulio Casagrande>
* 5b2e522 - Merge tag '0.9.1' into develop (6 weeks ago) <drebs>
|\
| * dff666e - (tag: 0.9.1) [pkg] update changelog (6 weeks ago) <drebs>
| * c44783f - [bug] fix import on create-user-db (6 weeks ago) <Victor Shyba>
</pre>
*(from redmine: created on 2017-01-11, closed on 2017-01-11)*
* Relations:
* blocks #8742https://0xacab.org/leap/soledad/-/issues/8742Soledad 0.9 migration script fails to execute2017-06-24T00:11:40ZVaracSoledad 0.9 migration script fails to execute<pre>
root@aardwolf:/tmp/test# git clone -b develop https://0xacab.org/leap/soledad.git
Cloning into 'soledad'...
remote: Counting objects: 13368, done.
remote: Compressing objects: 100% (5489/5489), done.
remote: Total 13368 (delta 7382...<pre>
root@aardwolf:/tmp/test# git clone -b develop https://0xacab.org/leap/soledad.git
Cloning into 'soledad'...
remote: Counting objects: 13368, done.
remote: Compressing objects: 100% (5489/5489), done.
remote: Total 13368 (delta 7382), reused 12211 (delta 6757)
Receiving objects: 100% (13368/13368), 2.82 MiB | 5.21 MiB/s, done.
Resolving deltas: 100% (7382/7382), done.
Checking connectivity... done.
root@aardwolf:/tmp/test# cd soledad/scripts/migration/0.9.0/
root@aardwolf:/tmp/test/soledad/scripts/migration/0.9.0# ./migrate.py --help
Traceback (most recent call last):
File "./migrate.py", line 30, in <module>
from leap.soledad.server import load_configuration
ImportError: cannot import name load_configuration
</pre>
*(from redmine: created on 2017-01-11, closed on 2017-01-11)*
* Relations:
* blocks #8744https://0xacab.org/leap/soledad/-/issues/8732Could not authenticate document's contents when syncing old emails with the n...2017-06-24T00:11:39ZAnikeCould not authenticate document's contents when syncing old emails with the new streaming Soledad featureWhile testing the updated Soledad version that contains streaming branch improvements in Pixelated, we were unable to sync old emails. In other words, all emails that were sent with the new version were received and synced without a prob...While testing the updated Soledad version that contains streaming branch improvements in Pixelated, we were unable to sync old emails. In other words, all emails that were sent with the new version were received and synced without a problem. However, the emails that the user already had were now not showing up.
We get the following error: "Could not authenticate document's contents"
In debug mode, we printed out the contents of the email that was being synched and eventually the error that we got.
<pre><code class="text">
2017-01-06 13:10:54 [stdout] INFO > /home/vagrant/user-agent-venv/src/leap.soledad.client/client/src/leap/soledad/client/http_target/fetch.py(118)__atomic_doc_parse()
2017-01-06 13:10:54 [stdout] INFO -> if is_symmetrically_encrypted(content):
l
2017-01-06 13:10:58 [stdout] INFO (Pdb) 113
2017-01-06 13:10:58 [stdout] INFO 114 @defer.inlineCallbacks
2017-01-06 13:10:58 [stdout] INFO 115 def __atomic_doc_parse(self, doc_info, content, total):
2017-01-06 13:10:58 [stdout] INFO 116 doc = SoledadDocument(doc_info['id'], doc_info['rev'], content)
2017-01-06 13:10:58 [stdout] INFO 117 import pdb; pdb.set_trace()
2017-01-06 13:10:58 [stdout] INFO 118 -> if is_symmetrically_encrypted(content):
2017-01-06 13:10:58 [stdout] INFO 119 content = yield self._crypto.decrypt_doc(doc)
2017-01-06 13:10:58 [stdout] INFO 120 elif old_crypto.is_symmetrically_encrypted(doc):
2017-01-06 13:10:58 [stdout] INFO 121 content = self._deprecated_crypto.decrypt_doc(doc)
2017-01-06 13:10:58 [stdout] INFO 122 doc.set_json(content)
2017-01-06 13:10:58 [stdout] INFO 123
content
2017-01-06 13:11:05 [stdout] INFO (Pdb) '{"_mac_method": "hmac", "_enc_scheme": "symkey", "_enc_json": "6c75c0c7e59be9b11c4460803bcd4bc1650fd68cdd4a980d2e458317a07bf47937b721b423b3c950ac0aea64c6111ff017749b18ae72a8d332b7af9aab21bde6c745653d03f8b05fae9bae467e3d5127890824e b1fdfa68bbaee4838d9f2dedf1130296cad9e929d4ba96ddb680df89415510e6a6fab3c9bcdd6ca1f4d79096d59be4d2dd443d8298c3ca0556204ede8a3d28e6ca57c89a766054c37f62a7ac651a13791be05421a5b276c6f82b68fd65fbdda3d5dbbbd07aefea770d5d20f6502d59895086ad2f6ee16370096a9f9cc132c121b197d39c4d63023 5cb4795d0270d48ed9196d39dade116a1d1ca7a849d76f5440a8eb14894a8245488145fba0c70d6d8b05a58cec3ed99d41ee0124d6daefab88cd469b89325946a63311277f", "_mac": "d08f9581675f670369e429eb0829327f8fe486bafc66892223df1e4cbc293961", "_enc_iv": "SUZldZ2G3xHHh6LBADK0vg==\\n", "_enc_method ": "aes-256-ctr"}'
c
2017-01-06 13:11:58 [leap.soledad.client.crypto] WARN wrong MAC while decrypting doc...
2017-01-06 13:11:58 [twisted.internet.defer] CRITICAL Unhandled error in Deferred:
2017-01-06 13:11:58 [stdout] INFO (Pdb) > /home/vagrant/user-agent-venv/local/lib/python2.7/site-packages/twisted/internet/defer.py(700)__del__()
2017-01-06 13:11:58 [stdout] INFO -> debugInfo = self._getDebugTracebacks()
l
2017-01-06 13:12:05 [stdout] INFO (Pdb) 695 # earlier tests; arguably it should be a single error message.
2017-01-06 13:12:05 [stdout] INFO 696 log.critical("Unhandled error in Deferred:",
2017-01-06 13:12:05 [stdout] INFO 697 isError=True)
2017-01-06 13:12:05 [stdout] INFO 698
2017-01-06 13:12:05 [stdout] INFO 699 import pdb; pdb.set_trace()
2017-01-06 13:12:05 [stdout] INFO 700 -> debugInfo = self._getDebugTracebacks()
2017-01-06 13:12:05 [stdout] INFO 701 if debugInfo:
2017-01-06 13:12:05 [stdout] INFO 702 format = "(debug: {debugInfo})"
2017-01-06 13:12:05 [stdout] INFO 703 else:
2017-01-06 13:12:05 [stdout] INFO 704 format = None
2017-01-06 13:12:05 [stdout] INFO 705
self.failResult
2017-01-06 13:12:13 [stdout] INFO (Pdb) <twisted.python.failure.Failure leap.soledad.common.crypto.WrongMacError: Could not authenticate document's contents.>
</code></pre>
Versions:
* Soledad: leap/develop branch, commit 2997fe41d6b971a5a53a562686a652be8fc0dac5
* Pixelated User Agent: pixelated/soledad-merge branch, commit 3097e7b16db204c4359a27b63d1a493a89e29b14 (but, locally, we changed the services/requirements.txt to use the leap/develop branch)
* Pixelated provider: dev.pixelated-project.org (which temporarily contains the soledad streaming feature)
*(from redmine: created on 2017-01-06)*Soledad 0.9.3https://0xacab.org/leap/soledad/-/issues/8731add blobs interface to pysqlcipher2017-06-29T14:28:35Zmeskioadd blobs interface to pysqlcipher*(from redmine: created on 2017-01-04)*
* Relations:
* relates #8690*(from redmine: created on 2017-01-04)*
* Relations:
* relates #8690meskiomeskiohttps://0xacab.org/leap/soledad/-/issues/8721verify if removal of syncable flag should be reverted or if bitmask client sh...2017-06-27T13:11:21Zdrebsverify if removal of syncable flag should be reverted or if bitmask client should be updatedThe `syncable` property of Soledad [was removed](https://0xacab.org/leap/soledad/commit/96c2aa022e42788fedcb2753d04828583e6fba06), but the bitmask client code [was not updated accordingly](https://0xacab.org/leap/bitmask-dev/blob/master/...The `syncable` property of Soledad [was removed](https://0xacab.org/leap/soledad/commit/96c2aa022e42788fedcb2753d04828583e6fba06), but the bitmask client code [was not updated accordingly](https://0xacab.org/leap/bitmask-dev/blob/master/src/leap/bitmask/core/mail_services.py#L129).
We have to check:
* what is the meaning of that flag, and if its removal should be reverted.
* if it doesn't need to be reverted, how to fix bitmask client.
*(from redmine: created on 2017-01-02)*Soledad 0.9.3https://0xacab.org/leap/soledad/-/issues/8711Measure couch token request overhead and compare with simple caching alternative2017-06-24T00:11:39ZdrebsMeasure couch token request overhead and compare with simple caching alternativeIt is not clear whether querying couch on every client request is a performance problem. We need to measure the performance of the current solution against a simple caching alternative to have a clear idea if improvement is needed.
Even...It is not clear whether querying couch on every client request is a performance problem. We need to measure the performance of the current solution against a simple caching alternative to have a clear idea if improvement is needed.
Even if it needed, it is not clear which direction to go for caching. Twisted sessions send the session id (an no session data) in cookie request headers back and forth to retrieve session. We already have that implemented if it is to be considered as a valid option.
*(from redmine: created on 2016-12-29)*
* Relations:
* relates #8667https://0xacab.org/leap/soledad/-/issues/8701Fix lintian problem2017-06-24T00:11:39ZdrebsFix lintian problem"Jenkins is reporting a lintian error when building soledad packages":http://squirrel.leap.se:8080/job/soledad-0.9-binaries/lastCompletedBuild/architecture=amd64,distribution=jessie/testReport/lintian/soledad-server/postrm_contains_addit..."Jenkins is reporting a lintian error when building soledad packages":http://squirrel.leap.se:8080/job/soledad-0.9-binaries/lastCompletedBuild/architecture=amd64,distribution=jessie/testReport/lintian/soledad-server/postrm_contains_additional_updaterc_d_calls_etc_init_d_soledad_server/:
<pre>
Failed
lintian.soledad-server.postrm-contains-additional-updaterc.d-calls etc/init.d/soledad-server (from (lintian.xml))
Failing for the past 4 builds (Since #110 )
Took 2 sec.
Error Message
E: soledad-server: postrm-contains-additional-updaterc.d-calls etc/init.d/soledad-server
Standard Output
N: The postrm de-registers an /etc/init.d script which has not been
N: registered in the postinst script before.
N:
N: Severity: important, Certainty: certain
N:
N: Check: init.d, Type: binary
</pre>
*(from redmine: created on 2016-12-23, closed on 2016-12-23)*