BlobsBackend (client): specs for blobs upload state

Proposal:

blobs are immutable (there's no replace or delete for now).
one soledad doc points at most to one blob (this can easily be changed).
blob metadata is stored inside a soledad doc's content.
there is a separate component called "blobs client" (or blobs manager) that queries soledad and handles actual download/upload queues and local storage/retrieval of blobs data (no metadata handling here).

If a doc points to a blob, this is its content:

doc.content = {
    'is_blob': True,
    'blob_id: '',
    'origin': ''
    'uploaded': False,
    'upload_attempts': [],
}

A list of missing uploads can be reconstructed by querying:

is_blob = True
origin = local_replica_uid
uploaded = False

Downloads can be done on demand, but if a list of missing downloads is wanted, it can be reconstructed by querying:

is_blob = True
origin != local_replica_uid
uploaded = True

Questions:

do we want to be able to point to multiple blobs in one soledad doc? We may just change the structure to be stored as a list in doc.content['blobs'].

(from redmine: written on 2016-12-28)

And here are some requirements for the blobs client, based on latest discussion about the problems we want to solve with this:

Blobs Client Requirements

Representing data

blobs are immutable
metadata is stored in soledad documents
blobs client only handles download/upload and crypto of raw data.
store upload state in soledad document
update upload state after upload success or fail

Transferring data

has queues for upload and download
put stores local and enqueues for upload
get fetches local or enqueues for download
encrypt outgoing, decrypt incoming
limit how much of a blob is in memory each time
queues have file descriptors to read/write from/to.
if the blob was just put, have a way to reuse the file descriptor after writing to sqlcipher
have ability to reconstruct queues from doc indexes (resume missing uploads/download)

Protecting data

use sqlcipher for local storage, it it provides file descriptors (done)
uses aes-256-gcm for chunked crypto and mac (done)

(from redmine: written on 2016-12-28)

mentioned in issue #8760 (closed)

added ~1565 specs labels

blob metadata is stored inside a soledad doc's content

I'm not sure how much I like this. This means that for each new doc we save the metadata doc, we try to upload the blob, and then we modify the document and queue it for upload (or push it in an alternate future not so tied to the u1db rest api).

The advantage is that... a remote replica can know for sure that an upload was successful (instead of having to guess based on dangling refs). Something else?

Probably it's just a matter of quantifying the overhead.

encrypt outgoing, decrypt incoming (...) queues have file descriptors to read/write from/to.

this is a good candidate to be represented as a Tube

mentioned in issue #8768 (closed)

added Blocked- and removed ~1565 labels

The advantage is that... a remote replica can know for sure that an upload was successful (instead of having to guess based on dangling refs). Something else?

That is the advantage i see, but I also don't understand for sure what would be an alternative. Do you mean not having the information about uploaded state in the metadata? Suppose the client crashes, how would it gather a list of blobs to upload, if the uploaded state information is not stored in the metadata doc?

In reply to some transmogrified bits by gitlab@0xacab.org on the 2017-02-20 16:23:36:

The advantage is that... a remote replica can know for sure that an upload
was successful (instead of having to guess based on dangling refs).
Something else?
That is the advantage i see, but I also don't understand for sure what would be an alternative. Do you mean not having the information about uploaded state in the metadata? Suppose the client crashes, how would it gather a list of blobs to upload, if the uploaded state information is not stored in the metadata doc?

From the perspective of the replica-1 (the sender), the metadata might just be put in some temporary tables of the BlobManager.

blob_id queued for upload
blob_id failed to upload
blob_id retried to upload n times
blob_id successfuly uploaded

Once the blob is successfully uploaded, we can erase that status data.

With this alternative in mind, I'm not sure if it makes sense to pollute the doc json with the status info (has the overhead of re-uploading the doc after upload has been accomplished).

From the perspective of the replica-2 (the downloader), stuff gets more complicated. Let's say this one has just finished a metadata sync, and the application code (mail) is trying to fetch a mail. We have several options, in the server:

Refuse to serve (return 404, or something more appropiate, 409?). This means that client must retry, and wait until replica-1 finishes its upload. we could probably accomplish this by copying over the incoming stream to some staging area first.
Dispatch the GET and register this request as a consumer for the ongoing producer (The one that is reading from the socket that's doing the PUT from the replica-1). I think this is totally possible and used by some of the file sharing services out there. However, if the upload from replica-1 is corrupted/stalled the download will also be.

signature.asc

mentioned in issue #8779 (closed)

removed assignee

An update: !88 (closed) adds the ability to get attachment state. That is currently naivelly calculated by listing local and remote attachments, and looking up the name of the attachment in these lists. That can be improved by, eg, adding a field on local database.

It's good to keep the discussion here, but I think we could move much of what's been stated here to a docs/blobs-spec.rst document

it would be nice for the board view if blocked was for work happening now. if it's an issue for the future then during scoping of cycles, we can decide if its blocked at that point. does that work? can i remove the blocked label?

@shyba can i remove the block on this?

removed Blocked- label

changed the description

added docs label

removed Bug label

changed milestone to %Soledad Backlog

added Research label

added 1d of time spent

changed milestone to %0.10.3

changed the description

changed milestone to %0.10.4 - Cycle 5

changed milestone to %Cycle 6

changed milestone to %Soledad Backlog

@kali @drebs I think this was implemented on crash recovery (retry strategy and refactor).

@kali @drebs I think this was implemented on crash recovery (retry strategy and refactor).

I'm not sure I follow. The definition of done for this issue was a spec for transfer commited to the docs folder, has that been done? You're free to reject the proposed ticket, but the use of "implemented" on a documentation issue confuses me.

Ops, yes. It's a documentation one.

added blobs label

BlobsBackend (client): specs for blobs upload state

Definition of Done

Discussion

Designs

Child items ...

Activity

Blobs Client Requirements

Representing data

Transferring data

Protecting data