sqlcipher blobs backend fails when there are too many concurrent operations

This is an issue which is a bit tricky because reproduction seems to depend on hardware (i.e. ssd vs hd) and there are some different errors that may appear. We should improve the details in this issue as they appear.

How to reproduce:

git clone https://0xacab.org/drebs/soledad -b 8942
cd soledad/testing/
tox -e benchmark -v -- -v -s -x -m sqlite_blobs_backend

The expected behaviour is to have tests passing and get benchmarking tables in output. What i actually get is:

E           FirstError: FirstError[#295, [Failure instance: Traceback: <class 'pysqlcipher.dbapi2.OperationalError'>: cannot commit transaction - SQL statements in progress
E           /home/drebs/dev/leap/repos/soledad/testing/.tox/benchmark/local/lib/python2.7/site-packages/twisted/internet/defer.py:500:errback
E           /home/drebs/dev/leap/repos/soledad/testing/.tox/benchmark/local/lib/python2.7/site-packages/twisted/internet/defer.py:567:_startRunCallbacks
E           /home/drebs/dev/leap/repos/soledad/testing/.tox/benchmark/local/lib/python2.7/site-packages/twisted/internet/defer.py:653:_runCallbacks
E           /home/drebs/dev/leap/repos/soledad/testing/.tox/benchmark/local/lib/python2.7/site-packages/twisted/internet/defer.py:1442:gotResult
E           --- <exception caught here> ---
E           /home/drebs/dev/leap/repos/soledad/testing/.tox/benchmark/local/lib/python2.7/site-packages/twisted/internet/defer.py:1384:_inlineCallbacks
E           /home/drebs/dev/leap/repos/soledad/testing/.tox/benchmark/local/lib/python2.7/site-packages/twisted/python/failure.py:393:throwExceptionIntoGenerator
E           /home/drebs/dev/leap/repos/soledad/src/leap/soledad/client/_db/blobs.py:463:put
E           /home/drebs/dev/leap/repos/soledad/testing/.tox/benchmark/local/lib/python2.7/site-packages/twisted/python/threadpool.py:250:inContext
E           /home/drebs/dev/leap/repos/soledad/testing/.tox/benchmark/local/lib/python2.7/site-packages/twisted/python/threadpool.py:266:<lambda>
E           /home/drebs/dev/leap/repos/soledad/testing/.tox/benchmark/local/lib/python2.7/site-packages/twisted/python/context.py:122:callWithContext
E           /home/drebs/dev/leap/repos/soledad/testing/.tox/benchmark/local/lib/python2.7/site-packages/twisted/python/context.py:85:callWithContext
E           /home/drebs/dev/leap/repos/soledad/testing/.tox/benchmark/local/lib/python2.7/site-packages/twisted/enterprise/adbapi.py:477:_runInteraction
E           /home/drebs/dev/leap/repos/soledad/testing/.tox/benchmark/local/lib/python2.7/site-packages/twisted/enterprise/adbapi.py:469:_runInteraction
E           ]]

Beware that pytest+twisted output might be misleading and you might see other errors surrounding the error above. Also, you might get errors other than cannot commit transaction, but in general they will indicate that there's some concurrency issue going on.

The strangest thing for me so far is that I wouldn't expect concurrency errors because the twisted adbapi module handles a db connection pool and should not allow for concurrent writes to the db (i guess). Maybe I'm mistaken about how twisted adbapi handles concurrency.

One way I could avoid this kind of errors was by using a semaphore to limit the concurrency level, but it is only effective in some scenarios and not in others. I can give more details as the error keeps showing up.