Syncing large blocks of data crashes bigcouch
In dev.bitmask.net (clam.dev.bitmask.net node), there were 2 users trying to sync at the same time, and the server died.
From Soledad Server (couchdb client) side:
2014-01-22 16:27:16+0000 [-] 190.195.199.186 - - [22/Jan/2014:16:27:16 +0000] "GET /user-1dbe3401678a7f8a6b64e1729bedba78/sync-from/292a5fb7c8b84a3e827aef03e9484be9 HTTP/1.1" 200 197 "-" "-"
2014-01-22 16:27:22+0000 [-] WSGI application error
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 195, in _worker
o = self.q.get()
File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 118, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 83, in callWithContext
self.contexts.pop()
File "/usr/lib/python2.7/dist-packages/twisted/web/wsgi.py", line 340, in run
self.started = True
--- ---
File "/usr/lib/python2.7/dist-packages/twisted/web/wsgi.py", line 315, in run
appIterator = self.application(self.environ, self.startResponse)
File "/usr/lib/python2.7/dist-packages/leap/soledad/server/__init__.py", line 199, in application
return application(environ, start_response)
File "/usr/lib/python2.7/dist-packages/leap/soledad/server/gzip_middleware.py", line 51, in __call__
app_iter = self.app(environ, dummy_start_response)
File "/usr/lib/python2.7/dist-packages/leap/soledad/server/auth.py", line 295, in __call__
return self._app(environ, start_response)
File "/usr/lib/python2.7/dist-packages/leap/soledad/server/__init__.py", line 149, in __call__
return http_app.HTTPApp.__call__(self, environ, start_response)
File "/usr/lib/python2.7/dist-packages/u1db/remote/http_app.py", line 581, in __call__
HTTPInvocationByMethodWithBody(resource, environ, self)()
File "/usr/lib/python2.7/dist-packages/u1db/remote/http_app.py", line 513, in __call__
return meth(args, None)
File "/usr/lib/python2.7/dist-packages/u1db/remote/http_app.py", line 174, in wrapper
return f(self, **args)
File "/usr/lib/python2.7/dist-packages/u1db/remote/http_app.py", line 356, in get
result = self.get_target().get_sync_info(self.source_replica_uid)
File "/usr/lib/python2.7/dist-packages/leap/soledad/common/couch.py", line 1210, in get_sync_info
my_gen, my_trans_id = self._db._get_generation_info()
File "/usr/lib/python2.7/dist-packages/leap/soledad/common/couch.py", line 446, in _get_generation_info
response = res.get_json()
File "/usr/lib/pymodules/python2.7/couchdb/http.py", line 393, in get_json
status, headers, data = self.get(*a, **k)
File "/usr/lib/pymodules/python2.7/couchdb/http.py", line 374, in get
return self._request('GET', path, headers=headers, **params)
File "/usr/lib/pymodules/python2.7/couchdb/http.py", line 419, in _request
credentials=self.credentials)
File "/usr/lib/pymodules/python2.7/couchdb/http.py", line 239, in request
resp = _try_request_with_retries(iter(self.retry_delays))
File "/usr/lib/pymodules/python2.7/couchdb/http.py", line 205, in _try_request_with_retries
raise e
socket.error: [Errno 111] Connection refused
2014-01-22 16:27:22+0000 [-] 190.195.199.186 - - [22/Jan/2014:16:27:22 +0000] "GET /user-800bf0183e754cdb967be84fe642e913/sync-from/1902617b857a41248258daef150bac22 HTTP/1.1" 500 - "-" "-"
2014-01-22 16:27:22+0000 [-] WSGI application error
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 195, in _worker
o = self.q.get()
File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 118, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 83, in callWithContext
self.contexts.pop()
File "/usr/lib/python2.7/dist-packages/twisted/web/wsgi.py", line 340, in run
self.started = True
--- ---
File "/usr/lib/python2.7/dist-packages/twisted/web/wsgi.py", line 315, in run
appIterator = self.application(self.environ, self.startResponse)
File "/usr/lib/python2.7/dist-packages/leap/soledad/server/__init__.py", line 199, in application
return application(environ, start_response)
File "/usr/lib/python2.7/dist-packages/leap/soledad/server/gzip_middleware.py", line 51, in __call__
app_iter = self.app(environ, dummy_start_response)
File "/usr/lib/python2.7/dist-packages/leap/soledad/server/auth.py", line 295, in __call__
return self._app(environ, start_response)
File "/usr/lib/python2.7/dist-packages/leap/soledad/server/__init__.py", line 149, in __call__
return http_app.HTTPApp.__call__(self, environ, start_response)
File "/usr/lib/python2.7/dist-packages/u1db/remote/http_app.py", line 581, in __call__
HTTPInvocationByMethodWithBody(resource, environ, self)()
File "/usr/lib/python2.7/dist-packages/u1db/remote/http_app.py", line 550, in __call__
meth_entry({}, entry)
File "/usr/lib/python2.7/dist-packages/u1db/remote/http_app.py", line 174, in wrapper
return f(self, **args)
File "/usr/lib/python2.7/dist-packages/u1db/remote/http_app.py", line 390, in post_stream_entry
self.sync_exch.insert_doc_from_source(doc, gen, trans_id)
File "/usr/lib/python2.7/dist-packages/u1db/sync.py", line 200, in insert_doc_from_source
replica_trans_id=trans_id)
File "/usr/lib/python2.7/dist-packages/leap/soledad/common/couch.py", line 1169, in _put_doc_if_newer
self._put_doc(cur_doc, doc)
File "/usr/lib/python2.7/dist-packages/leap/soledad/common/couch.py", line 669, in _put_doc
headers={'content-type': 'application/json'})
File "/usr/lib/pymodules/python2.7/couchdb/http.py", line 405, in put_json
status, headers, data = self.put(*a, **k)
File "/usr/lib/pymodules/python2.7/couchdb/http.py", line 384, in put
return self._request('PUT', path, body=body, headers=headers, **params)
File "/usr/lib/pymodules/python2.7/couchdb/http.py", line 419, in _request
credentials=self.credentials)
File "/usr/lib/pymodules/python2.7/couchdb/http.py", line 239, in request
resp = _try_request_with_retries(iter(self.retry_delays))
File "/usr/lib/pymodules/python2.7/couchdb/http.py", line 205, in _try_request_with_retries
raise e
socket.error: [Errno 111] Connection refused
2014-01-22 16:27:22+0000 [-] 186.231.118.122 - - [22/Jan/2014:16:27:22 +0000] "POST /user-847a6ece58ce498e4cd86cb10a6e9f98/sync-from/08c2962dc4c043479a7c381f208cfc5e HTTP/1.1" 500 - "-" "-"
From Couch DB server side:
[Wed, 22 Jan 2014 16:27:14 GMT] [info] [<0.19101.0>] [bd7319c6] undefined - - 'HEAD' /user-1dbe3401678a7f8a6b64e1729bedba78 200
[Wed, 22 Jan 2014 16:27:14 GMT] [info] [<0.19101.0>] [bd7319c6] 127.0.0.1 localhost:5984 HEAD /user-1dbe3401678a7f8a6b64e1729bedba78 200 ok 41
[Wed, 22 Jan 2014 16:27:14 GMT] [info] [<0.19110.0>] [d10642ea] undefined - - 'HEAD' /user-1dbe3401678a7f8a6b64e1729bedba78 200
[Wed, 22 Jan 2014 16:27:14 GMT] [info] [<0.19110.0>] [d10642ea] 127.0.0.1 localhost:5984 HEAD /user-1dbe3401678a7f8a6b64e1729bedba78 200 ok 58
[Wed, 22 Jan 2014 16:27:15 GMT] [info] [<0.19110.0>] [54a2df78] undefined - - 'GET' /user-1dbe3401678a7f8a6b64e1729bedba78/_design%2Fdocs 200
[Wed, 22 Jan 2014 16:27:15 GMT] [info] [<0.19110.0>] [54a2df78] 127.0.0.1 localhost:5984 GET /user-1dbe3401678a7f8a6b64e1729bedba78/_design%2Fdocs 200 ok 42
[Wed, 22 Jan 2014 16:27:15 GMT] [info] [<0.19110.0>] [7859995e] undefined - - 'GET' /user-1dbe3401678a7f8a6b64e1729bedba78/_design/syncs/_view/log?key=%22292a5fb7c8b84a3e827aef03e9484be9%22 2
[Wed, 22 Jan 2014 16:27:15 GMT] [info] [<0.19110.0>] [7859995e] 127.0.0.1 localhost:5984 GET /user-1dbe3401678a7f8a6b64e1729bedba78/_design/syncs/_view/log?key=%22292a5fb7c8b84a3e827aef03e948
[Wed, 22 Jan 2014 16:27:15 GMT] [info] [<0.19110.0>] [d932a411] undefined - - 'GET' /user-1dbe3401678a7f8a6b64e1729bedba78/_design/syncs/_view/log?key=%22292a5fb7c8b84a3e827aef03e9484be9%22 2
[Wed, 22 Jan 2014 16:27:15 GMT] [info] [<0.19110.0>] [d932a411] 127.0.0.1 localhost:5984 GET /user-1dbe3401678a7f8a6b64e1729bedba78/_design/syncs/_view/log?key=%22292a5fb7c8b84a3e827aef03e948
[Wed, 22 Jan 2014 16:27:15 GMT] [info] [<0.19110.0>] [67eee2f3] undefined - - 'GET' /user-1dbe3401678a7f8a6b64e1729bedba78/_design/syncs/_view/log?key=%22292a5fb7c8b84a3e827aef03e9484be9%22 2
[Wed, 22 Jan 2014 16:27:15 GMT] [info] [<0.19110.0>] [67eee2f3] 127.0.0.1 localhost:5984 GET /user-1dbe3401678a7f8a6b64e1729bedba78/_design/syncs/_view/log?key=%22292a5fb7c8b84a3e827aef03e948
[Wed, 22 Jan 2014 16:27:15 GMT] [info] [<0.19165.0>] [b0cb43fc] undefined - - 'GET' / 200
[Wed, 22 Jan 2014 16:27:15 GMT] [info] [<0.19165.0>] [b0cb43fc] 127.0.0.1 undefined GET / 200 ok 21
[Wed, 22 Jan 2014 16:27:15 GMT] [info] [<0.19185.0>] [2b7cdfa9] undefined - - 'GET' / 200
[Wed, 22 Jan 2014 16:27:15 GMT] [info] [<0.19185.0>] [2b7cdfa9] 127.0.0.1 undefined GET / 200 ok 18
[Wed, 22 Jan 2014 16:27:16 GMT] [info] [<0.19110.0>] [39b83e48] 127.0.0.1 localhost:5984 GET /user-1dbe3401678a7f8a6b64e1729bedba78/_design/transactions/_list/generation/log 200 ok 1136
[Wed, 22 Jan 2014 16:27:16 GMT] [info] [<0.19110.0>] [afc2d512] undefined - - 'GET' /user-1dbe3401678a7f8a6b64e1729bedba78/u1db_config 200
[Wed, 22 Jan 2014 16:27:16 GMT] [info] [<0.19110.0>] [afc2d512] 127.0.0.1 localhost:5984 GET /user-1dbe3401678a7f8a6b64e1729bedba78/u1db_config 200 ok 25
[Wed, 22 Jan 2014 16:27:23 GMT] [info] [<0.93.0>] [--------] Apache CouchDB has started on http://127.0.0.1:5986/
[Wed, 22 Jan 2014 16:27:23 GMT] [info] [<0.222.0>] [53d48221] undefined - - 'GET' / 200
[Wed, 22 Jan 2014 16:27:23 GMT] [info] [<0.224.0>] [03edb59a] undefined - - 'GET' / 200
[Wed, 22 Jan 2014 16:27:24 GMT] [error] [<0.222.0>] [53d48221] Uncaught error in HTTP request: {exit,normal}
[Wed, 22 Jan 2014 16:27:24 GMT] [error] [<0.224.0>] [03edb59a] Uncaught error in HTTP request: {exit,normal}
[Wed, 22 Jan 2014 16:27:24 GMT] [info] [<0.222.0>] [53d48221] Stacktrace: [{mochiweb_request,send,2,
[{file,"src/mochiweb_request.erl"},
{line,146}]},
{mochiweb_request,respond,2,
[{file,"src/mochiweb_request.erl"},
{line,330}]},
{couch_httpd,send_response,4,
[{file,"src/couch_httpd.erl"},
{line,618}]},
{chttpd,handle_request,1,
[{file,"src/chttpd.erl"},
{line,198}]},
{mochiweb_http,headers,5,
[{file,"src/mochiweb_http.erl"},
{line,126}]},
{proc_lib,init_p_do_apply,3,
[{file,"proc_lib.erl"},
{line,227}]}]
[Wed, 22 Jan 2014 16:27:24 GMT] [info] [<0.224.0>] [03edb59a] Stacktrace: [{mochiweb_request,send,2,
[{file,"src/mochiweb_request.erl"},
{line,146}]},
{mochiweb_request,respond,2,
[{file,"src/mochiweb_request.erl"},
{line,330}]},
{couch_httpd,send_response,4,
[{file,"src/couch_httpd.erl"},
{line,618}]},
{chttpd,handle_request,1,
[{file,"src/chttpd.erl"},
{line,198}]},
{mochiweb_http,headers,5,
[{file,"src/mochiweb_http.erl"},
{line,126}]},
{proc_lib,init_p_do_apply,3,
[{file,"proc_lib.erl"},
{line,227}]}]
[Wed, 22 Jan 2014 16:27:24 GMT] [info] [<0.222.0>] [53d48221] undefined - - 'GET' / 500
[Wed, 22 Jan 2014 16:27:24 GMT] [info] [<0.224.0>] [03edb59a] undefined - - 'GET' / 500
(from redmine: created on 2014-01-22, closed on 2014-04-15)