CouchDB restart while syncing
We are seeing the following error on Pixelated user-agent side while syncing (we are seeing several instances of these) -
. Mar 29 23:28:54 dev1 pixelated-user-agent[4399]: 2017-03-29 23:28:54 [leap.bitmask.mail.incoming.service] DEBUG fetching mail for: 024e8a3d57227f6821e92d30a736d778 test_user_rrajxlntonizsrzh@dev.pixelated-project.org
. Mar 29 23:28:54 dev1 pixelated-user-agent[4399]: 2017-03-29 23:28:54 [leap.bitmask.mail.incoming.service] INFO starting sync...
. Mar 29 23:28:54 dev1 pixelated-user-agent[4399]: 2017-03-29 23:28:54 [twisted.web.client._HTTP11ClientFactory] INFO Starting factory <twisted.web.client._HTTP11ClientFactory instance at 0x7f60381ab170>
. Mar 29 23:28:54 dev1 pixelated-user-agent[4399]: 2017-03-29 23:28:54 [leap.soledad.client.api] ERROR got exception when syncing!
. Mar 29 23:28:54 dev1 pixelated-user-agent[4399]: *--- Failure #5480 ---
C Mar 29 23:28:54 dev1 pixelated-user-agent[4399]: Failure: leap.soledad.common.l2db.errors.HTTPError: HTTPError(500, '\n<html>\n <head><title>500 - None</title></head>\n <body>\n <h1>None</h1>\n <p>None</p>\n </body>\n</html>\n')
. Mar 29 23:28:54 dev1 pixelated-user-agent[4399]: *--- End of Failure #5480 ---
C Mar 29 23:28:54 dev1 pixelated-user-agent[4399]: 2017-03-29 23:28:54 [leap.bitmask.mail.incoming.service] ERROR [Failure instance: Traceback (failure with no frames): <class 'leap.soledad.common.l2db.errors.HTTPError'>: HTTPError(500, '\n<html>\n <head><title>500 - None</title></head>\n <body>\n <h1>None</h1>\n <p>None</p>\n </body>\n</html>\n')
. Mar 29 23:28:54 dev1 pixelated-user-agent[4399]: ]
. Mar 29 23:28:54 dev1 pixelated-user-agent[4399]: 2017-03-29 23:28:54 [twisted.web.client._HTTP11ClientFactory] INFO Stopping factory <twisted.web.client._HTTP11ClientFactory instance at 0x7f60381ab170>
The corresponding log on Soledad below shows that the CouchDB service restarted while the sync was happening. We are not sure if this is an abnormal behavior or something that is an actual error.
Mar 29 23:28:54 dev1 soledad-server: [-] 0.0.0.0 - - [29/Mar/2017:23:28:53 +0000] "GET /shared/doc/84b95668044e49d76f0957933c6a8b280ec367698fa087c686d24692806da4a9?include_deleted=false HTTP/1.1" 500 110 "-" "-"
Mar 29 23:28:54 dev1 systemd[1]: couchdb.service: Service hold-off time over, scheduling restart.
Mar 29 23:28:54 dev1 systemd[1]: Stopped CouchDB Server.
Mar 29 23:28:54 dev1 systemd[1]: Starting CouchDB Server...
Mar 29 23:28:54 dev1 couchdb[11842]: Apache CouchDB has started, time to relax.
Mar 29 23:28:54 dev1 systemd[1]: couchdb.service: Failed to read PID from file /run/couchdb/couchdb.pid: Invalid argument
Mar 29 23:28:54 dev1 soledad-server: [_GenericHTTPChannelProtocol (TLSMemoryBIOProtocol),2800,0.0.0.0] HTTPAuthSessionWrapper.getChildWithDefault encountered unexpected error
Mar 29 23:28:54 dev1 soledad-server: [_GenericHTTPChannelProtocol (TLSMemoryBIOProtocol),2800,0.0.0.0] #011Traceback (most recent call last):
Mar 29 23:28:54 dev1 soledad-server: [_GenericHTTPChannelProtocol (TLSMemoryBIOProtocol),2800,0.0.0.0] #011 File "/usr/lib/python2.7/dist-packages/twisted/web/_auth/wrapper.py", line 147, in getChildWithDefault
Mar 29 23:28:54 dev1 soledad-server: [_GenericHTTPChannelProtocol (TLSMemoryBIOProtocol),2800,0.0.0.0] #011 return self._authorizedResource(request)
Mar 29 23:28:54 dev1 soledad-server: [_GenericHTTPChannelProtocol (TLSMemoryBIOProtocol),2800,0.0.0.0] #011 File "/usr/lib/python2.7/dist-packages/leap/soledad/server/session.py", line 107, in _authorizedResource
Mar 29 23:28:54 dev1 soledad-server: [_GenericHTTPChannelProtocol (TLSMemoryBIOProtocol),2800,0.0.0.0] #011 return util.DeferredResource(self._login(credentials))
Mar 29 23:28:54 dev1 soledad-server: [_GenericHTTPChannelProtocol (TLSMemoryBIOProtocol),2800,0.0.0.0] #011 File "/usr/lib/python2.7/dist-packages/twisted/web/_auth/wrapper.py", line 157, in _login
Mar 29 23:28:54 dev1 soledad-server: [_GenericHTTPChannelProtocol (TLSMemoryBIOProtocol),2800,0.0.0.0] #011 d = self._portal.login(credentials, None, IResource)
Mar 29 23:28:54 dev1 soledad-server: [_GenericHTTPChannelProtocol (TLSMemoryBIOProtocol),2800,0.0.0.0] #011 File "/usr/lib/python2.7/dist-packages/twisted/cred/portal.py", line 119, in login
Mar 29 23:28:54 dev1 soledad-server: [_GenericHTTPChannelProtocol (TLSMemoryBIOProtocol),2800,0.0.0.0] #011 return maybeDeferred(self.checkers[i].requestAvatarId, credentials
Mar 29 23:28:54 dev1 soledad-server: [_GenericHTTPChannelProtocol (TLSMemoryBIOProtocol),2800,0.0.0.0] #011--- <exception caught here> ---
Mar 29 23:28:54 dev1 soledad-server: [_GenericHTTPChannelProtocol (TLSMemoryBIOProtocol),2800,0.0.0.0] #011 File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 150, in maybeDeferred
Mar 29 23:28:54 dev1 soledad-server: [_GenericHTTPChannelProtocol (TLSMemoryBIOProtocol),2800,0.0.0.0] #011 result = f(*args, **kw)
Mar 29 23:28:54 dev1 soledad-server: [_GenericHTTPChannelProtocol (TLSMemoryBIOProtocol),2800,0.0.0.0] #011 File "/usr/lib/python2.7/dist-packages/leap/soledad/server/auth.py", line 125, in requestAvatarId
Mar 29 23:28:54 dev1 soledad-server: [_GenericHTTPChannelProtocol (TLSMemoryBIOProtocol),2800,0.0.0.0] #011 db = self._tokens_db()
Mar 29 23:28:54 dev1 soledad-server: [_GenericHTTPChannelProtocol (TLSMemoryBIOProtocol),2800,0.0.0.0] #011 File "/usr/lib/python2.7/dist-packages/leap/soledad/server/auth.py", line 112, in _tokens_db
Mar 29 23:28:54 dev1 soledad-server: [_GenericHTTPChannelProtocol (TLSMemoryBIOProtocol),2800,0.0.0.0] #011 db = server[dbname]
Mar 29 23:28:54 dev1 soledad-server: [_GenericHTTPChannelProtocol (TLSMemoryBIOProtocol),2800,0.0.0.0] #011 File "/usr/lib/python2.7/dist-packages/couchdb/client.py", line 141, in __getitem__
Mar 29 23:28:54 dev1 soledad-server: [_GenericHTTPChannelProtocol (TLSMemoryBIOProtocol),2800,0.0.0.0] #011 db.resource.head() # actually make a request to the database
Mar 29 23:28:54 dev1 soledad-server: [_GenericHTTPChannelProtocol (TLSMemoryBIOProtocol),2800,0.0.0.0] #011 File "/usr/lib/python2.7/dist-packages/couchdb/http.py", line 542, in head
Mar 29 23:28:54 dev1 soledad-server: [_GenericHTTPChannelProtocol (TLSMemoryBIOProtocol),2800,0.0.0.0] #011 return self._request('HEAD', path, headers=headers, **params)
Mar 29 23:28:54 dev1 soledad-server: [_GenericHTTPChannelProtocol (TLSMemoryBIOProtocol),2800,0.0.0.0] #011 File "/usr/lib/python2.7/dist-packages/couchdb/http.py", line 574, in _request
Mar 29 23:28:54 dev1 soledad-server: [_GenericHTTPChannelProtocol (TLSMemoryBIOProtocol),2800,0.0.0.0] #011 credentials=self.credentials)
Mar 29 23:28:54 dev1 soledad-server: [_GenericHTTPChannelProtocol (TLSMemoryBIOProtocol),2800,0.0.0.0] #011 File "/usr/lib/python2.7/dist-packages/couchdb/http.py", line 307, in request
Mar 29 23:28:54 dev1 soledad-server: [_GenericHTTPChannelProtocol (TLSMemoryBIOProtocol),2800,0.0.0.0] #011 conn = self.connection_pool.get(url)
Mar 29 23:28:54 dev1 soledad-server: [_GenericHTTPChannelProtocol (TLSMemoryBIOProtocol),2800,0.0.0.0] #011 File "/usr/lib/python2.7/dist-packages/couchdb/http.py", line 502, in get
Mar 29 23:28:54 dev1 soledad-server: [_GenericHTTPChannelProtocol (TLSMemoryBIOProtocol),2800,0.0.0.0] #011 conn.connect()
Mar 29 23:28:54 dev1 soledad-server: [_GenericHTTPChannelProtocol (TLSMemoryBIOProtocol),2800,0.0.0.0] #011 File "/usr/lib/python2.7/httplib.py", line 822, in connect
Mar 29 23:28:54 dev1 soledad-server: [_GenericHTTPChannelProtocol (TLSMemoryBIOProtocol),2800,0.0.0.0] #011 self.timeout, self.source_address)
Mar 29 23:28:54 dev1 soledad-server: [_GenericHTTPChannelProtocol (TLSMemoryBIOProtocol),2800,0.0.0.0] #011 File "/usr/lib/python2.7/socket.py", line 571, in create_connection
Mar 29 23:28:54 dev1 soledad-server: [_GenericHTTPChannelProtocol (TLSMemoryBIOProtocol),2800,0.0.0.0] #011 raise err
Mar 29 23:28:54 dev1 soledad-server: [_GenericHTTPChannelProtocol (TLSMemoryBIOProtocol),2800,0.0.0.0] #011socket.error: [Errno 111] Connection refused
Mar 29 23:28:54 dev1 soledad-server: [-] 0.0.0.0 - - [29/Mar/2017:23:28:53 +0000] "GET /user-024e8a3d57227f6821e92d30a736d778/sync-from/12435514b302448f84403ab197e7822a HTTP/1.1" 500 110 "-" "-"