logwatch: How to deal with couchdb not accessible
couchdb was down for ~20 secs:
[Sat, 26 Nov 2016 06:39:39 GMT] [info] [<0.2390.89>] 127.0.0.1 - - HEAD /user-4ba8cba7ad29d6ca53f9c0da5d4ebca5 200 [Sat, 26 Nov 2016 06:40:03 GMT] [info] [<0.104.0>] 127.0.0.1 - - GET / 200 [Sat, 26 Nov 2016 06:40:03 GMT] [info] [<0.103.0>] 127.0.0.1 - - GET / 200 [Sat, 26 Nov 2016 06:40:03 GMT] [info] [<0.32.0>] Apache CouchDB has started on http://127.0.0.1:5984/
webapp throws an HTTP status code 503:
. Nov 26 06:39:40 dev1 webapp[19563]: Started DELETE "/1/users/4ba8cba7ad29d6ca53f9c0da5d9ca503.json?identities=destroy" for 0.0.0.0 at 2016-11-26 06:39:40 +0000 . Nov 26 06:39:40 dev1 webapp[19563]: Processing by Api::UsersController#destroy as JSON . Nov 26 06:39:40 dev1 webapp[19563]: Parameters: {"identities"=>"destroy", "version"=>"1", "id"=>"4ba8cba7ad29d6ca53f9c0da5d9ca503"} . Nov 26 06:39:40 dev1 webapp[19563]: HTTP status code 503 . Nov 26 06:39:40 dev1 webapp[19563]: /srv/leap/webapp/vendor/bundle/ruby/2.1.0/gems/couchrest-2.0.0/lib/couchrest/connection.rb:241:in `raise_response_error'#012/srv/leap/webapp/vendor/bundle/ruby/2.1.0/gems/couchrest-2.0.0/lib/couchrest/connection.rb:196:in `handle_response_code'#012/srv/leap/webapp/vendor/bundle/ruby/2.1.0/gems/couchrest-2.0.0/lib/couchrest/connection.rb:185:in `send_and_parse_response'#012/srv/leap/webapp/vendor/bundle/ruby/2.1.0/gems/couchrest-2.0.0/lib/couchrest/connection.rb:170:in `execute'#012/srv/leap/webapp/vendor/bundle/ruby/2.1.0/gems/couchrest-2.0.0/lib/couchrest/connection.rb:68:in `get'#012/srv/leap/webapp/vendor/bundle/ruby/2.1.0/gems/couchrest-2.0.0/lib/couchrest/database.rb:98:in `get!'#012/srv/leap/webapp/vendor/bundle/ruby/2.1.0/gems/couchrest-2.0.0/lib/couchrest/database.rb:112:in `get'#012/srv/leap/webapp/vendor/bundle/ruby/2.1.0/gems/couchrest_model-0.0.0.0c1/lib/couchrest/model/document_queries.rb:56:in `get!'#012/srv/leap/webapp/vendor/bundle/ruby/2.1.0/gems/couchrest_model-0.0.0.0c1/lib/couchrest/model/document_queries.rb:36:in `get'#012/srv/leap/webapp/app/models/token.rb:30:in `find_by_token'#012/srv/leap/webapp/app/controllers/controller_extension/token_authentication.rb:8:in `block in token'#012/srv/leap/webapp/vendor/bundle/ruby/2.1.0/gems/actionpack-0.0.0.0/lib/action_controller/metal/http_authentication.rb:439:in `call'#012/srv/leap/webapp/vendor/bundle/ruby/2.1.0/gems/actionpack-0.0.0.0/lib/action_controller/metal/http_authentication.rb:439:in `authenticate'#012/srv/leap/webapp/vendor/bundle/ruby/2.1.0/gems/actionpack-0.0.0.0/lib/action_controller/metal/http_authentication.rb:414:in `authenticate_with_http_token'#012/srv/leap/webapp/app/controllers/controller_extension/token_authentication.rb:7:in `token'#012/srv/leap/webapp/app/controllers/controller_extension/token_authentication.rb:13:in `token_authenticate'#012/srv/leap/webapp/vendor/bundle/ruby/2.1.0/gems/activesupport-0.0.0.0/lib/active_support/callbacks.rb:432:in `block in make_lambda'#012/srv/leap/webapp C Nov 26 06:39:40 dev1 webapp[19563]: Completed 500 Internal Server Error in 97ms (Views: 0.6ms)
soledad logs a Traceback failing to create a db:
Nov 26 06:39:26 dev1 soledad-server: [-] 0.0.0.0 - - [26/Nov/2016:06:39:26 +0000] "GET /user-4ba8cba7ad29d6ca53f9c0da5d9ca503/sync-from/1c154e5f5cdf41aebb8fff25864d213e HTTP/1.1" 404 38 "-" "-" Nov 26 06:39:40 dev1 soledad-server: [-] Nov 26 06:39:40 dev1 soledad-server: [-] #011 Error while creating database (user-4ba8cba7ad29d6ca53f9c0da5d9ca503) with (sudo -u soledad-admin /usr/bin/create-user-db) command. Nov 26 06:39:40 dev1 soledad-server: [-] #011 Output: Traceback (most recent call last): Nov 26 06:39:40 dev1 soledad-server: [-] #011 File "/usr/bin/create-user-db", line 96, in Nov 26 06:39:40 dev1 soledad-server: [-] #011 ensure_database(args.dbname) Nov 26 06:39:40 dev1 soledad-server: [-] #011 File "/usr/bin/create-user-db", line 84, in ensure_database Nov 26 06:39:40 dev1 soledad-server: [-] #011 database_security=db_security) Nov 26 06:39:40 dev1 soledad-server: [-] #011 File "/usr/lib/python2.7/dist-packages/leap/soledad/common/couch/__init__.py", line 145, in open_database Nov 26 06:39:40 dev1 soledad-server: [-] #011 if dbname not in server: Nov 26 06:39:40 dev1 soledad-server: [-] #011 File "/usr/lib/python2.7/dist-packages/couchdb/client.py", line 94, in __contains__ Nov 26 06:39:40 dev1 soledad-server: [-] #011 self.resource.head(name) Nov 26 06:39:40 dev1 soledad-server: [-] #011 File "/usr/lib/python2.7/dist-packages/couchdb/http.py", line 542, in head Nov 26 06:39:40 dev1 soledad-server: [-] #011 return self._request('HEAD', path, headers=headers, **params) Nov 26 06:39:40 dev1 soledad-server: [-] #011 File "/usr/lib/python2.7/dist-packages/couchdb/http.py", line 574, in _request Nov 26 06:39:40 dev1 soledad-server: [-] #011 credentials=self.credentials) Nov 26 06:39:40 dev1 soledad-server: [-] #011 File "/usr/lib/python2.7/dist-packages/couchdb/http.py", line 307, in request Nov 26 06:39:40 dev1 soledad-server: [-] #011 conn = self.connection_pool.get(url) Nov 26 06:39:40 dev1 soledad-server: [-] #011 File "/usr/lib/python2.7/dist-packages/couchdb/http.py", line 502, in get Nov 26 06:39:40 dev1 soledad-server: [-] #011 conn.connect() Nov 26 06:39:40 dev1 soledad-server: [-] #011 File "/usr/lib/python2.7/httplib.py", line 822, in connect Nov 26 06:39:40 dev1 soledad-server: [-] #011 self.timeout, self.source_address) Nov 26 06:39:40 dev1 soledad-server: [-] #011 File "/usr/lib/python2.7/socket.py", line 571, in create_connection Nov 26 06:39:40 dev1 soledad-server: [-] #011 raise err Nov 26 06:39:40 dev1 soledad-server: [-] #011socket.error: [Errno 111] Connection refused Nov 26 06:39:40 dev1 soledad-server: [-] Nov 26 06:39:40 dev1 soledad-server: [-] #011 Exit code: 1 Nov 26 06:39:40 dev1 soledad-server: [-] Nov 26 06:39:40 dev1 soledad-server: [-] 0.0.0.0 - - [26/Nov/2016:06:39:40 +0000] "POST /user-4ba8cba7ad29d6ca53f9c0da5d9ca503/sync-from/1c154e5f5cdf41aebb8fff25864d213e HTTP/1.1" 401 27 "-" "-"
We configure logwatch to alert the admin on any error. When the admin deliberately stops couch and forgets about stopping the webapp/soledad, both will freak out in the logs.
I'm really unsure how to best deal with this. I'd like to discuss on the next sysdev meeting.
(from redmine: created on 2016-11-29)