Soledad-server fails to start due to Missing couch config document in database
On clam, Soledad-server fails to start due to a missing couch config document in database.
root@clam:~# tail /var/log/syslog
Jan 26 12:15:06 clam soledad-server: [-] set uid/gid 999/999
Jan 26 12:15:06 clam soledad-server[28580]: .
Jan 26 12:15:06 clam systemd[1]: Started LSB: Start soledad daemon at boot time.
Jan 26 12:15:06 clam soledad-server: [leap.soledad.common.couch.state#error] Missing couch config document in database user-3356da2eeb2396b90fb4a2afc5e45288
Jan 26 12:15:06 clam soledad-server: [-] (TLS Port 2323 Closed)
Jan 26 12:15:06 clam soledad-server: [-] Stopping factory <twisted.web.server.Site instance at 0x7f5cd7c859e0>
Jan 26 12:15:06 clam soledad-server: [-] Main loop terminated.
Jan 26 12:15:06 clam soledad-server: [-] Warning: No permission to delete pid file
Jan 26 12:15:06 clam soledad-server: [-] Server Shut Down.
Jan 26 12:17:01 clam CRON[28918]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
but the user/identities entry to this db doesn't exist anymore:
root@clam:~# curl --netrc 127.0.0.1:5984/user-3356da2eeb2396b90fb4a2afc5e45288
{"db_name":"user-3356da2eeb2396b90fb4a2afc5e45288","doc_count":6,"doc_del_count":0,"update_seq":13,"purge_seq":0,"compact_running":false,"disk_size":24679,"data_size":4995,"instance_start_time":"1485413147127833","disk_format_version":6,"committed_update_seq":13}
root@clam:~# curl -sS --netrc 127.0.0.1:5984/users/_all_docs/?include_docs=true | grep 3356da2eeb2396b90fb4a2afc5e45288
root@clam:~# curl -sS --netrc 127.0.0.1:5984/identities/_all_docs/?include_docs=true | grep 3356da2eeb2396b90fb4a2afc5e45288
After cleaning up old stale user-dbs like it is done by cron once a day it can get started again:
(/bin/date; /srv/leap/couchdb/scripts/cleanup-user-dbs) >> /var/log/leap/couchdb-cleanup.log
systemctl start soledad-server
So we should reconsider if it's sufficient to remove old stale user-dbs once a day.