automated restore (#40 pt 2)

context

part 2 of epic described in #40 (closed) and discussed at lenght in !158 (closed) .

dependent (?) on prior completion of #193 (closed)

there is an ansible script that can be run with make ansible.restore that:
- reads an sb_backup host from the inventory
- runs an scp script that pulls backups from sb_backup to (new) signalboost host using -i /home/sb_user/.ssh/id_sb_user (which assumes that pubkey & secrete key must be on signalboost and that pubkey must be in allowed_keys on sb_backup)
- runs a restore job on signalboost (which must run after provision and deploy) that restores the keystore volume and runs pg_restore on the db backup
- optionally: deletes signalboost files from any borked machine (?)

complete the "ansible requirements" section of #40 (closed) (or split into new card if that helps!)

some way of putting inventory (hence what backup and prod hosts and what users and keys are) under version control, without blasting away the ansible_user etc. values that any given dev might be using. (maybe put inventory.tmpl.gpg under version control?
pub key for sb_user must go on sb_backup host
sb_user and its pub/priv ssh keys must go on all prod instances
gpg keys should be imported into keyring as part of provision
cron job for running backup script must be put into all prod instances as part of provision.yml

Edited Jan 13, 2020 by aguestuser