automated restore (#40 pt 2)

context

part 2 of epic described in #40 (closed) and discussed at lenght in !158 (closed) .

dependent (?) on prior completion of #193 (closed)

behavior

• there is an ansible script that can be run with make ansible.restore that:
  • reads an sb_backup host from the inventory
  • runs an scp script that pulls backups from sb_backup to (new) signalboost host using -i /home/sb_user/.ssh/id_sb_user (which assumes that pubkey & secrete key must be on signalboost and that pubkey must be in allowed_keys on sb_backup)
  • runs a restore job on signalboost (which must run after provision and deploy) that restores the keystore volume and runs pg_restore on the db backup
  • optionally: deletes signalboost files from any borked machine (?)

optional

complete the "ansible requirements" section of #40 (closed) (or split into new card if that helps!)

ansible requirements

• some way of putting inventory (hence what backup and prod hosts and what users and keys are) under version control, without blasting away the ansible_user etc. values that any given dev might be using. (maybe put inventory.tmpl.gpg under version control?
• pub key for sb_user must go on sb_backup host
• sb_user and its pub/priv ssh keys must go on all prod instances
• gpg keys should be imported into keyring as part of provision
• cron job for running backup script must be put into all prod instances as part of provision.yml