WIP: Add initial restic support.

I've added a handler and example config file for restic. Since I'll be deploying this via Ansible, I'm not intending to write a helper, etc. But, I figured I'd share what basic progress I have made.

this seems to be the eternal problem of "what files should i include/exclude"... you're definitely missing some stuff here. for example, you're missing all of /var, which includes the critical /var/lib/mysql. i would just shove /var and /usr in there, at least. i also notice you're skipping /srv - for a lot of deployments i have done, that would be catastrophic because that's where the prod data resides.

my strategy, in general, is to include all the mountpoints i'm interested in, then make sure i use --one-filesystem (or whatever the equivalent is in my tool) to skip shit like /proc or /dev. then i pick a few narrow directories i want to skip. keep in mind backups like restic or borg are really efficient and fast. it's not a problem to backup a bunch of redundant shit - it's really useful to have a full backup so that you can restore the whole machine in one shot ("bare metal recovery", iirc), as opposed to having to reinstall the machine, all packages and everything. i know you use a lot of configuration management, but not everyone does, and CM is not backups: you may be forgetting something in that CM system that you can't recover directly.

here's my borg commandline for my home backups:

  borg create --one-file-system                  \
      --verbose                                  \
      --stats                                    \
      $REPOSITORY::$(hostname)-$(date +%Y-%m-%d)$tag \
      / /boot /usr /var /home                    \
      --exclude-caches                           \
      --exclude "/home/*/.cache"                 \
      --exclude "*/.Trash-*"                     \
      --exclude "*/[Cc]ache/*"                   \
      --exclude "*.vmdk"                         \
      --exclude "/tmp/*"                         \
      --exclude "*/build-area/*"                 \
      --exclude "/var/cache/*"                   \
      --exclude "/var/tmp/*"                     \
      --exclude "/var/log/*"

--exclude-caches is interesting: it's a "standard" way to skip "cache" directories (those that contain a CACHEDIR.TAG file, see this spec), i don't know if restic has something like this... otherwise, notice how i just backup everything except temporary files, cache files, some VM images, trash and logfiles.

Note that these are just the defaults, in case no include directives appear in the config file. I'm fine with altering the default settings, but these were copied from the tar handler, for consistency.

i see... maybe it's better to follow those defaults then... that said, tar is a very different beast than restic: much less efficient... so i think it's worth having different defaults.

there's something to be said about factoring out exclude patterns so that there are sane, ninja-wide defaults, but that's out of scope here...

Ok. I just committed the first step, supporting the --one-file-system option.

Interesting side-effects of this include:

1. expansion of the "exclude" parameters, such that it call backup with, for example, each subdir under /tmp.
2. local repos appear to attempt to recurse into themselves.

(1) should be addressed by quoting the options parameters as they're built. For (2), I guess I can add the repo path, if it's local, to the exclude list.

I've updated the includes and excludes, as well as how excludes are passed to the backupcommand.

this will fail if there are spaces in any of those options.

Presumably we'd need to fix this at the point where we're concatenating the options. I guess we'd need to add escaped quotes, or something?

you need "$repo", for example, but yes, in the case of include/exclude, it's at the concatenation time it needs to happen and explicitly not here.

As it turns out quoting the backup targets causes restic to fail to find the relevant file-paths. I think spaces in file-paths will have to be handled by escaping the space, as you would normally (i.e., when calling the command directly).

added 1 commit

• ae49b15f - Be explicit about password file failures.

Compare with previous version

added 1 commit

• 8272e549 - Be more generous with backup retention.

Compare with previous version

added 1 commit

• 97153e44 - Remove duplication in setting 'keep' parameters.

Compare with previous version

added 1 commit

• 6a04f4be - Support restic's --one-file-system option.

Compare with previous version

added 3 commits

• ded35b56 - Fix restic excludes.
• ff772420 - Improve default restic includes and excludes.
• 8f7ded15 - Fix whitespace.

Compare with previous version

Chatting with @anarcat on IRC, it became clear that using a --password-file didn't add any more security, versus just adding it to the config file directly. Since S3 and similar backends require keys to be set in the environment, we figured to also add these to the config file, and export them to the environment within the handler.

added 1 commit

• bda7cffe - Move credentials to config file.

Compare with previous version

added 1 commit

• eb05bcd1 - Fix typo.

Compare with previous version

added Handler: new label

Hello, here is what i did based on the work here.

restic handler

I've programmed a backup for tonight. I will let you know if it works correctly. I use the OpenStack Swift backend, so i will not be able to test the other ones.

Just tested manually (i didn't know how to do it 10 minutes ago) and it works.

Jun 05 16:46:51 Info: >>>> starting action /etc/backup.d/backup1-etc.restic (because current time matches everyday at 16:47)
Jun 05 16:46:51 Info: Taking backup snapshot.
Jun 05 16:46:59 Info: Removing old snapshots based on defined retention policy.
Jun 05 16:47:02 Info: <<<< finished action /etc/backup.d/backup1-etc.restic: SUCCESS
Jun 05 16:47:02 Info: FINISHED: 1 actions run. 0 fatal. 0 error. 0 warning.

Strange, on another server it does not work. FIXED

Info: >>>> starting action /etc/backup.d/etc_weekdays.restic (because current time matches everyday at 09)
Debug: yes
Debug: executing handler in locked section controlled by /var/lock/backupninja/_etc_backup.d_etc_weekdays.restic
Debug: The restic repository is: swift:backup:/ms2
Debug: The restic password is set.
Debug: The OpenStack Auth URL is: https://auth.cloud.ovh.net/v2.0/
Debug: The OpenStack tenant ID is: ***
Debug: The OpenStack tenant name is: ***
Debug: The OpenStack username is: ***
Debug: The OpenStack password is set.
Debug: These files and directories will be included in backups: /etc
Debug: Files matching the following patterns will be excluded from backups: /dev /lost+found /media /mnt /proc /run /sys /tmp /var/cache /var/lock /var/spool /var/run /var/tmp
Debug: The following additional flags will be applied to backups: --one-file-system
Debug: No tags defined.
Info: Taking backup snapshot.
Debug: Running: restic backup --one-file-system  /etc --exclude=/dev  --exclude=/lost+found  --exclude=/media  --exclude=/mnt  --exclude=/proc  --exclude=/run  --exclude=/sys  --exclude=/tmp  --exclude=/var/cache  --exclude=/var/lock  --exclude=/var/spool  --exclude=/var/run  --exclude=/var/tmp
Fatal: unable to open config file: conn.Object: Object Not Found
Is there a repository at the following location?
swift:backup:/ms2
Fatal: Restic backup failed.
Fatal: <<<< finished action /etc/backup.d/etc_weekdays.restic: FAILED

I have foked the project, and here is the current state of my restic handler: https://0xacab.org/nka/backupninja/blob/restic/handlers/restic