Skip to content
Snippets Groups Projects

Compare revisions

Changes are shown as if the source revision was being merged into the target revision. Learn more about comparing revisions.

Source

Select target project
No results found

Target

Select target project
  • liberate/backupninja
  • Benzhaomin/backupninja
  • ergonlogic/backupninja
  • louis/backupninja
  • guido/backupninja
  • ibauer/backupninja
  • romain/backupninja
  • gsubiron/backupninja
  • davidkg/backupninja
  • fkrauthan/backupninja
  • Glandos/backupninja
  • lyz/backupninja
  • nosmo/backupninja
  • orel/backupninja
  • raabf/backupninja
  • wu-lee/backupninja
  • huthamcau/backupninja
  • julien/backupninja
  • sensespidey/backupninja
  • LeLutin/backupninja
  • raT/backupninja
  • petrklima/backupninja
  • fancsali/backupninja
  • ko7ashiV/backupninja
  • yova/backupninja
  • jipem/backupninja
  • debian-janitor/backupninja
  • phlummox/backupninja
  • e1k/backupninja
  • jonhattan_/backupninja
  • illuusio/backupninja
  • maethor/backupninja
32 results
Show changes
Commits on Source (655)
......@@ -20,6 +20,8 @@ etc/cron.d/backupninja
etc/logrotate.d/backupninja
examples/Makefile
handlers/Makefile
handlers/borg
handlers/borg.helper
handlers/dup
handlers/dup.helper
handlers/ldap
......@@ -33,18 +35,24 @@ handlers/pgsql
handlers/pgsql.helper
handlers/rdiff
handlers/rdiff.helper
handlers/restic
handlers/restic.helper
handlers/rsync
handlers/sh
handlers/svn
handlers/sys
handlers/sys.helper
handlers/tar
handlers/tar.helper
handlers/trac
lib/Makefile
lib/easydialog
lib/parseini
lib/tools
lib/vserver
man/Makefile
src/Makefile
src/backupninja
src/ninjahelper
.vagrant
*.swp
*.swo
BACKUPNINJA was written by the Riseup Collective: intellectual property is theft.
Ninjas:
Ninjas:
elijah@riseup.net -- original code, bug fixes, man pages
micah@riseup.net -- debian package, vserver support, bug fixes
stefani@riseup.net -- makecd handler, man pages
intrigeri@boum.org -- dup handler, pgsql handler, vserver support, bug fixes
Charles Lepple -- trac handler
Petr Klma <petr.klima@madeta-group.cz> -- autotools, RPM support and sys checks
Petr Klma <petr.klima@madeta-group.cz> -- autotools, RPM support and sys checks
paulv@bikkel.org -- rsnap handler
Robert Napier -- improved RPM build
rhatto -- rub handler and patches
Patches:
Patches:
cmccallum@thecsl.org
Daniel.Bonniot@inria.fr -- mysql ignores and nodata
......@@ -21,7 +21,7 @@ garcondumonde@riseup.net
Martin Krafft madduck@debian.org -- admingroup patch
Anarcat -- lotsa patches
Jamie McClelland -- cstream patches
ale -- ldap cleanup
ale -- ldap cleanup, rdiff support for reading include/exclude from files
Sami Haahtinen <ressu@ressukka.net>
Matthew Palmer -- mysql enhancements
romain.tartiere@healthgrid.org -- ldap fixes
......@@ -29,4 +29,44 @@ Adam Monsen - spec file updates
Matthew Palmer <mpalmer@debian.org> -- halt loglevel feature
dan@garthwaite.org -- reportspace bugfix
Tuomas Jormola <tj@solitudo.net> -- "when = manual" option
Ian Beckwith <ianb@erislabs.net> -- dup bandwidthlimit fix
Ian Beckwith <ianb@erislabs.net> -- dup bandwidthlimit fix, dup helper fix
Olivier Berger <oberger@ouvaton.org> -- numerous contributions
stefan <s.freudenberg@jpberlin.de> -- dup support for Amazon S3 buckets
maniacmartin <martin@maniacmartin.com> -- rdiff confusing error message fix
Chris Nolan <chris@cenolan.com> -- maildir subdirectory expansion
Dan Carley -- mysql bugfix
Jordi Mallach <jordi@debian.org> -- do not error when no jobs are configured
Jacob Anawalt <jlanawalt@gmail.com> -- pg_dump format option
Sergio Talens-Oliag <sto@debian.org> -- pipefail fixes
Bruno Bigras <bigras.bruno@gmail.com> -- enable tar handler in the build system
aihtdikh -- Allow 'when = XXX' with spaces in .sh files.
Chris Lamb <lamby@debian.org> -- rdiff.helper bugfix
Yuval Kogman <nothingmuch@woobling.org> -- RackSpace's CloudFiles support for duplicity
exobuzz - mysql bugfixes
Glennie Vignarajah <glennie@glennie.fr> -- mysql bugfix
ddpaul <paul@reic.ru> -- rsync bugfix
ulrich -- duplicity bugfix preliminary patch
David Gasaway <dave@gasaway.org> -- rdiff's output_as_info option
Pierre ROUDIER <contact@pierreroudier.net> -- xz and test mode for tar handler
olb <olb@nebkha.net> -- update of duplicity/paramiko SSH options handling
Alexander Mette <mail@amette.eu> -- duplicity bugfix
Dominik George <nik@naturalnet.de> -- Support using a different passphrase for the signing key from the one used for the encryption key in the dup handler
Christian Prause <cprause@suse.com> -- Support suse in the sys handler
Jools Wills <jools@oxfordinspire.co.uk> -- Bugfix in the sys helper, indentation fixes
Mark Janssen <mark@sig-io.nl> -- ignore jobs whose filename ends with "~"
shred <riseup@ml.shredzone.de> -- Initial patch for test mode support in the rsync handler
Daniel Lo Nigro <daniel@dan.cx> -- Dropbox support for Duplicity
Matthijs Wensveen <matthijs.wensveen@gmail.com> -- fix symmetric encryption in dup handler
ulrich <ulrich@habmalnefrage.de> -- Added validation check for when
Romain Dessort <romain@univers-libre.net> -- Fix list of devices when dumping partition tables
Guillaume Subiron <guillaume@sysnove.fr> -- borg handler
Jerome Charaoui <jerome@riseup.net> -- borg handler
David Gasaway <dave@gasaway.org> -- Fixes for configuration files without suffix
Hugh Nowlan <nosmo@nosmo.me> -- dup check for archive dir
Lyz <lyz@riseup.net> -- sys support for LUKS in disk partitions
Glandos <bugs-0xacab@antipoul.fr> -- sys excludes zram devices
Nicolas Karolak <nicolas@karolak.fr> -- Add restic support
Derek Laventure -- Add restic helper
Colan Schwartz -- Fix restic options handler
... and other contributors, thank you!
This diff is collapsed.
This diff is collapsed.
Q: duplicity works fine when run standalone, but complains about gpg
"public key not found" when run from backupninja
A: We bet you're using sudo to run both duplicity and backupninja, and have been
using sudo as well when generating the GnuPG key pair used by duplicity.
Quick fix: generate a new GnuPG key pair in a root shell, or using
"sudo -H" instead of plain sudo.
Another solution: import the GnuPG keypair into the root user's keyring, taking
care of running "gpg --update-trustdb" in a root shell or using "sudo -H"
afterwards, in order to tag this keypair as "ultimately trusted".
Detailed explanation: sudo does not change $HOME by default, so GnuPG saved the
newly generated key pair to your own keyring, rather than to the root user's
keyring. Running "sudo duplicity" hides the problem, as it uses your own
keyring. Running "sudo backupninja" reveals the problem, as backupninja uses
"su" to make sure it runs duplicity in a real root environment, i.e. using the
root user's GnuPG keyring.
duplicity works fine when run standalone, but complains about gpg "public key not found" when run from backupninja
==================================================================================================================
We bet you're using sudo to run both duplicity and backupninja, and have been
using sudo as well when generating the GnuPG key pair used by duplicity.
Quick fix: generate a new GnuPG key pair in a root shell, or using
`sudo -H` instead of plain sudo.
Another solution: import the GnuPG keypair into the root user's keyring, taking
care of running `gpg --update-trustdb` in a root shell or using `sudo -H`
afterwards, in order to tag this keypair as "ultimately trusted".
Detailed explanation: sudo does not change `$HOME` by default, so GnuPG saved the
newly generated key pair to your own keyring, rather than to the root user's
keyring. Running `sudo duplicity` hides the problem, as it uses your own
keyring. Running `sudo backupninja` reveals the problem, as backupninja uses
`su` to make sure it runs duplicity in a real root environment, i.e. using the
root user's GnuPG keyring.
What should I do when rdiff-backup fails?
=========================================
If rdiff-backup fails, the meta data file may get corrupt. When this
happens, rdiff-backup will complain loudly every time it is run and
possibly fail to backup some or all the files.
To force rdiff-backup to rebuild the meta data, set this option in
the `.rdiff` backup action file:
options = --force
After a rdiff-backup run has been successful you should remove
this option.
How to restrict privileges on the backup server?
================================================
backupninja uses a "push" mechanism, where backups are sent from one
or several hosts to a centralized backup server.
Mount your backup partition with limited execution rights
---------------------------------------------------------
Edit `/etc/fstab` to mount your partition with limited rights. For example:
/home ext3 defaults,nosuid,noexec,nodev 0 2
Create a user for each client
-----------------------------
On the backup server, it is important to create a separate user for
each client.
Use a restricted shell and jail users
-------------------------------------
Furthermore, you may use a restricted shell like
[rssh](http://www.pizzashack.org/rssh/index.shtml) or
[scponly](http://sublimation.org/scponly/wiki/index.php/Main_Page),
which also offer the ability to jail connections.
On the backup server:
$ apt-get install scponly
$ adduser --disabled-password --home /home/backup/ninja-host1 --shell /usr/bin/scponly ninja-host1
You may now use `ninja-host1` user to connect to the
`/home/backup/ninja-host1` jail.
Installation Instructions
*************************
Copyright (C) 1994, 1995, 1996, 1999, 2000, 2001, 2002, 2004 Free
Software Foundation, Inc.
This file is free documentation; the Free Software Foundation gives
unlimited permission to copy, distribute and modify it.
Basic Installation
==================
These are generic installation instructions.
The `configure' shell script attempts to guess correct values for
various system-dependent variables used during compilation. It uses
those values to create a `Makefile' in each directory of the package.
It may also create one or more `.h' files containing system-dependent
definitions. Finally, it creates a shell script `config.status' that
you can run in the future to recreate the current configuration, and a
file `config.log' containing compiler output (useful mainly for
debugging `configure').
It can also use an optional file (typically called `config.cache'
and enabled with `--cache-file=config.cache' or simply `-C') that saves
the results of its tests to speed up reconfiguring. (Caching is
disabled by default to prevent problems with accidental use of stale
cache files.)
If you need to do unusual things to compile the package, please try
to figure out how `configure' could check whether to do them, and mail
diffs or instructions to the address given in the `README' so they can
be considered for the next release. If you are using the cache, and at
some point `config.cache' contains results you don't want to keep, you
may remove or edit it.
The file `configure.ac' (or `configure.in') is used to create
`configure' by a program called `autoconf'. You only need
`configure.ac' if you want to change it or regenerate `configure' using
a newer version of `autoconf'.
The simplest way to compile this package is:
1. `cd' to the directory containing the package's source code and type
`./configure' to configure the package for your system. If you're
using `csh' on an old version of System V, you might need to type
`sh ./configure' instead to prevent `csh' from trying to execute
`configure' itself.
Running `configure' takes awhile. While running, it prints some
messages telling which features it is checking for.
2. Type `make' to compile the package.
3. Optionally, type `make check' to run any self-tests that come with
the package.
4. Type `make install' to install the programs and any data files and
documentation.
5. You can remove the program binaries and object files from the
source code directory by typing `make clean'. To also remove the
files that `configure' created (so you can compile the package for
a different kind of computer), type `make distclean'. There is
also a `make maintainer-clean' target, but that is intended mainly
for the package's developers. If you use it, you may have to get
all sorts of other programs in order to regenerate files that came
with the distribution.
Compilers and Options
=====================
Some systems require unusual options for compilation or linking that the
`configure' script does not know about. Run `./configure --help' for
details on some of the pertinent environment variables.
You can give `configure' initial values for configuration parameters
by setting variables in the command line or in the environment. Here
is an example:
./configure CC=c89 CFLAGS=-O2 LIBS=-lposix
*Note Defining Variables::, for more details.
Compiling For Multiple Architectures
====================================
You can compile the package for more than one kind of computer at the
same time, by placing the object files for each architecture in their
own directory. To do this, you must use a version of `make' that
supports the `VPATH' variable, such as GNU `make'. `cd' to the
directory where you want the object files and executables to go and run
the `configure' script. `configure' automatically checks for the
source code in the directory that `configure' is in and in `..'.
If you have to use a `make' that does not support the `VPATH'
variable, you have to compile the package for one architecture at a
time in the source code directory. After you have installed the
package for one architecture, use `make distclean' before reconfiguring
for another architecture.
Installation Names
==================
By default, `make install' will install the package's files in
`/usr/local/bin', `/usr/local/man', etc. You can specify an
installation prefix other than `/usr/local' by giving `configure' the
option `--prefix=PREFIX'.
You can specify separate installation prefixes for
architecture-specific files and architecture-independent files. If you
give `configure' the option `--exec-prefix=PREFIX', the package will
use PREFIX as the prefix for installing programs and libraries.
Documentation and other data files will still use the regular prefix.
In addition, if you use an unusual directory layout you can give
options like `--bindir=DIR' to specify different values for particular
kinds of files. Run `configure --help' for a list of the directories
you can set and what kinds of files go in them.
If the package supports it, you can cause programs to be installed
with an extra prefix or suffix on their names by giving `configure' the
option `--program-prefix=PREFIX' or `--program-suffix=SUFFIX'.
Optional Features
=================
Some packages pay attention to `--enable-FEATURE' options to
`configure', where FEATURE indicates an optional part of the package.
They may also pay attention to `--with-PACKAGE' options, where PACKAGE
is something like `gnu-as' or `x' (for the X Window System). The
`README' should mention any `--enable-' and `--with-' options that the
package recognizes.
For packages that use the X Window System, `configure' can usually
find the X include and library files automatically, but if it doesn't,
you can use the `configure' options `--x-includes=DIR' and
`--x-libraries=DIR' to specify their locations.
Specifying the System Type
==========================
There may be some features `configure' cannot figure out automatically,
but needs to determine by the type of machine the package will run on.
Usually, assuming the package is built to be run on the _same_
architectures, `configure' can figure that out, but if it prints a
message saying it cannot guess the machine type, give it the
`--build=TYPE' option. TYPE can either be a short name for the system
type, such as `sun4', or a canonical name which has the form:
CPU-COMPANY-SYSTEM
where SYSTEM can have one of these forms:
OS KERNEL-OS
See the file `config.sub' for the possible values of each field. If
`config.sub' isn't included in this package, then this package doesn't
need to know the machine type.
If you are _building_ compiler tools for cross-compiling, you should
use the `--target=TYPE' option to select the type of system they will
produce code for.
If you want to _use_ a cross compiler, that generates code for a
platform different from the build platform, you should specify the
"host" platform (i.e., that on which the generated programs will
eventually be run) with `--host=TYPE'.
Sharing Defaults
================
If you want to set default values for `configure' scripts to share, you
can create a site shell script called `config.site' that gives default
values for variables like `CC', `cache_file', and `prefix'.
`configure' looks for `PREFIX/share/config.site' if it exists, then
`PREFIX/etc/config.site' if it exists. Or, you can set the
`CONFIG_SITE' environment variable to the location of the site script.
A warning: not all `configure' scripts look for a site script.
Defining Variables
==================
Variables not defined in a site shell script can be set in the
environment passed to `configure'. However, some packages may run
configure again during the build, and the customized values of these
variables may be lost. In order to avoid this problem, you should set
them in the `configure' command line, using `VAR=value'. For example:
./configure CC=/usr/local2/bin/gcc
will cause the specified gcc to be used as the C compiler (unless it is
overridden in the site shell script).
`configure' Invocation
======================
`configure' recognizes the following options to control how it operates.
`--help'
`-h'
Print a summary of the options to `configure', and exit.
`--version'
`-V'
Print the version of Autoconf used to generate the `configure'
script, and exit.
`--cache-file=FILE'
Enable the cache: use and save the results of the tests in FILE,
traditionally `config.cache'. FILE defaults to `/dev/null' to
disable caching.
`--config-cache'
`-C'
Alias for `--cache-file=config.cache'.
`--quiet'
`--silent'
`-q'
Do not print messages saying which checks are being made. To
suppress all normal output, redirect it to `/dev/null' (any error
messages will still be shown).
`--srcdir=DIR'
Look for the package's source code in directory DIR. Usually
`configure' can determine that directory automatically.
`configure' also accepts some other, not widely useful, options. Run
`configure --help' for more details.
Installation
============
On Debian, Ubuntu and derivatives
---------------------------------
Run `apt-get install backupninja`.
By hand
-------
Requirements:
bash gawk
Recommended:
borgbackup cryptsetup duplicity flashrom gzip hwinfo rdiff-backup restic rsync sfdisk
To install backupninja, simply do the following:
$ ./autogen.sh
$ ./configure
$ make
$ make install
You may wish to change the install locations, or other options. To find
the available possibilities, run `./configure --help`.
# vi: noexpandtab softtabstop=0
## Process this file with automake to produce Makefile.in
EXTRA_DIST = README COPYING AUTHORS INSTALL NEWS ChangeLog \
EXTRA_DIST = FAQ.md README.md COPYING AUTHORS INSTALL.md NEWS CHANGELOG.md \
backupninja.spec backupninja.spec.in autogen.sh
SUBDIRS = etc examples handlers lib man src
......
backupninja (0.9.8-1) UNRELEASED
* duplicity 0.6.01 and later defaults to using an archive (cache)
directory, which was previously opt-in. Starting with backupninja
0.9.8, the backupninja duplicity handler puts this cache into
/var/cache/backupninja/duplicity unless specified by the user with
the "options" setting the *.dup job.
When backups have been performed with backupninja older than 0.9.8 in
conjunction with duplicity 0.6.01 or later, e.g. when using Sid or
Squeeze at certain times of the Squeeze release cycle, cache files
were probably saved into /root/.cache/duplicity; one may want to
delete these files, or rather save bandwidth and just move the cache
directory to the new location:
mkdir -p /var/cache/backupninja
mv /root/.cache/duplicity /var/cache/backupninja/
It is probably desirable to exclude this cache directory from
duplicity backup sets to avoid some kind of reentrant backup problem.
backupninja (0.9.7-1) UNRELEASED
* mysql: output filenames to support shell meta-characters in
......@@ -19,7 +39,7 @@ backupninja (0.9.4-1) UNRELEASED
situations, with weird behavious, due to incompatibilities
between various readlink versions in this field. This has been made
clear eventually: globbing is fully supported again, whereas no
attempt is done to dereference symlinks anymore.
attempt is done to dereference symlinks anymore.
Please read the new /usr/share/doc/backupninja/examples/example.dup
or /usr/share/doc/backupninja/examples/example.rdiff file, and update
your own configuration files if needed.
......@@ -31,7 +51,7 @@ backupninja (0.9.4-1) UNRELEASED
sshoptions = -i /root/.ssh/id_dsa_duplicity
with:
sshoptions = -o IdentityFile=/root/.ssh/id_dsa_duplicity
backupninja (0.9.2-1) unstable; urgency=low
WARNING FOR DUPLICITY USERS
......
|\_
B A C K U P N I N J A /()/
`\|
a silent flower blossom death strike to lost data.
Backupninja allows you to coordinate system backup by dropping a few
simple configuration files into /etc/backup.d/. Most programs you
might use for making backups don't have their own configuration file
format. Backupninja provides a centralized way to configure and
coordinate many different backup utilities.
Features:
- easy to read ini style configuration files.
- you can drop in scripts to handle new types of backups.
- backup actions can be scheduled
- you can choose when status report emails are mailed to you
(always, on warning, on error, never).
- console-based wizard (ninjahelper) makes it easy to create
backup action configuration files.
- passwords are never sent via the command line to helper programs.
- works with Linux-Vservers (http://linux-vserver.org/)
Backup types:
- secure, remote, incremental filesytem backup (via rdiff-backup).
incremental data is compressed. permissions are retained even
with an unpriviledged backup user.
- backup of mysql databases (via mysqlhotcopy and mysqldump).
- backup of ldap databases (via slapcat and ldapsearch).
- basic system and hardware info
- encrypted remote backups (via duplicity).
- backup of subversion repositories.
The following options are available:
-h, --help This usage message
-d, --debug Run in debug mode, where all log messages are
output to the current shell.
-f, --conffile FILE Use FILE for the main configuration instead
of /etc/backupninja.conf
-t, --test Test run mode. This will test if the backup could run, without actually
preforming any backups. For example, it will attempt to authenticate
or test that ssh keys are set correctly.
-n, --now Perform actions now, instead of when they might be scheduled.
No output will be created unless also run with -d.
--run FILE Runs the specified action FILE (e.g. one of the /etc/backup.d/ files).
Also puts backupninja in debug mode.
CONFIGURATION FILES
===================
The general configuration file is /etc/backupninja.conf. In this file
you can set the log level and change the default directory locations.
You can force a different general configuration file with "backupninja
-f /path/to/conf".
To preform the actual backup, backupninja processes each configuration
file in /etc/backup.d according to the file's suffix:
.sh -- run this file as a shell script.
.rdiff -- filesystem backup (using rdiff-backup)
.dup -- filesystem backup (using duplicity)
.mysql -- backup mysql databases
.ldap -- backup ldap databases
.pgsql -- backup PostgreSQL databases
.sys -- general hardware, partition, and system reports.
.svn -- backup subversion repositories
.maildir -- incrementally backup maildirs (very specialized)
Support for additional configuration types can be added by dropping
bash scripts with the name of the suffix into /usr/share/backupninja.
The configuration files are processed in alphabetical order. However,
it is suggested that you name the config files in "sysvinit style."
For example:
00-disabled.ldap
10-runthisfirst.sh
20-runthisnext.mysql
90-runthislast.rdiff
Typically, you will put a '.rdiff' config file last, so that any
database dumps you make are included in the filesystem backup.
Configurations files with names beginning with 0 (zero) or ending with
.disabled (preferred method) are skipped.
Unless otherwise specified, the config file format is "ini style."
For example:
# this is a comment
[fishes]
fish = red
fish = blue
[fruit]
apple = yes
pear = no thanks \
i will not have a pear.
SCHEDULING
==========
By default, each configuration file is processed everyday at 01:00 (1
AM). This can be changed by specifying the 'when' option in a config
file.
For example:
when = sundays at 02:00
when = 30th at 22
when = 30 at 22:00
when = everyday at 01 <-- the default
when = Tuesday at 05:00
A configuration file will be processed at the time(s) specified by the
"when" option. If multiple "when" options are present, then they all
apply. If two configurations files are scheduled to run in the same
hour, then we fall back on the alphabetical ordering specified above.
If two configurations files are scheduled close to one another in
time, it is possible to have multiple copies of backupninja running if
the first instance is not finished before the next one starts.
Make sure that you put the "when" option before any sections in your
configuration file.
These values for 'when' are equivalent:
when = tuesday at 05:30
when = TUESDAYS at 05
These values for 'when' are invalid:
when = tuesday at 2am
when = tuesday at 2
when = tues at 02
REAL WORLD USAGE
================
Backupninja can be used to implement whatever backup strategy you
choose. It is intended, however, to be used like so:
(1) First, databases are safely copied or exported to /var/backups.
Typically, you cannot make a file backup of a database while it
is in use, hence the need to use special tools to make a safe copy
or export into /var/backups.
(2) Then, vital parts of the file system, including /var/backups, are
nightly pushed to a remote, off-site, hard disk (using
rdiff-backup). The local user is root, but the remote user is not
priviledged. Hopefully, the remote filesystem is encrypted.
There are many different backup strategies out there, including "pull
style", magnetic tape, rsync + hard links, etc. We believe that the
strategy outlined above is the way to go because: (1) hard disks are
very cheap these days, (2) pull style backups are no good, because then
the backup server must have root on the production server, and (3)
rdiff-backup is more space efficient and featureful than using rsync +
hard links.
SSH KEYS
========
In order for rdiff-backup to sync files over ssh unattended, you must
create ssh keys on the source server and copy the public key to the
remote user's authorized keys file. For example:
root@srchost# ssh-keygen -t dsa
root@srchost# ssh-copy-id -i /root/.ssh/id_dsa.pub backup@desthost
Now, you should be able to ssh from user 'root' on srchost to
user 'backup' on desthost without specifying a password.
Note: when prompted for a password by ssh-keygen, just leave it
blank by hitting return.
The included helper program "ninjahelper" will walk you through creating
an rdiff-backup configuration, and will set up the ssh keys for you.
INSTALLATION
============
Requirements:
apt-get install bash gawk
Recommended:
apt-get install rdiff-backup gzip hwinfo
Files:
/usr/sbin/backupninja -- main script
/etc/cron.d/backupninja -- runs main script nightly
/etc/logrotate.d/backupninja -- rotates backupninja.log
/etc/backup.d/ -- directory for configuration files
/etc/backupninja.conf -- general options
/usr/share/backupninja -- handler scripts which do the actual work
Installation:
There is no install script, but you just need to move files to the
correct locations. All files should be owned by root.
# tar xvzf backupninja.tar.gz
# cd backupninja
# mv backupninja /usr/sbin/backupninja
# mv ninjahelper /usr/sbin/ninjahelper
# mv etc/logrotate.d/backupninja /etc/logrotate.d/backupninja
# mv etc/cron.d/backupninja /etc/cron.d/backupninja
# mkdir /etc/backup.d/
# mv etc/backupninja.conf /etc/backupninja.conf
# mv handlers /usr/share/backupninja
VSERVERS
========
If you are using Linux-Vservers (http://linux-vserver.org/) there are some
special capabilities that different handlers have to make vserver
backups easier.
Set the variable "vservers" to be "yes" in /etc/backupninja.conf and see the
example configuration files for each handler to configure the vserver specific
variables.
Additional vserver variables that can be configured in /etc/backupninja.conf,
but they probably don't need to be changed:
VSERVERINFO (default: /usr/sbin/vserver-info)
VSERVER (default: /usr/sbin/vserver)
VROOTDIR (default: `$VSERVERINFO info SYSINFO |grep vserver-Rootdir | awk '{print $2}'`)
NINJAHELPER
===========
Ninjahelper is an additional script which will walk you through the process of
configuring backupninja. Ninjahelper has a menu driven curses based interface
(using dialog).
To add an additional 'wizard' to ninjahelper, follow these steps:
(1) to add a helper for the handler "blue", create the file
blue.helper in the directory where the handlers live.
(ie /usr/share/backupninja).
(2) next, you need to add your helper to the global HELPERS variable
and define the main function for your helper (the function name
is always <helper>_wizard). for example, blue.helper:
HELPERS="$HELPERS blue:description_of_this_helper"
blue_wizard() {
... do work here ...
}
(3) look at the existing helpers to see how they are written. Try to re-use
functions, such as the dialog functions that are defined in easydialog.sh,
or the vserver functions defined in lib/vserver.
(4) test, re-test, and test again. Try to break the helper by going backwards,
try to think like someone who has no idea how to configure your handler
would think, try to make your helper as simple as possible. Walk like a cat,
become your shadow, don't let your senses betray you.
|\_
B A C K U P N I N J A /()/
`\|
a silent flower blossom death strike to lost data.
Backupninja allows you to coordinate system backup by dropping a few
simple configuration files into `/etc/backup.d/`. Most programs you
might use for making backups don't have their own configuration file
format. Backupninja provides a centralized way to configure and
coordinate many different backup utilities.
Features
========
The key features of backupninja are:
- easy to read ini style configuration files
- you can drop in scripts to handle new types of backups
- backup actions can be scheduled
- you can choose when status report emails are mailed to you
(always, on warning, on error, never)
- console-based wizard (ninjahelper) makes it easy to create
backup action configuration files
- passwords are never sent via the command line to helper programs
The following backup types are supported:
- secure, remote, incremental filesystem backup (via rdiff-backup)
incremental data is compressed. permissions are retained even
with an unpriviledged backup user
- backup of mysql databases (via mysqlhotcopy and mysqldump)
- basic system and hardware info
- encrypted remote backups (via duplicity or borgbackup)
- backup of subversion repositories
Installation
============
See the [installation documentation](INSTALL.md).
Options
=======
The following options are available:
- `-h`, `--help`: this usage message
- `-d`, `--debug`: run in debug mode, where all log messages are
output to the current shell
- `-f`, `--conffile FILE`: use FILE for the main configuration
instead of `/etc/backupninja.conf`
- `-t`, `--test`: test run mode. This will test if the backup could
run, without actually preforming any backups. For example, it will
attempt to authenticate or test that ssh keys are set correctly.
- `-n`, `--now`: perform actions now, instead of when they might
be scheduled. No output will be created unless also run with -d.
- `--run FILE`: runs the specified action `FILE` (e.g. one of the
`/etc/backup.d/` files). Also puts backupninja in debug mode.
ninjahelper
===========
ninjahelper is an additional script which will walk you through the process of
configuring backupninja. Ninjahelper has a menu driven curses based interface
(using dialog).
To add an additional 'wizard' to ninjahelper, follow these steps:
1. To add a helper for the handler "blue", create the file
`blue.helper` in the directory where the handlers live.
(ie `/usr/share/backupninja`).
2. Next, you need to add your helper to the global `HELPERS` variable
and define the main function for your helper (the function name
is always `<helper>_wizard`). for example, `blue.helper`:
HELPERS="$HELPERS blue:description_of_this_helper"
blue_wizard() {
... do work here ...
}
3. Look at the existing helpers to see how they are written. Try to re-use
functions, such as the dialog functions that are defined in `easydialog.sh`.
4. Test, re-test, and test again. Try to break the helper by going backwards,
try to think like someone who has no idea how to configure your handler
would think, try to make your helper as simple as possible. Walk like a cat,
become your shadow, don't let your senses betray you.
Configuration
=============
Configuration files
-------------------
The general configuration file is `/etc/backupninja.conf`. In this file
you can set the log level and change the default directory locations.
You can force a different general configuration file with `backupninja
-f /path/to/conf`.
To preform the actual backup, backupninja processes each configuration
file in `/etc/backup.d` according to the file's suffix:
- `.sh`: run this file as a shell script.
- `.rdiff`: filesystem backup (using rdiff-backup)
- `.restic`: filesystem backup (using restic)
- `.dup`: filesystem backup (using duplicity)
- `.borg`: filesystem backup (using borg)
- `.mysql`: backup mysql databases
- `.pgsql`: backup PostgreSQL databases
- `.sys`: general hardware, partition, and system reports.
- `.svn`: backup subversion repositories
- `.maildir`: incrementally backup maildirs (very specialized)
Support for additional configuration types can be added by dropping
bash scripts with the name of the suffix into `/usr/share/backupninja`.
The configuration files are processed in alphabetical order. However,
it is suggested that you name the config files in "sysvinit style."
For example:
00-disabled.pgsql
10-runthisfirst.sh
20-runthisnext.mysql
90-runthislast.rdiff
Typically, you will put a `.rdiff` config file last, so that any
database dumps you make are included in the filesystem backup.
Configurations files with names beginning with 0 (zero) or ending with
`.disabled` (preferred method) are skipped.
Unless otherwise specified, the config file format is "ini style."
For example:
# this is a comment
[fishes]
fish = red
fish = blue
[fruit]
apple = yes
pear = no thanks \
i will not have a pear.
The [example configuration files](examples) document all options
supported by the handlers shipped with backupninja.
Scheduling
----------
By default, each configuration file is processed everyday at 01:00 (1
AM). This can be changed by specifying the 'when' option in a config
file.
For example:
when = sundays at 02:00
when = 30th at 22
when = 30 at 22:00
when = everyday at 01 <-- the default
when = Tuesday at 05:00
A configuration file will be processed at the time(s) specified by the
`when` option. If multiple `when` options are present, then they all
apply. If two configurations files are scheduled to run in the same
hour, then we fall back on the alphabetical ordering specified above.
If two configurations files are scheduled close to one another in
time, it is possible to have multiple copies of backupninja running if
the first instance is not finished before the next one starts.
Make sure that you put the `when` option before any sections in your
configuration file.
These values for `when` are equivalent:
when = tuesday at 05:30
when = TUESDAYS at 05
These values for `when` are invalid:
when = tuesday at 2am
when = tuesday at 2
when = tues at 02
SSH keys
--------
In order for rdiff-backup to sync files over ssh unattended, you must
create ssh keys on the source server and copy the public key to the
remote user's authorized keys file. For example:
root@srchost# ssh-keygen -t rsa -b 4096
root@srchost# ssh-copy-id -i /root/.ssh/id_rsa.pub backup@desthost
Now, you should be able to ssh from user `root` on `srchost` to
user `backup` on `desthost` without specifying a password.
Note: when prompted for a password by `ssh-keygen`, just leave it
blank by hitting return.
The included helper program `ninjahelper` will walk you through creating
an rdiff-backup configuration, and will set up the ssh keys for you.
Amazon Simple Storage Service (S3)
----------------------------------
Duplicity can store backups on Amazon S3 buckets, taking care of encryption.
Since it performs incremental backups it minimizes the number of request per
operation therefore reducing the costs. The boto Python interface to Amazon
Web Services is needed to use duplicity with S3 (Debian package: `python-boto`).
.sh configuration files
-----------------------
Shell jobs may use the following features:
* logging and control flow functions:
`halt`, `fatal`, `error`, `warning`, `info`, `debug`, `passthru`.
All such functions take a list of strings a parameters.
Those strings are passed to whatever logging mechanism is enabled,
and colored if relevant.
* Using `exit N` is useless, and has unspecified consequences.
Just don't do it.
* `when=TIME` works as documented above; at may also be written
`when = TIME`.
* The `$BACKUPNINJA_DEBUG` environment variable is set when
backupninja is invoked with the `-d` option.
Real world usage
================
Backupninja can be used to implement whatever backup strategy you
choose. It is intended, however, to be used like so:
1. First, databases are safely copied or exported to `/var/backups`.
Typically, you cannot make a file backup of a database while it
is in use, hence the need to use special tools to make a safe copy
or export into `/var/backups`.
2. Then, vital parts of the file system, including `/var/backups`, are
nightly pushed to a remote, off-site, hard disk (using
rdiff-backup). The local user is root, but the remote user is not
priviledged. Hopefully, the remote filesystem is encrypted.
There are many different backup strategies out there, including "pull
style", magnetic tape, rsync + hard links, etc. We believe that the
strategy outlined above is the way to go because:
1. hard disks are very cheap these days;
2. pull style backups are no good, because then the backup server must
have root on the production server;
3. rdiff-backup is more space efficient and featureful than using
rsync + hard links.
More information
================
FAQ
---
See the [FAQ](FAQ.md).
Mailing list
------------
The backupninja
[mailing list](https://lists.riseup.net/www/info/backupninja) is
suitable for usage and development questions.
When developping fixes or new features for backupninja, it is highly recommended
to run the test suite to help spot potential problems.
The test suite is based on Vagrant, and is configured to rely on the VirtualBox
provider. The required package may be installed using the following command:
apt install vagrant virtualbox
On Debian 10 (buster) these packages aren't available in the default upstream
repositories, so you will need to use an alternative such as the one provided
by an individual Debian developper here:
https://people.debian.org/~lucas/virtualbox-buster/
Once the requirements are in place, the test suite may be run in this manner:
git clone git@0xacab.org:liberate/backupninja.git
cd backupninja
vagrant up
vagrant ssh -c "sudo /vagrant/test/test.sh"
It's possible to only test a specific handler with:
vagrant ssh -c "sudo /vagrant/test/test.sh rdiff"
To synchronise changes in the source code and rebuild backupninja:
vagrant rsync local && vagrant ssh -c "build-backupninja.sh"
Please report any problems with the test suite on the issue tracker at:
https://0xacab.org/liberate/backupninja/-/issues
Upstream
========
* run the full testsuite
vagrant rsync local && \
vagrant ssh -c "build-backupninja.sh && sudo /vagrant/test/test.sh"
* prepare the environment:
export VERSION=x.y.z
* update `configure.ac` and `CHANGELOG.md`
perl -pi -E \
"s{^AC_INIT\(\[backupninja\],\[[0-9.\-rc]+\],}{AC_INIT([backupninja],[$VERSION],}" \
configure.ac
RELEASE_DATE=$(LC_ALL=C date '+%Y-%m-%d'); perl -pi -E \
"s{^## \[Unreleased\].*}{## [$VERSION] - $RELEASE_DATE}" \
CHANGELOG.md
* commit and created signed tag
git commit configure.ac CHANGELOG.md \
-m "Releasing backupninja $VERSION" && \
git clean -fdx -e .vagrant && \
git tag -s "backupninja-$VERSION" \
-m "Releasing backupninja $VERSION" && \
./autogen.sh && \
./configure && \
make dist
* compare the content of the generated tarball with the content of the
previous one
diffoscope --text-color=always ../tarballs/backupninja-x.y.z.tar.gz \
backupninja-$VERSION.tar.gz | less -R
* move the tarball outside of the Git working copy and clean up
mkdir -p ../tarballs && \
mv backupninja-$VERSION.tar.gz ../tarballs/ && \
make distclean && \
git clean -fdx -e .vagrant
Debian
======
Prepare a new package:
git checkout debian && \
gbp import-orig --upstream-vcs-tag="backupninja-$VERSION" \
../tarballs/backupninja-$VERSION.tar.gz && \
gbp dch --auto && \
dch -e && \
export DEBIAN_VERSION=$(dpkg-parsechangelog -SVersion) && \
git commit debian/changelog \
-m "Releasing backupninja ($DEBIAN_VERSION) to Debian unstable" && \
gbp buildpackage
Install the `.deb` and test.
Release
=======
* push the release to GitLab
git checkout debian && \
gbp buildpackage --git-tag-only --git-sign-tags && \
git push --follow-tags origin \
master:master \
debian:debian \
pristine-tar:pristine-tar \
upstream:upstream
* create a new GitLab release
* announce the release on the backupninja mailing-list,
pointing to the milestone web page
* upload to Debian or ask someone listed in the `Uploaders` control
field to review and upload
Open the next development cycle
===============================
* `git checkout master`
* Add an empty new section in `CHANGELOG.md`, commit and push.
......@@ -5,16 +5,14 @@ you are working on it!
. Fix all bugs reported on the Debian BTS:
http://bugs.debian.org/cgi-bin/pkgreport.cgi?which=pkg&data=backupninja&archive=no
. Fix all bugs reported on our Redmine instance:
https://labs.riseup.net/code/projects/backupninja/issues
. Fix all bugs reported on our Gitlab instance:
https://0xacab.org/liberate/backupninja/issues
. Make ninjahelper allow you to pick what type of backup you want (instead
of just assuming you want local-to-remote, or push backups. Some people
want local-to-local, or remote-to-local, or pull backups). This has been
reported for the duplicity handler as Debian bug #346040.
. Allow vsnames "all" in the msyql handler.
. Factorize the rdiff.helper's connection-related functions into a lib, so
that they can be used by dup.helper too. (NB: don't forget that the dup
handler has a sshoptions configuration setting, that is often used to
......
# -*- mode: ruby -*-
# vi: set ft=ruby :
ENV["LC_ALL"] = "en_US.UTF-8"
base_box = "debian/testing64"
empty_disk = '.vagrant/tmp/empty.vdi'
lvm_disk = '.vagrant/tmp/lvm.vdi'
lukspart_disk = '.vagrant/tmp/lukspart.vdi'
luksdev_disk = '.vagrant/tmp/luksdev.vdi'
Vagrant.configure("2") do |config|
config.vm.define "remote" do |remote|
remote.vm.box = base_box
remote.vm.hostname = "bntest1"
remote.vm.network "private_network", ip: "192.168.181.5"
remote.vm.provision "shell", inline: <<-SHELL
export DEBIAN_FRONTEND=noninteractive
echo "en_US.UTF-8 UTF-8" >> /etc/locale.gen
locale-gen
apt-get update
apt-get install -y borgbackup duplicity rdiff-backup restic rsync
sed -i 's/^PasswordAuthentication no/PasswordAuthentication yes/' /etc/ssh/sshd_config
echo "Port 22" >> /etc/ssh/sshd_config
echo "Port 7722" >> /etc/ssh/sshd_config
systemctl reload sshd
echo -e "vagrant\nvagrant" | passwd vagrant
chown vagrant: /var/backups
wget -q https://github.com/restic/rest-server/releases/download/v0.10.0/rest-server_0.10.0_linux_amd64.tar.gz -O - | tar -xz -C /usr/local/bin --strip-components=1
SHELL
end
config.vm.define "local", primary: true do |local|
local.vm.box = base_box
local.vm.hostname = "bntest0"
local.vm.network "private_network", ip: "192.168.181.4"
local.vm.provision "shell", inline: <<-SHELL
export DEBIAN_FRONTEND=noninteractive
echo "root: vagrant" >> /etc/aliases
echo "en_US.UTF-8 UTF-8" >> /etc/locale.gen
locale-gen
apt-get update
apt-get install -y automake make dialog sshpass
BUILDSCRIPT="/usr/local/bin/build-backupninja.sh"
echo "#!/bin/sh" >> $BUILDSCRIPT
echo "cd /vagrant" >> $BUILDSCRIPT
echo "make clean" >> $BUILDSCRIPT
echo "./autogen.sh" >> $BUILDSCRIPT
echo "./configure --prefix=/usr --mandir=/usr/share/man --sysconfdir=/etc --localstatedir=/var --libdir=/usr/lib --libexecdir=/usr/lib" >> $BUILDSCRIPT
echo "make" >> $BUILDSCRIPT
echo "sudo make install" >> $BUILDSCRIPT
chmod +x $BUILDSCRIPT
$BUILDSCRIPT
mkdir -p /root/.ssh
yes y | ssh-keygen -t ed25519 -f /root/.ssh/id_ed25519 -N ''
echo "StrictHostKeyChecking accept-new" >> /root/.ssh/config
echo "192.168.181.5 bntest1" >> /etc/hosts
sshpass -p vagrant scp /root/.ssh/id_ed25519.pub vagrant@bntest1:/tmp/bntest.pub
sshpass -p vagrant ssh vagrant@bntest1 "cat /tmp/bntest.pub >> /home/vagrant/.ssh/authorized_keys"
sshpass -p vagrant ssh vagrant@bntest1 "chmod 400 /home/vagrant/.ssh/authorized_keys"
ssh vagrant@bntest1 "sudo sed -i 's/^PasswordAuthentication yes/PasswordAuthentication no/' /etc/ssh/sshd_config"
ssh vagrant@bntest1 "sudo systemctl reload sshd"
SHELL
local.vm.provider :virtualbox do |vb|
unless File.exist?(empty_disk)
vb.customize ['createhd', '--filename', empty_disk, '--size', 100 ]
end
unless File.exist?(empty_disk)
vb.customize ['createhd', '--filename', lvm_disk, '--size', 100 ]
end
unless File.exist?(lukspart_disk)
vb.customize ['createhd', '--filename', lukspart_disk, '--size', 100 ]
end
unless File.exist?(luksdev_disk)
vb.customize ['createhd', '--filename', luksdev_disk, '--size', 100 ]
end
vb.customize ['storageattach', :id, '--storagectl', 'SATA Controller', '--port', 1, '--device', 0, '--type', 'hdd', '--medium', empty_disk]
vb.customize ['storageattach', :id, '--storagectl', 'SATA Controller', '--port', 2, '--device', 0, '--type', 'hdd', '--medium', lvm_disk]
vb.customize ['storageattach', :id, '--storagectl', 'SATA Controller', '--port', 3, '--device', 0, '--type', 'hdd', '--medium', lukspart_disk]
vb.customize ['storageattach', :id, '--storagectl', 'SATA Controller', '--port', 4, '--device', 0, '--type', 'hdd', '--medium', luksdev_disk]
end
end
config.vm.synced_folder ".", "/vagrant", type: "rsync",
rsync__exclude: ".git/",
rsync__args: ["--recursive", "--delete"]
end
......@@ -8,10 +8,10 @@ fi
if [ "x$1" = "x-f" ]
then
autoscan
[ -f "configure.in" ] && cp "configure.in" "configure.in.old"
mv -f "configure.scan" "configure.in"
echo "## This is just AUTOSCAN draft of configure.in"
$EDITOR "configure.in"
[ -f "configure.ac" ] && cp "configure.ac" "configure.ac.old"
mv -f "configure.scan" "configure.ac"
echo "## This is just AUTOSCAN draft of configure.ac"
$EDITOR "configure.ac"
fi
### použít jen když je třeba použít configure.h.in
......
......@@ -7,11 +7,11 @@ Version: %{version}
Release: 1
License: GPL
Group: Applications/System
URL: https://labs.riseup.net/code/projects/show/backupninja
URL: https://0xacab.org/liberate/backupninja
Source: %{name}-%{version}.tar.gz
Requires: bash, gawk, rdiff-backup, gzip
Provides: %{name}
Packager: Petr Klima <Petr.Klima@madeta-group.cz>
Packager: Petr Klima <Petr.Klima@madeta.cz>
BuildRoot: %{_tmppath}/%{name}-%{version}
Prefix: %{_prefix}
......@@ -50,7 +50,7 @@ rm -fr %{buildroot}
%ghost %{_localstatedir}/log/backupninja.log
%doc AUTHORS COPYING ChangeLog INSTALL NEWS README
%doc AUTHORS COPYING CHANGELOG.md INSTALL.md NEWS README.md
%{_mandir}/man1/*
%{_mandir}/man5/*
......
......@@ -3,15 +3,12 @@
# The maintainer mode is causing me grief with newest versions of autotools
#AM_MAINTAINER_MODE
AC_INIT([backupninja],[0.9.6],[backupninja@lists.riseup.net])
AC_INIT([backupninja],[1.2.2],[backupninja@lists.riseup.net])
AC_CONFIG_SRCDIR([src/backupninja.in])
AM_INIT_AUTOMAKE
AM_INIT_AUTOMAKE([foreign])
# Checks for programs.
# BASH may already be set in the shell, if the admin then changes the
# the /bin/sh symlink to a non-bash shell, all hell will break lose.
unset BASH
AC_PATH_PROGS(BASH, bash, "no", [$PATH:/bin:/usr/bin:/usr/sbin])
if test x$BASH = "xno"; then
AC_MSG_ERROR([bash is required])
......
......@@ -13,6 +13,10 @@
# 1 -- Fatal errors (only)
loglevel = 4
# Produce prometheus metrics of backup status (default = no).
# Requires `prometheus-node-exporter` to be installed
reportprom = false
# send a summary of the backup status to
# this email address:
reportemail = root
......@@ -21,6 +25,10 @@ reportemail = root
# even if all modules reported success. (default = yes)
reportsuccess = yes
# if set to 'yes', info messages from handlers will be
# sent into the email (default = no)
reportinfo = no
# if set to 'yes', a report email will be generated
# even if there was no error. (default = yes)
reportwarning = yes
......@@ -31,18 +39,21 @@ reportspace = no
# where to rsync the backupninja.log to be aggregated in
# a ninjareport
reporthost =
reporthost =
# what user to connect to reporthost to sync the
# backupninja.log
reportuser = ninja
# where on the reporthost should the report go
# NOTE: the name of the log will be used in the report,
# NOTE: the name of the log will be used in the report,
# use a globally unique name, preferably the hostname
reportdirectory = /var/lib/backupninja/reports
# set to the administration group that is allowed to
# number of columns the report email body should wrap to
#reportwrap = 80
# set to the administration group that is allowed to
# read/write configuration files in /etc/backup.d
admingroup = root
......@@ -68,9 +79,6 @@ usecolors = yes
# default value for 'when'
when = everyday at 01:00
# if running vservers, set to yes
vservers = no
# programs paths
# SLAPCAT=/usr/sbin/slapcat
# LDAPSEARCH=/usr/bin/ldapsearch
......@@ -79,10 +87,9 @@ vservers = no
# MYSQL=/usr/bin/mysql
# MYSQLHOTCOPY=/usr/bin/mysqlhotcopy
# MYSQLDUMP=/usr/bin/mysqldump
# PSQL=/usr/bin/psql
# PGSQLDUMP=/usr/bin/pg_dump
# PGSQLDUMPALL=/usr/bin/pg_dumpall
# GZIP=/bin/gzip
# GZIP_OPTS='--rsyncable'
# RSYNC=/usr/bin/rsync
# VSERVERINFO=/usr/sbin/vserver-info
# VSERVER=/usr/sbin/vserver
# VROOTDIR=/var/lib/vservers