backupninja issueshttps://0xacab.org/liberate/backupninja/-/issues2023-08-27T11:49:15Zhttps://0xacab.org/liberate/backupninja/-/issues/11348Borg handler should detect OOM killing borg2023-08-27T11:49:15ZOlivier BergerBorg handler should detect OOM killing borgI noticed a strange behaviour, when running a borg backup (see #11347) and tried an reproduce the borg behaviour by running it manually with same options.
This lead to seeing that the return code was 137 (OOM killed).
Looking at the co...I noticed a strange behaviour, when running a borg backup (see #11347) and tried an reproduce the borg behaviour by running it manually with same options.
This lead to seeing that the return code was 137 (OOM killed).
Looking at the code of the borg handler, it doesn't seem to exhibit an OOM in any particular way.
All I see in the report received by mail is :
```
... Killed
Fatal: Failed backing up source.
`̀``
Maybe that would be useful to get notified of that particular return code with a specific error message, more verbose than the laconic "killed".
Thanks in adavance.https://0xacab.org/liberate/backupninja/-/issues/11347borg handler should display error messages on separate lines2023-08-27T11:49:16ZOlivier Bergerborg handler should display error messages on separate linesHere's an example message received by mail using a borg handler rule with borg 1.2.4 on the client side (Debian stable):
```
*failed* -- /etc/backup.d/90.borg
== fatal errors from /etc/backup.d/90.borg ==
Info: Repository was already i...Here's an example message received by mail using a borg handler rule with borg 1.2.4 on the client side (Debian stable):
```
*failed* -- /etc/backup.d/90.borg
== fatal errors from /etc/backup.d/90.borg ==
Info: Repository was already initialized
Error: Remote: Borg 0.29.0: exception in RPC call: Remote: Traceback (most recent call last): Remote: File ".../borgbackup-0.29.0/borg/remote.py", line 96, in serve Remote: TypeError: open() takes from 2 to 5 positional arguments but 7 were given Remote: Command 'uname -p 2> /dev/null' is not in the allowed list. Remote: Platform: FreeBSD ... 13.1-RELEASE-p3 FreeBSD 13.1-RELEASE-p3 rsync_13_1 amd64 Remote: Python: CPython 3.4.3 Remote: Please note: If you see a TypeError complaining about the number of positional arguments given to open(), you can ignore it if it comes from a borg version < 1.0.7. This TypeError is a cosmetic side effect of the compatibility code borg clients >= 1.0.7 have to support older borg servers. This problem will go away as soon as the server has been upgraded to 1.0.7+. Killed stale lock ...@117286549492085.15781-0. Removed stale exclusive roster lock for host ...@117286549492085 pid 15781 thread 0.
Removed stale exclusive roster lock for host ...@117286549492085 pid 15781 thread 0. Killed
Fatal: Failed backing up source.
```
AFICS, there are several errors, starting with 'Remote:', and maybe an info message (Killed stale lock, etc.).
It would be much easier to have Remote messages on individual lines, and as a side effect this would avail issues with mail forwarding such as "message has lines too long for transport" when there are more issues reported :-/
Also there may be something weird here due to error reported whereas it's just a stale lock removal... but that's another issue, I guess.
Hope this makes sense, and I understand correctly what's happening there.
Best regards,https://0xacab.org/liberate/backupninja/-/issues/11346Suggested values for reportprom are both ambiguous and false for deactivation2023-07-12T08:38:37ZOlivier BergerSuggested values for reportprom are both ambiguous and false for deactivationAFAIU, the reportprom setting should be set to nothing, in the provided default backupninja.conf, given the checks performed in the code.
However, currebtly, the default settings set (or suggested by the comments) in the .conf file wil...AFAIU, the reportprom setting should be set to nothing, in the provided default backupninja.conf, given the checks performed in the code.
However, currebtly, the default settings set (or suggested by the comments) in the .conf file will activate it: anything like "no" or "false" is not empty string, and thus leads to activating prometheus reporting.
Which leads to an error whenever prometheus isn't activated, on most boxes I guess: "Error: /var/lib/prometheus/node-exporter does not exist!"
See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1035758 for my initial report downstream.https://0xacab.org/liberate/backupninja/-/issues/11343ninjahelper doesn't set B2 options for restic2023-02-12T20:13:15ZGene Catninjahelper doesn't set B2 options for resticI'm new to backupninja. I tried setting up a restic-based backup for the first time. It failed due to lack of B2 configuration--backblaze is my backup destination. After going through the backupninja source, I found that `b2_account_id...I'm new to backupninja. I tried setting up a restic-based backup for the first time. It failed due to lack of B2 configuration--backblaze is my backup destination. After going through the backupninja source, I found that `b2_account_id` and `b2_account_key` are supposed to be read from a `[b2]` section in the restic backup config. After manually adding those in to the config, the backup worked.
Solution:
1. Have ninjahelper ask for B2/S3/etc configuration fields when configuring restichttps://0xacab.org/liberate/backupninja/-/issues/11341example.maildir: srcsubdir no longer exists2022-11-18T01:05:00Ztaggartexample.maildir: srcsubdir no longer existsIn [example.maildir](https://0xacab.org/liberate/backupninja/-/blob/master/examples/example.maildir#L43) there is a reference to a srcsubdir variable that no longer exists in the code. If you attempt to use it, it gets ignored and instea...In [example.maildir](https://0xacab.org/liberate/backupninja/-/blob/master/examples/example.maildir#L43) there is a reference to a srcsubdir variable that no longer exists in the code. If you attempt to use it, it gets ignored and instead everything in srcdir gets backed up.https://0xacab.org/liberate/backupninja/-/issues/11340Borg handler compatibility with borg 1.22023-10-12T15:34:13ZGuillaume SubironBorg handler compatibility with borg 1.2Current borg handler is not fully compatible with borg 1.2.
- New warnings can cause a lot of noise (https://github.com/borgbackup/borg/issues/6379).
- We need to explicitly call `borg compact` (https://borgbackup.readthedocs.io/en/stabl...Current borg handler is not fully compatible with borg 1.2.
- New warnings can cause a lot of noise (https://github.com/borgbackup/borg/issues/6379).
- We need to explicitly call `borg compact` (https://borgbackup.readthedocs.io/en/stable/usage/notes.html#separate-compaction)
I am working on a new version of the handler. I will submit my changes when they are well tested.Guillaume SubironGuillaume Subironhttps://0xacab.org/liberate/backupninja/-/issues/11338PostgreSQL handler doesn't work on host installed on Debian 11 (Bullseye)2021-12-22T10:22:42ZGuillaume SubironPostgreSQL handler doesn't work on host installed on Debian 11 (Bullseye)On Debian 11 (Bullseye), postgres user is installed with nologin as shell
```
# grep postgres /etc/passwd
postgres:x:108:65534::/home/postgres:/usr/sbin/nologin
```
Which causes an error when the hander list databases :
```
# su - pos...On Debian 11 (Bullseye), postgres user is installed with nologin as shell
```
# grep postgres /etc/passwd
postgres:x:108:65534::/home/postgres:/usr/sbin/nologin
```
Which causes an error when the hander list databases :
```
# su - postgres -c 'psql -AtU postgres -c \"SELECT datname FROM pg_database WHERE NOT datistemplate\"'
This account is currently not available.
```
And, even worse, the handler continues and tries to dump databases "This", "Account", "is", "currently", etc.
```
# ls /var/backups/postgres/
account.pg_dump.gz currently.pg_dump.gz is.pg_dump.gz This.pg_dump.gz
available..pg_dump.gz globals.sql.gz not.pg_dump.gz
```
`chsh postgres -s /bin/bash` fixes the problem, but backupninja should work out of the box. I don't know yet what would be the best solution. Using `sudo -u` instead of `su -` works, but backupninja should not depends on sudo…https://0xacab.org/liberate/backupninja/-/issues/11337Feature request: pass arbitrary options to "restic backup"2022-03-05T18:46:21ZphlummoxFeature request: pass arbitrary options to "restic backup"The current format for a ".restic" config file only permits a limited number of options to be passed to the "restic backup" command - one can't pass, for instance "`--exclude-larger-than`", "`--ignore-ctime`", and various other options.
...The current format for a ".restic" config file only permits a limited number of options to be passed to the "restic backup" command - one can't pass, for instance "`--exclude-larger-than`", "`--ignore-ctime`", and various other options.
Could the config file format be altered so that arbitrary options can be passed? Then users could set whatever options they like.
This feature already exists for the Duplicity handler (see [here][dup-example-l9] and [here][dup-conf-l8]).
[dup-example-l9]: https://0xacab.org/liberate/backupninja/-/blob/backupninja-1.2.1/examples/example.dup#L9
[dup-conf-l8]: https://0xacab.org/liberate/backupninja/-/blob/backupninja-1.2.1/handlers/dup.in#L124
I've a [patch][patch] over on GitHub that adds this feature - let me know if you'd like me to make a merge request here on https://0xacab.org.
[patch]: https://github.com/phlummox-patches/backupninja/pull/3/commits/1942fb4a103f66f31b9e64b380f20d1f36e3fd1fhttps://0xacab.org/liberate/backupninja/-/issues/11336Impossible to create a `dup` backup action on Debian using ninjahelper2023-07-08T10:34:09ZphlummoxImpossible to create a `dup` backup action on Debian using ninjahelperOn Debian-based systems (I tested on Debian 11, "Bullseye", and Ubuntu 18.04, "Bionic"), it seems impossible to use `ninjahelper` to create a `dup` backup action, as trying to select "src" directories results in the error:
```
/usr/shar...On Debian-based systems (I tested on Debian 11, "Bullseye", and Ubuntu 18.04, "Bionic"), it seems impossible to use `ninjahelper` to create a `dup` backup action, as trying to select "src" directories results in the error:
```
/usr/share/backupninja/dup.helper: line 484: do_dup_src: command not found
```
## Version of backupninja used
1.2.1-1
(Installed from the Debian 11 repository.)
## Steps to reproduce
This can be reproduced by running Debian in a Docker container -- `docker run --rm -it debian:bullseye`.
Within the container:
1\. install `duplicity` and `backupninja`:
```
$ apt-get update && apt-get install -y --no-install-recommends duplicity backupninja
```
2\. Run `ninjahelper`:
```
$ ninjahelper
```
3\. Select "new / create a new backup action"
4\. Select "dup / incremental encrypted remote filesystem backup"
5\. Select "src / choose files to include and exclude"
## Expected behaviour
The user should be able to specify files to include and exclude.
## Actual behaviour
The error message
```
/usr/share/backupninja/dup.helper: line 484: do_dup_src: command not found
```
briefly appears at the bottom of the screen, and it's impossible to progress further through the `ninjahelper` "wizard".https://0xacab.org/liberate/backupninja/-/issues/11335backupninja failes with latest duplicity2022-03-05T18:47:05Zyovabackupninja failes with latest duplicityOption `--extra-clean` was removed in [0.8.11](https://gitlab.com/duplicity/duplicity/-/blob/master/CHANGELOG.md#rel0811-2020-02-24).Option `--extra-clean` was removed in [0.8.11](https://gitlab.com/duplicity/duplicity/-/blob/master/CHANGELOG.md#rel0811-2020-02-24).https://0xacab.org/liberate/backupninja/-/issues/11334rdiff verison2 not compatible2022-03-05T18:49:23ZDaniel Horstmannrdiff verison2 not compatibleHi,
after uprading Debian to version 11 (bullseye), rdiff was upgraded, to (from version 1 to 2),
its not possible anymore to use backupninja with rdiff Configs.
I guess the cli options changed, so backupninja can't handle that anymore...Hi,
after uprading Debian to version 11 (bullseye), rdiff was upgraded, to (from version 1 to 2),
its not possible anymore to use backupninja with rdiff Configs.
I guess the cli options changed, so backupninja can't handle that anymore.
Output of backupninja trying to backup:
```
Error: Fatal Error: Switches missing or wrong number of arguments
See the rdiff-backup manual page for more information.
Fatal Error: Truncated header string (problem probably originated remotely)
[...]
```
Does anyone else have this problem and maybe solved it already?
Thanks and Best regardshttps://0xacab.org/liberate/backupninja/-/issues/11331restic handler with Swift misses some environment variables2022-03-05T18:50:53ZAcidy Skankrestic handler with Swift misses some environment variablesHi, posting here for the sake of documenting my findings.
This was found through trial and error, I haven't enough knowledge of Swift and backupninja to know for sure what the problem is.
For it to work with servers.com cloud storage I ...Hi, posting here for the sake of documenting my findings.
This was found through trial and error, I haven't enough knowledge of Swift and backupninja to know for sure what the problem is.
For it to work with servers.com cloud storage I have had to modify the restic handler a bit to remove some mandatory environment variables and add some other, otherwise it would fail on authenticating.
I have simply removed the line that fatals when one of the variable is missing, it seems the client gets confused depending on which variable is set or not, ending with authentication errors.
I have added 3 more variables. First at the top of the file in the swift `setsection`, then in the export section that sets the environment variables for restic. And I almost forgot the `unset` lines at the end of the file, important! :)
The according values also have to be added to the *.restic file and variables like os_tenant_id can be left commented out.
If that is allowed, I suggest that no variable should be mandatory in the *.restic file and the choice of which to enable by uncommenting the line should be left to the user depending on their Swift authentication requirements.
The OpenStack Swift repository section for servers.com looks like that:
```bash
# OpenStack Swift repository
if [ "$(echo "$repository" | /usr/bin/awk -F ':' '{print $1}')" == "swift" ]; then
export_debug OS_AUTH_URL "$os_auth_url"
#export_debug OS_TENANT_ID "$os_tenant_id"
export_debug OS_TENANT_NAME "$os_tenant_name"
export_debug OS_USERNAME "$os_username"
export_debug OS_PASSWORD "$os_password"
#export_debug OS_REGION_NAME "$os_region_name"
export_debug OS_USER_DOMAIN_NAME "$os_user_domain_name"
export_debug OS_DEFAULT_DOMAIN_NAME "$os_default_domain_name"
export_debug OS_IDENTITY_API_VERSION "$os_identity_api_version"
fi
```
Restic issue that put me on the right track: https://forum.restic.net/t/where-to-find-an-existing-swift-container-name/3664https://0xacab.org/liberate/backupninja/-/issues/11330[PROPOSAL] Allow separation of secrets2021-06-27T21:46:25ZDaniel Fancsali[PROPOSAL] Allow separation of secrets# Problem statement
Nowadays infrastructure config is handled as code, and stored in git repositories. However, it's undoubtedly bad practice to store secrets in said repository. Also, the same config template might be used for several i...# Problem statement
Nowadays infrastructure config is handled as code, and stored in git repositories. However, it's undoubtedly bad practice to store secrets in said repository. Also, the same config template might be used for several instances, where only minor things would change: maybe the username and password the host uses to contact the remote backup service.
Currently it is impossible to keep the secrets (or similar host-specific) values in a separate file, so one can deploy the config in an automated manner, and upload the secrets to the host separately.
# Goal
Allow the user to create a **split** configuration: meaning the config file with all the configuration can be deployed *separately* from the config-snippets storing the actual secrets.
Thinking about the possible implementations for a moment in advance, it might be beneficial to aim for a **hierarchical** approach, so that several config files can load and freely override the same values, to adhere to the DRY-principle
# Proposed approach(es)
## Include functionality
Include the same file into the respective config files in `/etc/backup.d`. (The config stored in the git repository would contain the include statement, and the file would be deployed separately)
## Host specific defaults/override file
The `getconf` logic could be amended so that besides the hard-coded defaults, it can do hierarchical lookup:
- **Either**: if a value is not found, it will go back to a *defaults* file (e.g. `/etc/backupninja.local.conf`) and return that value; only if that is also missing would it fall back to the default specified by the caller;
- **or** the file could be called something like `/etc/backupninja.override.conf`, and would be consulted before the actual config files in `/etc/backup.d` when `getconf` is called.
## Allow (environment) variable access when specifying values
The files in `/etc/backup.d` would allow something like `foo = $bar` to be specified, which in turn would fetch the value of the environment variable `bar` at run-time and use that.
This case would, however would also depend on environment variables allowed to be specifies somewhere -- and specifying them in `/etc/backupninja.conf` would defeat the purpose of this piece of config being truly host-specific: then we would move config data into the secret store. (And in general, for the `getconf`-case, the fallback to `/etc/backupninja.conf` would not satisfy the goal.
On the other hand this would allow access to standard UNIX environment variables, like: `backup_target = b2:some-bucket-name:/backup/$hostname`
## ... further options
Please feel free to chip in with your own ideas; the reason I am raising is to tap into the hive-mind -- I am more than happy to contribute the code, but wanted to see, what the community and the seasoned backupninja contributors think about the topic.https://0xacab.org/liberate/backupninja/-/issues/11329Duplicity with S3 needs the AWS credentials as env vars2021-05-22T08:58:39ZjoachimDuplicity with S3 needs the AWS credentials as env varsAfter having the following error while testing Duplicity with a S3-compatible Object Storage service (Scaleway)
```
NoAuthHandlerFound: No handler was ready to authenticate. 1 handlers were checked. ['HmacAuthV1Handler'] Check your cred...After having the following error while testing Duplicity with a S3-compatible Object Storage service (Scaleway)
```
NoAuthHandlerFound: No handler was ready to authenticate. 1 handlers were checked. ['HmacAuthV1Handler'] Check your credentials
```
I tried passing my credentials through environment variables
```sh
export AWS_ACCESS_KEY_ID="aws_access_key_id"
export AWS_SECRET_ACCESS_KEY="aws_secret_access_key"
```
The error didn't appear in the following test and the backup worked well.https://0xacab.org/liberate/backupninja/-/issues/11326Harmonise include / exclude defaults across handlers2021-01-15T00:48:24ZJerome CharaouiHarmonise include / exclude defaults across handlersThere are about as many defaults for includes / excludes as there are handlers and helpers.
We should settle on a single, sane recommendation for a typical modern system and implement that in the different examples, handlers and helpers...There are about as many defaults for includes / excludes as there are handlers and helpers.
We should settle on a single, sane recommendation for a typical modern system and implement that in the different examples, handlers and helpers.
Also reported in DEBBUG-734795.https://0xacab.org/liberate/backupninja/-/issues/11324Using bwlimit and sshoptions results in duplicate --remote-schema2021-01-12T04:23:59ZJerome CharaouiUsing bwlimit and sshoptions results in duplicate --remote-schemahttps://0xacab.org/liberate/backupninja/-/issues/11323Fix and re-enable the ldap handler2021-01-15T00:42:39ZJerome CharaouiFix and re-enable the ldap handlerThe LDAP handler is currently disabled in defaults builds because it's broken.
We should drop the current code and implement a new one based on this version which is used by some contributors:
* helper: https://paste.sysnove.net/paste/...The LDAP handler is currently disabled in defaults builds because it's broken.
We should drop the current code and implement a new one based on this version which is used by some contributors:
* helper: https://paste.sysnove.net/paste/qGUN3Ztl#Bj9gS4Zz
* handler: https://paste.sysnove.net/paste/9HozrSv0#R1aCSSS+
* example: https://paste.sysnove.net/paste/-MFnffKK#nXf7O4yM
(Also reported in DEBBUG-726115)Jerome CharaouiJerome Charaouihttps://0xacab.org/liberate/backupninja/-/issues/11322Add tests for the maildir handler2021-01-07T14:03:18ZJerome CharaouiAdd tests for the maildir handlerhttps://0xacab.org/liberate/backupninja/-/issues/11321Add tests for the wget handler2021-01-07T14:02:55ZJerome CharaouiAdd tests for the wget handlerhttps://0xacab.org/liberate/backupninja/-/issues/11320Add tests for the dsync handler2021-01-07T14:02:25ZJerome CharaouiAdd tests for the dsync handler