Run our custom programs test suite on interesting commits

For now the scope of this ticket is limited to our 3 Perl code bases (perl5lib, persistence-setup, iuk)… simply because our other custom programs (sadly) have no test suite.

One way to do this would be to run our custom programs’ test suite, using Vagrant, on our existing isobuilders (that already have Vagrant installed and some room for storing baseboxes) or isotesters (which would feel more logical, because “testers”) or both. What it takes is:

• at the root of each Git repo we want to test:
  • add a suitable Vagrantfile that installs all build/test deps during provisioning
  • add a Makefile/Rakefile/script that starts the Vagrant VM, runs the test suite in there, and tears down the VM
• for each Git repo we want to test:
  • create a Jenkins job that runs the aforementioned script / rake test / make test

Open questions:

• how to deal with inter-dependencies between these code bases? e.g. persistence-setup and iuk depend on perl5lib; our options:
  • start by testing perl5lib that has no such problem, gain experience with the general idea during this first iteration, and think about this problem later iff. the general idea is validated
  • install current perl5lib master in the VM with dzil install: this works iff. we don’t delegate to the Debian packaging anything that’s required to use perl5lib (currently we install MO files via debian/rules, but perhaps we can do without them here)
  • set up the APT source for our devel custom APT repo and treat perl5lib just like any other build dependency
• how to avoid creating yet another complex setup for managing baseboxes? our options:
  • don’t keep any basebox persistently on the Jenkins “slaves” (sic :/) that run the tests, instead download & provision them every time; we can probably leverage our existing apt-cacher-ng to speed things up (see file:///usr/share/doc/apt-cacher-ng/html/howtos.html#ssluse to make caching of HTTPS downloads work). This option implies we use Vagrant baseboxes published by third-parties (otherwise we’re just moving the basebox management problem elsewhere) which is probably acceptable here; an official Debian Stretch Vagrant box is available, and we can dist-upgrade it during provisioning to whatever version of Debian we want to test on.
  • use the baseboxes we already use & manage for building ISO images: this probably requires quite some refactoring to extract code from tails.git, and forces us to test only on whatever snapshot of Debian we currently use, while we would like to test on current Debian sid too.
  • other ideas?
• how to store or compute build & test dependencies? We already store them in two different places (dist.ini on the master branch, debian/control on the debian branch) so it would be nice to avoid adding a 3rd one. Likely the easiest way is to extract build deps from the debian/control file and install them with something like `controlfile=$(mktemp) ; git show origin/debian:debian/control

  $controlfile ; mk-build-deps -i -r --root-cmd sudo $controlfile ; rm $controlfile`

• how does this relate to #7036 and #6220-note_11? Can we implement this ticket in a way that also makes us progress on that other front? At least having to deal with custom APT sources, APT snapshots, build deps, will teach us a thing or three about the issues we’ll face when we tackle that other project, but perhaps we can do better.

Ideally we would run these tests both on Debian sid (to detect breakage in advance and make our RM’s life easier), whatever snapshot of Debian we currently use, and an unfrozen version of the Debian release we use. But already running them on one of them (probably sid) would be a pretty good start!

Subtasks

• #6217 (closed)

Related issues

• Blocked by #7036

Original created by @intrigeri on 6218 (Redmine)