Retrieving ISO build artifacts sometimes fails on Jenkins

E.g. https://jenkins.tails.boum.org/job/build_Tails_ISO_feature-12599/17/console, https://jenkins.tails.boum.org/job/reproducibly_build_Tails_ISO_feature-5630-deterministic-builds/4/console and https://jenkins.tails.boum.org/job/reproducibly_build_Tails_ISO_testing/10/consoleFull expose this problem after successfully building an ISO image:

15:05:41 Retrieving artifacts from Vagrant build box.
15:05:42 Warning: Permanently added '192.168.121.131' (ECDSA) to the list of known hosts.
15:05:42 Warning: Permanently added '192.168.121.131' (ECDSA) to the list of known hosts.
15:21:47 packet_write_wait: Connection to 192.168.121.131 port 22: Broken pipe
15:21:47 lost connection
15:21:49 ==> default: Domain is not running. Please run `vagrant up` or `vagrant resume` first.
15:21:49 ==> default: Domain is not running. Please run `vagrant up` or `vagrant resume` first.
15:21:50 ==> default: Domain is not running. Please run `vagrant up` or `vagrant resume` first.
15:21:51 ==> default: Domain is not running. Please run `vagrant up` or `vagrant resume` first.
15:21:53 rake aborted!
15:21:53 CommandError: command ["scp", "-i", "/var/lib/jenkins/workspace/build_Tails_ISO_feature-12599/vagrant/.vagrant/machines/default/libvirt/private_key", "-o", "StrictHostKeyChecking=no", "-o", "UserKnownHostsFile=/dev/null", "vagrant@192.168.121.131:/home/vagrant/amnesia/tails-amd64-feature_12599-3.0-20170530T1428Z-4e95409+testing@60c7405.iso.apt-sources", "vagrant@192.168.121.131:/home/vagrant/amnesia/tails-amd64-feature_12599-3.0-20170530T1428Z-4e95409+testing@60c7405.iso", "vagrant@192.168.121.131:/home/vagrant/amnesia/tails-amd64-feature_12599-3.0-20170530T1428Z-4e95409+testing@60c7405.iso.buildlog", "vagrant@192.168.121.131:/home/vagrant/amnesia/tails-amd64-feature_12599-3.0-20170530T1428Z-4e95409+testing@60c7405.iso.packages", "vagrant@192.168.121.131:/home/vagrant/amnesia/tails-amd64-feature_12599-3.0-20170530T1428Z-4e95409+testing@60c7405.iso.build-manifest", "build-artifacts/"] failed with exit status 1
15:21:53 /var/lib/jenkins/workspace/build_Tails_ISO_feature-12599/Rakefile:71:in `run_command'
15:21:53 /var/lib/jenkins/workspace/build_Tails_ISO_feature-12599/Rakefile:434:in `block in <top (required)>'
15:21:53 Tasks: TOP => build
15:21:53 (See full trace by running task with --trace)
15:21:53 Build step 'Execute shell' marked build as failure
15:21:53 [PostBuildScript] - Execution post build scripts.

One way to list such problems is to look for unusually small ISO images in the artifacts directory:
ssh jenkins.lizard ls -l --sort=size /var/lib/jenkins/jobs/*/builds/*/archive/build-artifacts/*.iso.

I don’t know if it’s mere coincidence, but these 3 builds were run on isobuilder2.

Parent Task: #12531 (closed)

Related issues

• Blocked by #13302 (closed)

Original created by @intrigeri on 12618 (Redmine)