Jenkins ISO builders are highly unreliable
Since a couple weeks, lots of ISO build jobs fail due to Jenkins crashing (OOM killer?).
The first affected builds are:
- https://jenkins.tails.boum.org/view/RM/job/build_Tails_ISO_stable/1284/ (Nov 16, 2016 7:15:00 AM)
- https://jenkins.tails.boum.org/view/RM/job/build_Tails_ISO_devel/2063/ (Nov 17, 2016 11:36:13 AM)
The only relevant changes I see immediately before this date are allocating a bit more huge pages on lizard, and giving each ISO builder VM a bit more RAM (8912896 KiB → 9437184 KiB). Then, on November 25 bertagaz gave them 11461632 KiB.
This should be fixed / workarounded by #11972 (closed), but depending on the timing of that work, we may need to fix the problem earlier, and in a different way.
Related issues
- Related to #11858 (closed)
- Related to #11972 (closed)
Original created by @intrigeri on 12009 (Redmine)