Test suite became unreliable on Jenkins: OOM kills QEMU, OpenJDK memory allocation failure aborts the test suite run

OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x000000073d700000, 16777216, 0) failed; error='Cannot allocate memory' (errno=12)

For example:

• https://jenkins.tails.boum.org/job/test_Tails_ISO_stable/1988/
• https://jenkins.tails.boum.org/job/test_Tails_ISO_feature-tor-nightly-master/748/

Initial investigation:

• No trace of this on the Jenkins master (jenkins.log, Journal) so the problem must be on the isotesters.
• It’s hard to investigate this on isotesters as we reboot them next time a test job is scheduled on them. Will need to make some logs persist reboots.
• Every time it happens, it seems that the last action of the test suite used Sikuli (Java).
• The Java VM has its own internal memory management, that can be tweaked in various ways. But the error message looks like the java process asked the OS to allocate some memory and its request was denied.

Ideas:

• Check if I see this on my local Jenkins.
• Compare RAM allocated to isotesters on my local Jenkins vs. lizard.
• It could simply be that Cucumber 2.4.0 or Jenkins slave 2.x use more RAM → bump isotesters memory a bit and see?

Related issues

• Related to #10068 (closed)
• Related to #11175 (closed)
• Related to #8681 (closed)
• Blocks #13284
• Blocked by #17103 (closed)

Original created by @intrigeri on 17088 (Redmine)