Investigate why the CI is sometimes failling

The CI is failing from time to time, but only on the Large Richard runner (and not on Whitewhale).

@pollo, the (amazing) admin of Large Richard is investigating

mentioned in issue riseup/0xacab#57 (closed)

Here are 2 things I think might be causing the problem:

1. Whitewhale and Large Richard aren't running the same gitlab-runner version. Whitewhale is using 9.5.1 whereas Large Richard uses 11.3.0. A 2 major version diff is quite a lot and might be the source of the inconsistency. This issue has been reported here.

2. Whitewhale and Large Richard have different caching policies. Large Richard purges docker images on a daily basis, whereas Whitewhale might keep them longer. This mean running tests on archlinux:latest might have different results since the tests might run on two different images. If you don't explicitly need to test on bleeding edge, running tests on stable versions or explicit versions might be a better idea.

Whitewhale is now tagged properly, so a hotfix for your current problem would be to tag it (add a tags: - whitewhale to your tests) so the tests only run on Whitewhale and not on Large Richard.

This is been done in 7f629ed2: the testsuite is now green, thank you

changed milestone to %0.5.0 - Slug

mentioned in merge request !17 (merged)

mentioned in issue #61 (closed)

changed milestone to %0.7.0 - Roaster

changed milestone to %0.8.0 - Lobster

Ok, so I'm not 100% crazy. Trying to debug this issue I found out that:

1 - There is indeed a problem with Large Richard with regards to tests:debian_with_bubblewrap. I'm going to need help on this one, as I'm really not a Docker person.

It seems bubblewrap (which I've never used) does some unprivileged containers magic and requires privileges that are missing on that box. The error message that keeps coming back on that job is:

No permissions to creating new namespace, likely because the kernel does not allow non-privileged user namespaces. On e.g. debian this can be enabled with 'sysctl kernel.unprivileged_userns_clone=1'.

User namespace creation was indeed disabled at first, but I changed stuff and it isn't the case anymore:

root@large-richard:/# cat /etc/sysctl.d/00-local-userns.conf 
kernel.unprivileged_userns_clone=1

root@large-richard:/# cat /proc/sys/kernel/unprivileged_userns_clone
1

I can't replicate this on Gitlab.com, so this is indeed something specific with the way Large Richard was built. Any idea? It's a VM on a bare metal KVM hypervisor running vanilla Debian Stretch.

2 - The bugs with tests:archlinux and tests:fedora can be replicated on Gitlab.com.

• tests:archlinux: large-richard VS gitlab.com
• tests:fedora: large-richard VS gitlab.com

I know it's working on Whitewhale, but I'd suggest this is caused by a change in how Gitlab CI behaves. Whitewhale is running old versions of Gitlab CI and Docker (it's a Jessie machine), whereas Large Richard is an up to date Stretch machine.

I won't actively be debugging this, as I don't really have the time to play with your .gitlab-ci.yaml file, but it looks like some dependencies aren't passed correctly from the build to the test stage.

Maybe try to spool an actual Arch or Fedora VM and see if it's a real bug?

@pollo I ran into this as well: For now, I'm running the runner in privileged mode, as I do already for runner which need to provide Docker-in-Docker. What made me wondering: It seems, this job works in whitewhale, but AFAIK, this runner doesn't run in privileged mode, though, I might be wrong on this.

Anyway, it might be interesting to ask the whitewhale people regarding this, and a possible difference in the configuration. If you do, please tell me, I'm curious.

Closing this, as we're now using a private runner.

closed