Environment deploy to mail.bitmask.net is stalling for unknown reasons

i checked beluga, there is plenty of disk space, so it is not that

I tried to re-deploy - it seems the same thing is happening. I logged into alpaca to try and see if there was some long running process related to deployment, but there was nothing

Even more strange, I see a lot of the CI jobs running as user 'check_mk':

root@Beluga:~# ps auxww |grep check_mk
check_mk  3194  0.0  0.0  13248  2804 ?        Ss   21:35   0:00 bash -c set -o pipefail; stdbuf -oL -eL ./ci-build.sh | ts
check_mk  3195  0.0  0.0  15468  3224 ?        S    21:35   0:00 /bin/bash ./ci-build.sh
check_mk  3196  0.0  0.1  26236  6916 ?        S    21:35   0:00 /usr/bin/perl /usr/bin/ts
check_mk  4242  0.0  0.0  13248  2812 ?        Ss   21:40   0:00 bash -c /usr/bin/unbuffer /bin/bash -o pipefail ./ci-build.sh | /usr/bin/ts
check_mk  4243  0.0  0.0  41860  5432 ?        Sl   21:40   0:00 tclsh8.6 /usr/bin/unbuffer /bin/bash -o pipefail ./ci-build.sh
check_mk  4244  0.0  0.1  26236  6976 ?        S    21:40   0:00 /usr/bin/perl /usr/bin/ts
check_mk  4246  0.0  0.0  13356  3088 pts/0    Ss+  21:40   0:00 /bin/bash -o pipefail ./ci-build.sh
check_mk  4295  0.0  0.6 247088 36908 pts/0    Sl+  21:40   0:00 ruby2.1 /builds/leap/platform/tests/platform-ci/vendor/ruby/2.1.0/bin/leap -v2 --yes deploy demomail
check_mk  6988  6.2  0.8 945100 52160 ?        Sl   22:00   0:01 ruby2.1 /builds/micah/platform/tests/platform-ci/vendor/ruby/2.1.0/bin/leap -v2 --yes deploy single
root      7117  0.0  0.0  12732  2264 pts/0    S+   22:00   0:00 grep check_mk
root@Beluga:~#

Oh, i think the uid weirdness is because those commands are being run inside of the docker namespace as 'cirunner' and not related to this problem

I logged into the running docker container and I see this running:

root@runner-7c4e02b0-project-509-concurrent-0:/var/lib/gems/2.1.0/gems/bundler-1.15.0# ps auxww
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0  21848  3344 ?        Ss   21:40   0:00 /bin/bash
root         8  0.0  0.0  21952  2508 ?        S    21:40   0:00 /bin/bash
root        10  0.0  0.0  46360  3068 ?        S    21:40   0:00 su -c /usr/bin/unbuffer /bin/bash -o pipefail ./ci-build.sh | /usr/bin/ts cirunner
cirunner    11  0.0  0.0  13248  2812 ?        Ss   21:40   0:00 bash -c /usr/bin/unbuffer /bin/bash -o pipefail ./ci-build.sh | /usr/bin/ts
cirunner    12  0.0  0.0  41860  5432 ?        Sl   21:40   0:00 tclsh8.6 /usr/bin/unbuffer /bin/bash -o pipefail ./ci-build.sh
cirunner    13  0.0  0.1  26236  6976 ?        S    21:40   0:00 /usr/bin/perl /usr/bin/ts
cirunner    15  0.0  0.0  13356  3088 pts/0    Ss+  21:40   0:00 /bin/bash -o pipefail ./ci-build.sh
cirunner    63  0.0  0.6 247088 36908 pts/0    Sl+  21:40   0:01 ruby2.1 /builds/leap/platform/tests/platform-ci/vendor/ruby/2.1.0/bin/leap -v2 --yes deploy demomail
root       141  0.0  0.0  21908  3720 ?        Ss   22:04   0:00 bash
root       223  0.0  0.0  46368  3188 ?        S    22:08   0:00 su - cirunner
cirunner   224  0.0  0.0  20264  3228 ?        S+   22:08   0:00 -su
root       237  0.0  0.0  21908  3760 ?        Ss   22:11   0:00 bash
root       298  0.0  0.0  19108  2504 ?        R+   22:22   0:00 ps auxww
root@runner-7c4e02b0-project-509-concurrent-0:/var/lib/gems/2.1.0/gems/bundler-1.15.0#

The process id 63 doesn't appear to be doing much... strace tells me its getting some connection timeouts... but I can't tell much else.

I wanted to try and run the leap command myself to try and get some additional debugging information, but I can't seem to do so:

cirunner@runner-7c4e02b0-project-509-concurrent-0:/builds/leap/platform/tests/platform-ci/bitmask$ ruby2.1 /builds/leap/platform/tests/platform-ci/vendor/ruby/2.1.0/bin/leap -v6 --yes deploy demomail
/usr/lib/ruby/2.1.0/rubygems/dependency.rb:298:in `to_specs': Could not find 'leap_cli' (>= 0) among 10 total gem(s) (Gem::LoadError)
	from /usr/lib/ruby/2.1.0/rubygems/dependency.rb:309:in `to_spec'
	from /usr/lib/ruby/2.1.0/rubygems/core_ext/kernel_gem.rb:53:in `gem'
	from /builds/leap/platform/tests/platform-ci/vendor/ruby/2.1.0/bin/leap:22:in `<main>'
cirunner@runner-7c4e02b0-project-509-concurrent-0:/builds/leap/platform/tests/platform-ci/bitmask$

Ok, I've done more digging... The reason it is failing is because it gets stuck on this prompt:

The platform is pinned to branch 'master' by the `platform.branch` property in provider.demomail.json, but the current branch is '' (for directory '/builds/leap/platform')
hint: use --force to skip this prompt.

Indeed, the checkout is not on the master branch:

cirunner@runner-7c4e02b0-project-509-concurrent-0$ cd /builds/leap/platform
cirunner@runner-7c4e02b0-project-509-concurrent-0:/builds/leap/platform$ git branch
* (detached from c0ddb0d)
  master
cirunner@runner-7c4e02b0-project-509-concurrent-0:/builds/leap/platform$

This issue https://gitlab.com/gitlab-org/gitlab-ce/issues/19421 says that is normal because the runner runs in a detached head state.

The current checkout that is trying to run is c0ddb0da.

So the options are:

pass --force to skip the prompt
do a git checkout master in the ci-build.sh to make sure its on the right branch

I decided to do the second option, and have pushed a branch to fix this:

https://0xacab.org/micah/platform/commits/fix_mail_deploy

mentioned in merge request !113 (closed)

closed via commit 55b784f2

@micah great, thx !

reopened

Damn, same behaviour for demo.bitmask.net. I deployed to demo.bitmask.net by clicking the button from gitlab CI, but the deploy job hangs: https://0xacab.org/leap/platform/-/jobs/18030

added CI label

changed milestone to %Platform 0.10.1

closed via commit d66bbeb0

changed milestone to %Platform 0.10

mentioned in commit 55b784f2

mentioned in commit d66bbeb0

Environment deploy to mail.bitmask.net is stalling for unknown reasons

Designs

Child items ...

Activity