Install and integrate gitlab kubernetes agent server (KAS)

It seems that when the KAS is enabled, you connect to it via a wss:// endpoint, based on what that documentation says:

The agent server for Kubernetes is installed and available on GitLab.com
at wss://kas.gitlab.com. If you use self-managed GitLab, you must
install an agent server or specify an external installation.

If that is true that is always on a wss://, then we likely will need to know what nginx modifications would be needed to support websockets like this.

Is that something you can find out?

Hej @micah, I can try, although I must admit that I never ever administrated a web socket application

Are you using the gitlab omnibus installer ?

Yes, its the omnibus version - however you can 'bring your own' nginx, and not use the built-in omnibus one, which is what we are doing.

Actually, we do already have some websocket pieces setup in the nginx configuration. I did those to make the web terminal work for k8s in our earlier experimentation!

I'm unsure if they are sufficient, but I've enabled the kas option, so maybe you could try it and see if it works?

I created a testgroup, and clicked on the Kubernetes menu entry, but I only see the option to use the deprecated cert-based kubernetes integration (0xacab.org runs gitlab 14.8.4):

https://0xacab.org/groups/testgroup/-/clusters

Whereas at gitlab.com (14.9.0-pre) it looks like this (see the Action Menu options):

Scratch that, on gitlab.com I also don't find any chance to connect my cluster, I'll read through the docs how to properly connect it using KAS.

I see that you created a KAS agent registration token, but it hasn't connected yet. Perhaps this has to be sorted before you can connect a cluster without a certificate?

can you check the service logs for your agent:

kubectl logs -f -l=app=gitlab-agent -n gitlab-kubernetes-agent

the only thing I see on the server in the kas logs is:

2022-03-20_19:09:30.94970 {"level":"error","time":"2022-03-20T12:09:30.949-0700","msg":"PathFetcher","correlation_id":"01FYMBR3E18FDK6QFMB8TGTMMH","grpc_service":"gitlab.agent.configuration_project.rpc.ConfigurationProject","grpc_method":"ListAgentConfigFiles","project_id":"testgroup/kubernetes-agent-setup","error":"RpcError: GetTreeEntries.Recv: RPC failed: .gitlab/agents: rpc error: code = Unknown desc = reference not found"}

Sorry for the delay, but I'm in the process of creating a VM -> k8s cluster -> gitlab agent, which turned out to take longer than expected - who would have thought that I'll let you know once I made it to the top of that list

Could you set the group and the project public? I found this issue:

https://gitlab.com/gitlab-org/gitlab/-/issues/347166#note_763755815

Now I finally installed the agent and configured it with the token I got from registering it in the gitlab UI (and the KAS server URL wss://0xacab.org, hope this is right (wss://kas.0xacab.org didn't resolve).

I see these errors in the agent pod:

{"level":"error","time":"2022-03-21T22:10:37.069Z","msg":"Error handling a connection","mod_name":"reverse_tunnel","error":"Connect(): rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing failed to WebSocket dial: expected handshake response status code 101 but got 400\""}
{"level":"warn","time":"2022-03-21T22:11:41.625Z","msg":"GetConfiguration failed","error":"rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing failed to WebSocket dial: expected handshake response status code 101 but got 400\""}

Could you set the group and the project public? I found this issue: https://gitlab.com/gitlab-org/gitlab/-/issues/347166#note_763755815

I did that, but it didn't help.

I tried connecting with wscat,

gitlab.com said Disconnected (code: 1002, reason: "Expecting "ws-tunnel" subprotocol, got """):

❯ /home/varac/node_modules/.bin/wscat -c wss://kas.gitlab.com
Connected (press CTRL+C to quit)
Disconnected (code: 1002, reason: "Expecting "ws-tunnel" subprotocol, got """)

while 0xacab.org responded with an error: Unexpected server response: 400 (which matches the above logs)

❯ /home/varac/node_modules/.bin/wscat -c wss://0xacab.org    
error: Unexpected server response: 400

I got another agent connected fine to gitlab.com with the same setup (only a different URL/token combination), so i'm pretty sure the issue is on 0xacab's site. Happy to help however I can from my side.

Ok, I've made some changes to nginx so that this seems to work now:

wscat -c wss://0xacab.org/-/kubernetes-agent/
Connected (press CTRL+C to quit)
Disconnected (code: 1002, reason: "Expecting "ws-tunnel" subprotocol, got """)

to get here, I had to set a location and pass the websocket back:

upstream kaswebsocket {
  server 127.0.0.1:8150;
}

map $http_upgrade $connection_upgrade_gitlab_ssl {
    default upgrade;
    ''      close;
}

  location /-/kubernetes-agent/ {
   proxy_pass http://kaswebsocket;
   proxy_http_version 1.1;
   proxy_set_header X-Real-IP $remote_addr;
   proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
   proxy_set_header X-NginX-Proxy true;
   proxy_set_header Host $host;
   proxy_set_header Sec-WebSocket-Protocol $http_sec_websocket_protocol;
   proxy_set_header Sec-WebSocket-Extensions $http_sec_websocket_extensions;
   proxy_set_header Sec-WebSocket-Key $http_sec_websocket_key;
   proxy_set_header Sec-WebSocket-Version $http_sec_websocket_version;
   proxy_set_header Upgrade $http_upgrade;
   proxy_set_header Connection $connection_upgrade_gitlab_ssl;
   proxy_cache_bypass $http_upgrade;
  }

...

On 2022-03-21 15:24:16, Varac (@varac) wrote:

Varac commented on a discussion: #120 (comment 367801)

I tried connecting with wscat,

gitlab.com said Disconnected (code: 1002, reason: "Expecting "ws-tunnel" subprotocol, got """):
❯ /home/varac/node_modules/.bin/wscat -c wss://kas.gitlab.com
Connected (press CTRL+C to quit)
Disconnected (code: 1002, reason: "Expecting "ws-tunnel" subprotocol, got """)
while 0xacab.org responded with an error: Unexpected server response: 400 (which matches the above logs)
❯ /home/varac/node_modules/.bin/wscat -c wss://0xacab.org    
error: Unexpected server response: 400

Thanks ! That did the trick:

I wonder how to communicate that KAS url to 0xacab's users, but maybe the demand is pretty limited anyhow.

Cool! How did you get it to connect? After I made the websocket changes, I tried a few things (eg. made a change to the agent yaml) but couldn't get a change. Maybe you had to start it on your side?

From what I was finding, gitlab puts things on https://kas.gitlab.com otherwise it is typically found at /-/kubernetes-agent (this is the default in the configuration). I think the demand is pretty limited right now, we have barely anyone even using the pages feature, but I suspect because people don't really know what is possible.

Speaking of that, I'm really curious to know what you can do with this, besides just... connecting an agent!

Also, my ! There is a dark mode in gitlab

I'm using flux for gitOps-style deployments on my k8s cluster, and used these two resources to deploy the agent. The only config helm values I needed to add were the kasAddress and token (which I got after registering the agent in gitlab).

---
apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: GitRepository
metadata:
  name: gitlab-k8s-agent
  namespace: flux-system
spec:
  # The interval at which to check the upstream for updates
  interval: 24h
  url: https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent
  ref:
    branch: master
---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
    name: gitlab-k8s-agent-0xacab
    namespace: varac
spec:
    releaseName: gitlab-k8s-agent-0xacab
    chart:
        spec:
            chart: ./build/deployment/gitlab-agent-chart
            sourceRef:
                kind: GitRepository
                name: gitlab-k8s-agent
                namespace: flux-system
    interval: 24h
    values:
        # https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/build/deployment/gitlab-agent-chart/values.yaml
        config:
            kasAddress: wss://0xacab.org/-/kubernetes-agent/
            token: ...

Btw, I still can't access the agent from any other porject than the one I configured it for, which makes it pretty useless for me atm. I think I am experiencing the same as describes here:
KAS CI/CD Tunnel ci_access does not work for projects/groups outside of the group with the agent config project

And I want it because I'd like to enable review apps for a website project - You could still use review apps without a k8s cluster but that's what I'd like to do.

I can't make the agent authorization work - I tried both here and on gitlab.com where I created an issue to track this: https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/issues/243

I finally did get authoization to work properly \o, everything is fine!

closed

Unfortunately I need to reopen this issue. I thought everything was working, but it's not. I configured my agent in varac-projects/kubernetes-agent-setup and gave access to it to varac-projects/testproject.

Until now I could get an use the kubernetes context in my testproject (which I have been struggeling with before), but I'm unable to connect to the cluster, where I get a 426 error code. See https://0xacab.org/varac-projects/testproject/-/jobs/261378:

++ kubectl config get-contexts
$ kubectl config get-contexts
CURRENT   NAME                                                 CLUSTER   AUTHINFO   NAMESPACE
          varac-projects/kubernetes-agent-setup:varac-agent1   gitlab    agent:4    
++ echo '$ kubectl config use-context varac-projects/kubernetes-agent-setup:varac-agent1'
++ kubectl config use-context varac-projects/kubernetes-agent-setup:varac-agent1
$ kubectl config use-context varac-projects/kubernetes-agent-setup:varac-agent1
Switched to context "varac-projects/kubernetes-agent-setup:varac-agent1".
++ echo '$ kubectl version'
++ kubectl version
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.8", GitCommit:"7061dbbf75f9f82e8ab21f9be7e8ffcaae8e0d44", GitTreeState:"clean", BuildDate:"2022-03-16T14:10:06Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}
Error from server: the server responded with the status code 426 but did not return more information

I tried with the same setup on gitlab.com where I could successfully list all pods.

@micah Is there anything in the (nginx|KAS) logs that might help ?

All I see in the KAS logs is this over and over:

2022-03-24_13:30:38.43005 {"level":"debug","time":"2022-03-24T06:30:38.429-0700","msg":"Config: no updates","correlation_id":"01FYY14TXSRPM4X3D1CH95YSKV","grpc_service":"gitlab.agent.agent_configuration.rpc.AgentConfiguration","grpc_method":"GetConfiguration","agent_id":4,"project_id":"varac-projects/kubernetes-agent-setup","commit_id":"17665bec0494fd1c00094cdc314dfdfb153a438d"}

(i have the KAS debugging log level enabled)

This looks ok to me, anything in the nginx logs ? Somehow I suspect nginx to reply with 426, but I might be wrong

hm. I'm not seeing a 426 in the nginx logs (grep varac gitlab_access.log |grep 426 returns nothing).

I noticed you moved the project, maybe that confuses things? Could you try and set it up again from scratch?

Thanks for checking !

I just re-configured it from scratch yesterday after moving already :/ (Meaning removing the agent and re-registered it ). Hugh, i'm a bit clueless right now. I'll setup my own runner so I can connect to the job container and poke a bit around.

I increased the verbosity of the kubectl version cmd and got a more detailled error (see https://0xacab.org/varac-projects/testproject/-/jobs/261542 for full job output):

$ kubectl -v 10 version || /bin/true
++ kubectl -v 10 version
I0324 14:36:01.894108      25 loader.go:372] Config loaded from file:  /builds/varac-projects/testproject.tmp/KUBECONFIG
I0324 14:36:01.894658      25 round_trippers.go:435] curl -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.22.8 (linux/amd64) kubernetes/7061dbb" -H "Authorization: Bearer <masked>" 'https://0xacab.org/-/kubernetes-agent/k8s-proxy/version?timeout=32s'
I0324 14:36:02.527251      25 round_trippers.go:454] GET https://0xacab.org/-/kubernetes-agent/k8s-proxy/version?timeout=32s 426 Upgrade Required in 632 milliseconds
I0324 14:36:02.527274      25 round_trippers.go:460] Response Headers:
I0324 14:36:02.527282      25 round_trippers.go:463]     X-Content-Type-Options: nosniff
I0324 14:36:02.527286      25 round_trippers.go:463]     Server: nginx
I0324 14:36:02.527290      25 round_trippers.go:463]     Date: Thu, 24 Mar 2022 14:36:02 GMT
I0324 14:36:02.527446      25 round_trippers.go:463]     Content-Type: text/plain; charset=utf-8
I0324 14:36:02.527456      25 round_trippers.go:463]     Content-Length: 81
I0324 14:36:02.527525      25 round_trippers.go:463]     Upgrade: websocket
I0324 14:36:02.531758      25 request.go:1181] Response Body: WebSocket protocol violation: Connection header "close" does not contain Upgrade
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.8", GitCommit:"7061dbbf75f9f82e8ab21f9be7e8ffcaae8e0d44", GitTreeState:"clean", BuildDate:"2022-03-16T14:10:06Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}
I0324 14:36:02.534095      25 helpers.go:217] server response object: [{
  "metadata": {},
  "status": "Failure",
  "message": "the server responded with the status code 426 but did not return more information",
  "details": {
    "causes": [
      {
        "reason": "UnexpectedServerResponse",
        "message": "WebSocket protocol violation: Connection header \"close\" does not contain Upgrade"
      }
    ]
  },
  "code": 426
}]
F0324 14:36:02.534117      25 helpers.go:116] Error from server: the server responded with the status code 426 but did not return more information

The only issue mentioning a 426 Upgrade Required error I found is https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/issues/198

I copied the kubeconfig from that container to my laptop and am able to reproduce this locally. In case you need that kubeconfig file to try yourself let me know:

❯ export KUBECONFIG=/home/varac/.kube/kas-0xacab.yml
❯ kc -v 5 version                                   
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.5", GitCommit:"c285e781331a3785a7f436042c65c5641ce8a9e9", GitTreeState:"clean", BuildDate:"2022-03-17T03:51:43Z", GoVersion:"go1.17.8", Compiler:"gc", Platform:"linux/amd64"}
I0324 15:55:33.719972  619850 helpers.go:219] server response object: [{
  "metadata": {},
  "status": "Failure",
  "message": "the server responded with the status code 426 but did not return more information",
  "details": {
    "causes": [
      {
        "reason": "UnexpectedServerResponse",
        "message": "WebSocket protocol violation: Connection header \"close\" does not contain Upgrade"
      }
    ]
  },
  "code": 426
}]
Error from server: the server responded with the status code 426 but did not return more information

I0324 14:36:01.894658 25 round_trippers.go:435] curl -v -XGET -H "Accept: application/json, /" -H "User-Agent: kubectl/v1.22.8 (linux/amd64) kubernetes/7061dbb" -H "Authorization: Bearer " 'https://0xacab.org/-/kubernetes-agent/k8s-proxy/version?timeout=32s'

and in the header is:

Upgrade: websocket

Then this is the problem it is returning:

I0324 14:36:02.531758 25 request.go:1181] Response Body: WebSocket protocol violation: Connection header "close" does not contain Upgrade

What I have configured in nginx is this:

map $http_upgrade $connection_upgrade_gitlab_ssl {
    default upgrade;
    ''      close;
}

This is the nginx map module which lets you create variables in Nginx’s configuration file whose values are conditional — that is, they depend on other variables’ values.

I don't fully understand this, but I believe it is mapping the $http_upgrade (which in this case is 'websocket') to something else, maybe by default do an upgrade, but if its the empty string, do a close.

I also have configured proxy_set_header Connection $connection_upgrade_gitlab_ssl; but I find elsewhere that people just have this set to proxy_set_header Connection upgrade; so I'm going to try to set that instead.

can you check on it now?

Unfortunately it is still the same: https://0xacab.org/varac-projects/testproject/-/jobs/261548

We really need gitlab to provide us with a proper nginx config, because I'm really shooting in the dark here.

also, i dont have permissions to see that job log

also, i dont have permissions to see that job log

sorry, I added you as member, do you see it now ?

We really need gitlab to provide us with a proper nginx config, because I'm really shooting in the dark here.

Yeah that's annoying... Sorry for the trouble... Should I create an upstream issue and see if the gitlab team can help here ?

That might be good - you seem to be getting good responses from the people who are core on this specific setup! I mentioned the nginx config I was using in https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6685#note_884011944 and asked for clarification there, but I think maybe an actual issue might be better.

reopened

Now it seems to say:

Response Body: WebSocket protocol violation: Upgrade header "" does not contain websocket

Where before it said:

WebSocket protocol violation: Connection header \"close\" does not contain Upgrade

https://gitlab.com/gitlab-org/gitlab/-/issues/351477 seems related

Is there a way I can trigger the error, so I can try different things and see the results?

I just sent you the kubeconfig via email, you should be able to reproduce with kubectl -v 5 version (or even higher verbosity)

RFC6455 says, "If the response lacks a |Connection| header field or the |Connection| header field doesn't contain a token that is an ASCII case-insensitive match for the value "Upgrade", the client MUST Fail the WebSocket Connection."

So something over there is setting the Connection header to 'close' which causes this to happen. This seems to not be something caused by the nginx configuration at all but something the kas-agent is doing?

If I strace that process while doing the connection, I see:

[pid 12924] write(16, "HTTP/1.1 426 Upgrade Required\r\nContent-Type: text/plain; charset=utf-8\r\nServer: gitlab-kas/v14.9.0/ffe1935\r\nUpgrade: websocket\r\nX-Content-Type-Options: nosniff\r\nDate: Thu, 24 Mar 2022 16:15:00 GMT\r\nContent-Length: 81\r\nConnection: close\r\n\r\nWebSocket protoco"..., 320 <unfinished ...>

So what do we know? We know that this worked the other day, and then it stopped working for some reason. We didn't change anything over here, so I'm wondering if there is something you did that is making it not work right.

It never ever worked completely. In the first phase I was having authorization issues which resulted in my testproject CI container not having any kubeconfig file, which are solved now. I thought with this the kubectl cluster access was working, but it isn't. So there were nowhen a state where everything worked :/

One thing I can think of is you had tried to make the test project public, but it didn't solve the problem, but then you moved it to your area, and perhaps that being private is the problem?

Can you try and set this up again in the test project?

then you moved it to your area, and perhaps that being private is the problem? Can you try and set this up again in the test project?

The new group https://0xacab.org/varac-projects is public, and so are the two contained project, so there's nothing to improve here :/ I also removed the agent and re-registered it again without success. So I'll create an upstream issue now, and thanks again for bearing with me !

Here it is, hope I'll get help as quick as last time: https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/issues/245

I now get a different output from kubectl version:

❯ kc -n varac exec -it -c build runner-6vdudhzk-project-3939-concurrent-0zz5qq -- sh -c "KUBECONFIG=/builds/varac-projects/testproject.tmp/KUBECONFIG kubectl -v 5 version"          
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.8", GitCommit:"7061dbbf75f9f82e8ab21f9be7e8ffcaae8e0d44", GitTreeState:"clean", BuildDate:"2022-03-16T14:10:06Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}
error: unable to parse the server version: invalid character '<' looking for beginning of value
command terminated with exit code 1

I increased the kubectl verbosity to 10 in https://0xacab.org/varac-projects/testproject's CI runs, here's the last one: https://0xacab.org/varac-projects/testproject/-/jobs/261791. It looks like that the GET for https://0xacab.org/-/kubernetes-agent/k8s-proxy/version?timeout=32s redirects to the sigin page. @micah can you read the CI logs now ? I scheduled them to run every hour for now.

Also reproducible by using the kubeconfig I sent you with KUBECONFIG=~/.kube/kas-0xacab.yml kubectl -v 10 version

yeah, i updated that gitlab.com ticket with this information. I don't know why it is redirecting to the sign-in page, but it usually does that when there is not a proper authentication.

@micah can you read the CI logs now ? I scheduled them to run every hour for now.

I can read the CI logs now.

We finally made it work !

closed

Install and integrate gitlab kubernetes agent server (KAS)

Designs

Child items 0

Activity