We've successfully implemented user authentication in our TKG clusters. Now, we'd like to update the client ID/secret for our TKG Management Cluster.
TKG uses Pinniped to support authentication, including to OIDC providers. However, exactly which Kubernetes object we need to update/reconfigure is not immediately obvious. Fortunately, we only need to update one object, and the cluster takes care of the rest for us.
At a high level, here's what we'll do:
- Find the
pinniped-addonsecret, which contains our OIDC connection details, encoded in base64
- Decode the payload, update our configs
- Re-encode the configs, and update the payload in the
- Observe and confirm success
As a reminder, everything we're doing here should be done to the Management Cluster. The downsteram workload clusters will inherit your update configs (yay, ClusterAPI!).
Find the Existing Pinniped Config
NOTE: I highly recommend k9s when operating with your Kubernetes clusters. Moving around in a CLI has never been such a breeze for me, and it makes discoverability and tactical operations much faster.
In the tkg-system namespace, there's a secret called
$CLUSTER_NAME-pinniped-addon. That secret has your OIDC configs in a base64-encoded payload. You'll want to fetch that secret. So if your k8s cluster is named "donkey", you'll do:
kubectl get secret donkey-pinniped-addon -n tkg-system -o yaml
The important part is the section
data.values.yaml, as seen here:
apiVersion: v1 data: values.yaml: ArbleGarbleRepeatingThisShouldBeBase64StuffsArbleGarbleRepeatingThisShouldBeBase64StuffArbleGarbleRepeatingThisShouldBeBase64StuffArbleGarbleRepeatingThisShouldBeBase64StuffArbleGarbleRepeatingThisShouldBeBase64StuffArbleGarbleRepeatingThisShouldBeBase64StuffArbleGarbleRepeatingThisShouldBeBase64StuffArbleGarbleRepeatingThisShouldBeBase64StuffArbleGarbleRepeatingThisShouldBeBase64StuffArbleGarbleRepeatingThisShouldBeBase64StuffArbleGarbleRepeatingThisShouldBeBase64Stuffs kind: Secret metadata:
Decode and Update Pinniped Config
Take that base64 payload copy/paste into a file, then decode it. Suppose you put it in a file called
cat pinniped-config.b64 | base64 -d > pinniped-config.yml
Now edit that file to update your credentials. Mine looks something like this:
#@data/values #@overlay/match-child-defaults missing_ok=True --- infrastructure_provider: aws tkg_cluster_role: management . . . identity_management_type: oidc pinniped: cert_duration: 2160h cert_renew_before: 360h supervisor_svc_endpoint: https://0.0.0.0:31234 supervisor_ca_bundle_data: ca_bundle_data_of_supervisor_svc supervisor_svc_external_ip: 0.0.0.0 supervisor_svc_external_dns: null upstream_oidc_client_id: REDACTED # <---- here's your Client ID upstream_oidc_client_secret: REDACTED # <---- here's your Client Secret upstream_oidc_issuer_url: https://some-okta-thing.com
Re-Encode and Upload
Once you've got things just the way you like it, it's time to update the values for real in the k8s management cluster
cat pinniped-config.yml | base64 -w 0
That will base64-encode the values and dump them to
stdout. Note, that we're supplying the
-w 0 flag to be sure our value is all on one line. We cannot have any line breaks in this value.
Now edit the secret where you originally got the config:
kubectl edit secret donkey-pinniped-addon -n tkg-system
REMEMBER to be sure it's all one line, no extra line breaks, no extra indention. Details matter here.
After you save & quit, head on over to the
pinniped-supervisor namespace and watch for the
post-deploy pod to recreate. You'll know this is happening because its age will be very young.
NAME AGE pinniped-post-deploy-job-557j7 1m pinniped-supervisor-bdfdcbbdf-cxrfj 1m pinniped-supervisor-bdfdcbbdf-zcx9w 1m
The status of that
pinniped-post-deploy pod should eventually turn to "Complete" and the logs should end with something like the following:
Successfully configured the Pinniped
Super simple, right? At this point, you have successfully reconfigured your Pinniped credentials! Test things out. If you still have issues, retrace your steps more carefully to see what went wrong. Remember, the details (indention in yaml, namespaces, encodings) matter.