Rotation and Restart

Rotating a Certificate and restarting affected pods

Setup

With a default Quarks Operator installation, that uses the ‘staging’ namespace. First, create the certificate that will be used by the statefulset.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
---
apiVersion: quarks.cloudfoundry.org/v1alpha1
kind: QuarksSecret
metadata:
  name: gen-gora-cert
spec:
  request:
    certificate:
      alternativeNames:
        - foo.com
        - '*.foo.com'
      commonName: routerSSL
      isCA: false
      signerType: cluster
  secretName: gora-cert
  type: certificate

This statefulset starts quarks-gora directly from the docker image, not from the BOSH release.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
apiVersion: v1
kind: Service
metadata:
  name: gora
  labels:
    app: quarks-gora
spec:
  ports:
  - port: 8443
    name: ssl
  clusterIP: None
  selector:
    app: quarks-gora
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: quarks-gora
spec:
  selector:
    matchLabels:
      app: quarks-gora
  serviceName: "gora"
  template:
    metadata:
      labels:
        app: quarks-gora
      annotations:
        quarks.cloudfoundry.org/restart-on-update: "true"
    spec:
      containers:
      - name: gora
        image: ghcr.io/cfcontainerizationbot/quarks-gora:latest
        imagePullPolicy: Never
        command:
        - /usr/local/bin/quarks-gora
        volumeMounts:
        - name: cert
          readOnly: true
          mountPath: "/etc/gora"
        env:
        - name: SSL
          value: "true"
        - name: PORT
          value: "443"
        - name: SERVER_CRT
          value: "/etc/gora/certificate"
        - name: SERVER_KEY
          value: "/etc/gora/key"
        - name: KEYDATA
          valueFrom:
            secretKeyRef:
              name: gora-cert
              key: private_key
      volumes:
      - name: cert
        secret:
          secretName: gora-cert
          items:
          - key: private_key
            path: key
          - key: certificate
            path: certificate

Note that the statefulset has the quarks.cloudfoundry.org/restart-on-update: "true" annotation, to opt in to restarts.

Rotation

By creating the rotation config, the secret gets updated.

1
2
3
4
5
6
7
8
9
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: rotate
  labels:
    quarks.cloudfoundry.org/secret-rotation: "true"
data:
  secrets: '["gen-gora-cert"]'

The quarks-restart controller detects the change and restarts the statefulsets of annotated pods.

You can run kubectl run -it --rm --restart=Never curl --image=curlimages/curl sh to spawn a shell inside KinD and access gora at https://gora.staging.

Some connections might hang and fail as the statefulset restarts and the pod IPs change.