Entanglements

Quarks Links can provide secrets to pods outside the deployment and can consume existing secrets

Also known as “Quarks Links” - they provide a way to share/discover information between BOSH and Kube Native components.

Using k8s Native Values in BOSH Deployments

Note

Native -> BOSH

In this case, the native component is a provider, and the BOSH component is a consumer.

We construct link information from the native resources like this:

BOSH Link	Native	Description
address	Service	DNS address of a k8s service annotated `quarks.cloudfoundry.org/provides = LINK_NAME`
azs	N/A	not supported
properties		properties retrieved from a secret annotated `quarks.cloudfoundry.org/provides = LINK_NAME`
instances.name	Pod	name of pod selected by the k8s service that’s annotated `quarks.cloudfoundry.org/provides = LINK_NAME`
instances.id	Pod	pod uid
instances.index	Pod	set to a value 0-(pod replica count)
instances.az	N/A	not supported
instances.address	Pod	ip of pod
instances.bootstrap	Pod	set to true if index == 0

If multiple secrets or services are found with the same link information, the operator should error

Example

When a job consumes a link, it will need a section like this in the in its job spec (job.MF), e.g. the nats release:

1
2
3


consumes:
- name: nats
  type: nats

The deployment manifests needs to explicitly consume the link:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33


---
apiVersion: v1
kind: Secret
metadata:
  name: cf-operator-testing-manifest
stringData:
  manifest: |
    ---
    releases:
    - name: quarks-gora
      version: "0.0.15"
      url: ghcr.io/cloudfoundry-incubator
      stemcell:
        os: SLE_15_SP1
        version: 27.10-7.0.0_374.gb8e8e6af
    instance_groups:
    - name: quarks-gora
      instances: 2
      lifecycle: errand
      jobs:
      - name: smoke-tests
        release: quarks-gora
        consumes:
          quarks-gora: {from: quarks-gora}
---
apiVersion: quarks.cloudfoundry.org/v1alpha1
kind: BOSHDeployment
metadata:
  name: cfo-test-deployment
spec:
  manifest:
    name: cf-operator-testing-manifest
    type: secret

Complete source code: https://github.com/cloudfoundry-incubator/quarks-operator/blob/master/docs/examples/quarks-link/native-to-bosh/boshdeployment.yaml

To fulfill the link we need to create a k8s secret, like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


apiVersion: v1
kind: Secret
metadata:
  name: somesecret
  labels:
    quarks.cloudfoundry.org/deployment-name: "cfo-test-deployment"
  annotations:
    quarks.cloudfoundry.org/provides: '{"name":"quarks-gora","type":"quarks-gora"}'
stringData:
  link: |
    quarks-gora.ssl: false
    quarks-gora.port: "1234"
    text_message: admin

Complete source code: https://github.com/cloudfoundry-incubator/quarks-operator/blob/master/docs/examples/quarks-link/native-to-bosh/link-secret.yaml

The quarks-gora release can then use the links in its eruby templates:

"<%= p("quarks-gora.ssl") %>"

Furthermore, if there is a matching k8s service, it will be used in the link:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


apiVersion: v1
kind: Service
metadata:
  labels:
    quarks.cloudfoundry.org/deployment-name: "cfo-test-deployment"
  annotations:
    quarks.cloudfoundry.org/link-provider-name: quarks-gora
  name: testservice
spec:
  ports:
  - name: client
    port: 2181
    protocol: TCP
    targetPort: client
  selector:
    app: linkpod
  type: ClusterIP
status:
  loadBalancer: {}

Complete source code: https://github.com/cloudfoundry-incubator/quarks-operator/blob/master/docs/examples/quarks-link/native-to-bosh/link-service.yaml

Using this service, I should be able to use link("quarks-gora").address, and I should get a value of testservice.

This service selects for Pods that have the label app: linkpod. The instances array should be populated using information from these pods.

If the secret is changed, consumers of the link are automatically restarted.

If the service is changed, or the list of pods selected by the service is changed, consumers of the link are automatically restarted.

Using BOSH Variables in k8s Pods

Note

BOSH -> Native

In this case, the BOSH component is a provider, and the native component is a consumer. The native component is a pod, which might belong to a deployment or statefulset.

The operator creates link secrets for all providers in a BOSH deployment. Each secret contains a flattened map with the provided properties:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


apiVersion: v1
kind: Secret
metadata:
  name: link-nats-nats
  labels:
    quarks.cloudfoundry.org/container-name=nats
    quarks.cloudfoundry.org/deployment-name=nats-deployment
    quarks.cloudfoundry.org/entanglement=true
    quarks.cloudfoundry.org/remote-id=nats
data:
  nats.password: YXBwYXJlbnRseSwgeW91Cg==
  nats.port: aGF2ZSB0b28K
  nats.user: bXVjaCB0aW1lCg==

If a pod is annotated with the following:

1
2
3


quarks.cloudfoundry.org/consumes: '[{"name":"nats","type":"nats"}]'
quarks.cloudfoundry.org/deployment: nats-deployment
quarks.cloudfoundry.org/restart-on-update: "true"

The operator will mutate the pod to:

mount the link secrets as /quarks/link/DEPLOYMENT/<type>-<name>/<key>
add an environment variable for each key in the secret data mapping: LINK_<key>

The <name> and <type> are the respective link type and name. For example, the nats release uses nats for both the name and the type of the link. The <key> describes the BOSH property, flattened (dot-style), for example nats.password. The key name is modified to be upper case and without dots in the context of an environment variable, therefore nats.password becomes LINK_NATS_PASSWORD in the container.

If link information changes and the pod has the ‘restart-on-update’ annotation, the operator will trigger an update (restart) of the deployment or statefulset owning the pod.

Example

The following BOSH deployment, will create secrets for all links.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46


---
apiVersion: v1
kind: Secret
metadata:
  name: nats-manifest
stringData:
  manifest: |
    ---
    name: nats-deployment
    releases:
    - name: nats
      version: "33"
      url: ghcr.io/cloudfoundry-incubator
      stemcell:
        os: SLE_15_SP1
        version: 27.8-7.0.0_374.gb8e8e6af
    instance_groups:
    - name: nats
      instances: 1
      jobs:
      - name: nats
        release: nats
        properties:
          nats:
            user: admin
            password: onetwothreefour
          quarks:
            ports:
            - name: "nats"
              protocol: "TCP"
              internal: 4222
            - name: "nats-routes"
              protocol: TCP
              internal: 4223
    variables:
    - name: nats_password
      type: password
---
apiVersion: quarks.cloudfoundry.org/v1alpha1
kind: BOSHDeployment
metadata:
  name: nats-deployment
spec:
  manifest:
    name: nats-manifest
    type: secret

Complete source code: https://github.com/cloudfoundry-incubator/quarks-operator/blob/master/docs/examples/quarks-link/boshdeployment.yaml

The k8s deployment looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


apiVersion: apps/v1
kind: Deployment
metadata:
  name: entangled-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      example: owned-by-dpl
  template:
    metadata:
      annotations:
        quarks.cloudfoundry.org/consumes: '[{"name":"nats","type":"nats"}]'
        quarks.cloudfoundry.org/deployment: nats-deployment
        quarks.cloudfoundry.org/restart-on-update: "true"
      labels:
        example: owned-by-dpl
      name: entangled
    spec:
      containers:
      - command:
        - sleep
        - "3600"
        image: busybox
        imagePullPolicy: Always
        name: busybox
      restartPolicy: Always
      terminationGracePeriodSeconds: 1

Complete source code: https://github.com/cloudfoundry-incubator/quarks-operator/blob/master/docs/examples/quarks-link/entangled-dpl.yaml

The nats release has the corresponding provides: section.

Last modified May 25, 2021: Fix code display in release docs (2bc24a5)