To implement the BOSH behaviour during rollout of a new manifest, an special controller is introduced.
instance_groupsinto Kubernetes resources.
The QuarksStatefulSet component can be understood as the set of controllers responsible for translating the BOSH manifest
instance_groups into Kubernetes resources.
The QuarksStatefulset component is a categorization of a set of controllers, under the same group. Inside the QuarksStatefulset component, we have a set of 2 controllers together with one separate reconciliation loop per controller.
Figure 1 illustrates a QuarksStatefulset component diagram that covers the set of controllers it uses.
Fig. 1: The QuarksStatefulset component
Fig. 2: The QuarksStatefulset controller
This controller will generate a Kubernetes statefulset for each
instance_group defined in the BOSH manifest. This
Statefulset will also include a set of Kubernetes services, so that each component can be accessed on specific ports.
Watches in sts controller
Reconciliation in sts controller
Will generate versioned
Statefulsets with the required data to make all jobs of the
Scaling Restrictions (not implemented)
Ability to set restrictions on how scaling can occur: min, max, odd replicas.
Automatic Restart of Containers
When an env value or mount changes due to a
Secret change, containers are restarted.
The operator watches all the
Secrets referenced by the
StatefulSet, and automatically performs the update, without extra workarounds.
Exposing QuarksStatefulSets Publicly
quarksstatefulsets is similar to exposing
statefulsets in kubernetes. A Kubernetes service makes use of labels to select the pods which should be in the service. We need to use two labels to group the pods of a single instance group.
Following is the example which creates a service with type ClusterIP for a single instance group named
nats in deployment
nats-deployment for exposing port 4222.
Complete example can be found here.
Though, by default, quarks creates three services of type ClusterIP as defined here for any instance group.
For creating a service type LoadBalancer, we just need to change the
LoadBalancer in the above example. The LoadBalancer Ingress is your public IP specified in the output of this command
kubectl describe service nats-service.
Ingress doesn’t use any labels but just sits on top of services and acts as a smart router. You can create services of different types based on the above examples and use them as values in the ingress Kubernetes spec. An example of Ingress can be found here
For more information about Kubernetes services, we recommend you to read this.
Extended Upgrade Support
When an update needs to happen, a second
StatefulSet for the new version is deployed, and both coexist until canary conditions are met.
Annotated with a version (auto-incremented on each update). The annotation key is
Ability to upgrade even though
StatefulSet pods are not ready.
Detects if StatefulSet versions are running
During upgrades, there is more than one
StatefulSet version for an
QuarksStatefulSet resource. The operator lists available versions and keeps track of which are running.
A running version means that at least one pod that belongs to a
StatefulSet is running. When a version n is running, any version lower than n is deleted.
The controller continues to reconcile until there’s only one version.
zones key defines the availability zones the
QuarksStatefulSet needs to span.
zoneNodeLabel defines the node label that defines a node’s zone.
The default value for
The example below defines an
QuarksStatefulSet that should be deployed in two availability zones, us-central1-a and us-central1-b.
QuarksStatefulSet controller creates one
StatefulSet version for each availability zone, and adds affinity information to the pods of those
If zones are set for an
QuarksStatefulSet, the following occurs:
The name of each created
StatefulSetis generated as
<quarks statefulset name>-z<index of az>.
Podsare labeled with the following:
quarks.cloudfoundry.org/az-index: "0" quarks.cloudfoundry.org/az-name: "us-central1-a"
Podsare annotated with an ordered JSON array of all the availability zones:
quarks.cloudfoundry.org/zones: '["us-central1-a", "us-central1-b"]'
As defined above, each pod is modified to contain affinity rules.
Each container and init container of each pod have the following env vars set:
1 2 3 4
KUBE_AZ="zone name" BOSH_AZ="zone name" CF_OPERATOR_AZ="zone name" AZ_INDEX="zone index"
Taints and tolerations is a concept defined in kubernetes to repel pods from nodes link. Defining tolerations is same as defined in the kubernetes docs. Keep in mind the affinity rules added by the controller when az’s are defined. An example is specified in the examples folder.
Restarting on Config Change
QuarksStatefulSets can be automatically updated when the environment/mounts have changed due to a referenced
ConfigMap or a
Secret being updated. This behavior is controlled by the
updateOnConfigChange flag which defaults to
Watches in cleanup controller
Reconciliation in cleanup controller
It will delete statefulsets with old versions, only after the new statefulset version instances are up and running.
QuarksStatefulSet Active-Passive Controller
Fig. 3: The QuarksStatefulset active/passive controller
Active/passive model is application model that have multiple running instances, but only one instance is active and all other instances are passive (standby). If the active instance is down, one of the passive instances will be promoted to active immediately.
activePassiveProbes key defines active probe to be performed on a container. The controller examines the active probe periodically to see if the active one is still active. If active pod is down or there isn’t an active pod, the first running pod will be promoted as active and label it as
The controller manages this active probing and provides pod designation label to the service’s selectors. Any requests sent to the service will then only be sent to the active pod.
Relationship with the BDPL component
Fig. 4: Relationship with the BPM controller
Figure 4 illustrates the interaction of the BPM Controller with the QuarksStatefulSet Controller. Once the BPM controller consumes the data persisted in secrets from the
QuarksJob Component, it will use that data to generate new
QuarksStatefulset instances. When these resources are generated, the QuarksStatefulSet controller will be watching and trigger its reconciliation loop.