Testing

Tests description

Based on upstreams documentation https://github.com/thtanaka/kubernetes/blob/master/docs/devel/testing.md we use three levels of testing: unit, integration and e2e.

Before starting, run make tools to install the required dependencies.

Running make test executes all the test suites.

We use ginkgo for testing. Every package needs a suite_test.go for setup. It can be generated by running ginkgo bootstrap in the sub folder. Rename the generated file afterwards, to stay consistent. There is also ginkgo generate to create skeleton test files.

Unit

While unit testing we:

• test classes in isolation
• pass all dependencies to the constructor, so we can inject fakes for testing
• use counterfeiter and gomock/mockgen to generate and update fakes and mocks
• don’t test private methods, tests are in a separate _test package
• try not to nest ginkgo contexts too deep and keep tests DRY by extracting useful helpers
• assert incoming messages produce the expected state
• assert outgoing commands happened, like a file gets written
• assert all handled error cases are triggered
• can ignore outgoing queries, which only change internal state

Setup Ruby

Ruby gem for template rendering

1

gem install bosh-template

Integration

Integration tests formulate expectations on the interactions of several components. They require access to a Kubernetes, preferably minikube.

Integration tests start our operator directly, bypassing the command line. They do require the operator docker image and the bosh-template Ruby gem.

The environment package provides helpers to start the operator, get the kubeconfig and use the clients to create objects. In testing the catalog defines test objects.

Integration tests use a special logger, which does not log to stdout and whose messages can be accessed as a an array by calling env.AllLogMessages().

When using bin/test-integration the integration tests are run in parallel. Each Ginkgo test node has a separate namespace, log file and webhook server port and certificate.

The node index starts at 1 and is used as following to generate names:

1
2
3

namespace: $TEST_NAMESPACE + <node_index>
webhook port: $CF_OPERATOR_WEBHOOK_SERVICE_PORT + <node_index>
log file: $CF_OPERATOR_TESTING_TMP/quarks-operator-tests-<node_index>.log

Integration tests use the TEST_NAMESPACE environment variable as a base to calculate the namespace name. Test namespaces are deleted automatically once the tests are completed.

CF_OPERATOR_TESTING_TMP can be used to set a tmp directory for storing logs and other files generated during testing. If this variable is not set /tmp will be used instead.

The tests will create some NodePort services; normally the test can detect an IP address automatically. CF_OPERATOR_NODE_IP can set to the node IP of any arbitrary node to override this (e.g. for OpenStack Kubernetes clusters).

Generated files will be cleand up after the test run unless SKIP_CF_OPERATOR_TESTING_TMP_CLEANUP is set to true.

Webhook Configuration

Quarks StatefulSet requires a k8s webhook to mutate the volumes of a pod. Kubernetes will call back to the operator for certain requests and use the modified pod manifest, which is returned. Quarks Operator also uses a validating webhook to validate the BOSH deployment spec and the creation of reference resources specified in the spec. Secret validation admission webhook restricts the user from updating a versioned secret.

The quarks-operator integration tests use CF_OPERATOR_WEBHOOK_SERVICE_PORT as a base value to calculate the port number to listen to on CF_OPERATOR_WEBHOOK_SERVICE_HOST.

The tests use a mutatingwebhookconfiguration and a validatingwebhookconfiguration to configure Kubernetes to connect to this address. The address needs to be reachable from the cluster.

The configuration only applies to a single namespace, by using a selector. It contains the URL of the webhooks, build from CF_OPERATOR_WEBHOOK_SERVICE_HOST and the calculated port. It also contains SSL certificates and CA, which are necessary to connect to the webhook.

Note: If you have issues to start integration tests, and they fail by contacting the webhook server, for example if you see a error message like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

    Unexpected error:
          <*errors.StatusError | 0xc0002b4780>: {
              ErrStatus: {
                  TypeMeta: {Kind: "", APIVersion: ""},
                  ListMeta: {
                      SelfLink: "",
                      ResourceVersion: "",
                      Continue: "",
                      RemainingItemCount: nil,
                  },
                  Status: "Failure",
                  Message: "Internal error occurred: failed calling webhook \"mutate-statefulsets.quarks.cloudfoundry.org\": Post https://192.168.100.66:40603/mutate-statefulsets?timeout=30s: dial tcp 192.168.100.66:40603: connect: no route to host",
                  Reason: "InternalError",
                  Details: {
                      Name: "",
                      Group: "",
                      Kind: "",
                      UID: "",
                      Causes: [
                          {
                              Type: "",
                              Message: "failed calling webhook \"mutate-statefulsets.quarks.cloudfoundry.org\": Post https://192.168.100.66:40603/mutate-statefulsets?timeout=30s: dial tcp 192.168.100.66:40603: connect: no route to host",
                              Field: "",
                          },
                      ],
                      RetryAfterSeconds: 0,
                  },
                  Code: 500,
              },
          }

Check your firewall if it’s preventing the webhook server to be contacted from your target cluster or either if CF_OPERATOR_WEBHOOK_SERVICE_HOST is configured correctly

The certificates and keys are written to disk, so the webhook server can use them. They are also cached in a k8s secret for production, but that is not being used in integration tests, since they delete the test namespaces.

Tests suites should clean up their, namespace dependant, webhook configuration automatically.

End-to-End

The e2e tests are meant to test acceptance scenarios. They are written from an end user perspective. They are split into two types, ‘cli’ and ‘kube’.

The e2e CLI test exercise different command line options and commands which don’t need a running Kubernetes, like template rendering. The CLI tests build the operator binary themselves.

The second type of e2e tests use helm to install the CF operator into the k8s cluster and use the files from docs/examples for testing.

Running tests

In minikube

The following steps are necessary to have a proper environment setup, where all types of tests can be executed:

1. Start minikube

   1	minikube start --kubernetes-version v1.15.5

2. Switch to minikube docker daemon

   1	eval $(minikube docker-env)

   Note: Template rendering for BOSH jobs is done at deployment time by the operator binary. Therefore the operator docker image needs to be made available to Kubernetes cluster.

3. Export the CF_OPERATOR_WEBHOOK_SERVICE_HOST env variable

   1	export CF_OPERATOR_WEBHOOK_SERVICE_HOST=$(ip -4 a s dev $(ip r l 0/0 | cut -f5 -d' ') | grep -oP 'inet \K\S+(?=/)')

   Note: On Mac, use export CF_OPERATOR_WEBHOOK_SERVICE_HOST=$(ip a s $(ip r g 0/0 | cut -f5 -d' ') | grep -oE 'inet [^ /]+' | cut -f2 -d' '), because grep cannot handle perl regexs. Note: You can also find the correct IP, by running ip addr. The IP address under vboxnet1 is the IP that you need.

4. Export the OPERATOR_TEST_STORAGE_CLASS env variable

   1	export OPERATOR_TEST_STORAGE_CLASS=standard

   Note: Require for the PVC test creation, in minikube.

5. Ensure GO111MODULE is set

   1	export GO111MODULE=on

   Note: When you have a vendor folder (either from the submodule or manually created) settings this to off speeds up the build-image target.

6. Build the quarks-operator binary

   1

   bin/build

7. Build the quarks-operator docker image

   1	bin/build-image

Note: Consider setting DOCKER_IMAGE_TAG to a fixed variable. This will avoid rebuilding the docker image everytime, when doing changes in files not related to the quarks-operator binary.

Note: When not running in CI, nothing ensures a proper cleanup of resources after the deletion of the quarks-operator in the environment. You can make sure to manually verify that none old resources will interfere with a future installation, by:

1
2

# Deleting old mutating webhooks configurations
kubectl get mutatingwebhookconfiguration -oname | xargs -n 1 kubectl delete

In KinD

The following steps are necessary to have a proper environment setup, where all types of tests can be executed:

1. Install KinD

Follow the instructions from https://github.com/kubernetes-sigs/kind/

1. Start cluster

   1	kind create cluster --image kindest/node:v1.15.6

2. Export the CF_OPERATOR_WEBHOOK_SERVICE_HOST env variable. Use the IP of the docker bridge or your public IP. Firewall rules may interfere.

   1	export CF_OPERATOR_WEBHOOK_SERVICE_HOST=$(ip -4 a s dev $(ip r l 0/0 | cut -f5 -d' ') | grep -oP 'inet \K\S+(?=/)')

Note: On Mac, use export CF_OPERATOR_WEBHOOK_SERVICE_HOST=$(ip a s $(ip r g 0/0 | cut -f5 -d' ') | grep -oE 'inet [^ /]+' | cut -f2 -d' '), because grep cannot handle perl regexs.

1. Export the OPERATOR_TEST_STORAGE_CLASS env variable

   1	export OPERATOR_TEST_STORAGE_CLASS=standard

   Note: Required for the PVC tests.

2. Build the quarks-operator docker image

   First set the version to something static, not dependant on git:

   1	export DOCKER_IMAGE_TAG=${DOCKER_IMAGE_TAG:-dev}

   1	bin/build-image

   Or if you have local changes and use go mod edit --replace, follow the instructions from standalone components.

3. Load image into KinD

   kind load docker-image cfcontainerization/quarks-operator:$DOCKER_IMAGE_TAG
   

4. Set QuarksJob dependency. Choose a tag from docker.io.

   export QUARKS_JOB_IMAGE_TAG=${QUARKS_JOB_IMAGE_TAG:-dev}
   

   If using a locally built quarks-job image, load it via

   kind load docker-image cfcontainerization/quarks-job:$QUARKS_JOB_IMAGE_TAG

   (see standalone components).

Makefile

The following are the make targets available and their actions. When building and running the targets manually on the quarks-operator codebase, please set PROJECT=quarks-operator.

The common scripts shared between the quarks-operator components are in the quarks-utils project repository. To download them, make sure to run bin/tools, before running any other script. The Makefile should download them automatically.

The Makefile is intended for users, who don’t want to use the scripts in bin directly. It conveniently sets up some environment variables.

Note: CI and automation should not use the make targets to avoid indirection and declare variable explicitly.

General Targets

Build Targets

Test Targets

Generate Targets

CI

Our Concourse pipeline definitions are kept in the quarks-ci repo.