Skip to main content
Version: v1alpha4

Helm

Overview​

This guide demonstrates how Holos makes it easier to integrate multiple Helm charts together with strong type checking and validation. Holos adds valuable capabilities to Helm and Kustomize:

  1. Inject the same value into two or more charts to integrate them safer than Helm alone.
  2. Add strong type checking and validation of constraints for Helm input values.
  3. Easily implement the rendered manifests pattern.

This guide works through managing the prometheus and blackbox Helm Charts along side the httpbin Kustomize base, integrating all three together in a unified way with CUE.

Requirements​

We want to probe the httpbin service to make sure it's up and running. Our organization uses prometheus in the observability system. The httpbin service doesn't expose a metrics endpoint, so we'll use the blackbox exporter to probe httpbin and publish a scrape target for prometheus.

Third party software should be managed using the upstream distribution method. The prometheus community publishes Helm Charts. httpbin publishes a Kustomize base.

Try Locally​

This guide is written for use with a local Kubernetes cluster which can be built quickly with our Local Cluster guide.

Without Holos​

Install prometheus and blackbox.

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

helm install \
  prometheus \
  prometheus-community/prometheus
helm install \
  --set service.port=80 \
  --set fullnameOverride=blackbox \
  prometheus-blackbox-exporter \
  prometheus-community/prometheus-blackbox-exporter

Install httpbin

kubectl apply -k github.com/mccutchen/go-httpbin/kustomize

Problems​

Helm is simple and straightforward to get started, but there are a number of problems we'll quickly run into if we go down this path.

  1. The prometheus chart tries to connect to blackbox at http://blackbox:80 here but the blackbox chart is listening at http://prometheus-blackbox-exporter:9115 here.
  2. The two charts are not well integrated, they don't work together by default.
  3. The prometheus chart does not expose an ergonomic way to reconfigure the blackbox endpoint. Therefore we can only configure the blackbox chart.
  4. Configuring the endpoint requires indexing into two deeply nested lists. Indexing into lists is unreliable, the target position may change.
  5. The blackbox chart authors forgot to include fullnameOverride in the values.yaml file.
  6. httpbin is managed with Kustomize, not Helm, requiring a different process and toolchain.
  7. The above commands don't achieve the goal, we still need to manually edit the httpbin Service to add the prometheus.io/probe: "true" annotation. Automation requires crafting another Kustomization layer to patch the base Service.

These problems complicate the task of integrating blackbox, prometheus, and httpbin for this simple use case. We can install the blackbox chart with --set fullnameOverride=blackbox to get it working with the prometheus chart, but doing so is papering over a pitfall for the next teammate who travels this path. When the prometheus chart changes the blackbox endpoint, it won't be clear why or where the integration breaks.

The crux of the issue is there is no good way to pass the same hostname and port to both charts. It would be easier, safer, and more reliable if we could ensure both charts are configured in lock step with one another.

Solution​

Holos leverages CUE making it easy to configure both of these charts in lock step with each other. In CUE, this is called configuration unification, the C and U in CUE. Holos also provides a generalized rendering pipeline that makes it easier to manage Kustomize bases and Helm Charts using the same tool and process.

Installation​

Install holos with the following command or one of the methods described in the Installation guide.

go install github.com/holos-run/holos/cmd/holos@latest

Initialization​

First, generate the directory structure we're going to work in. Start in an empty directory then run holos generate platform v1alpha4 to initialize the directory structure.

holos generate platform v1alpha4
git init . && git add . && git commit -m initial

A platform is a collection of components. A component is a helm chart, a kustomize base, resources defined from CUE, etc...

Platforms are empty by default.

holos render platform ./platform

Holos uses CUE to build a platform specification which is really just a fancy way of saying a list of components to manage.

cue export --out=yaml ./platform

This yaml looks like a Kubernetes resource, but is not. The holos executable processes this Platform resource when you run holos render platform.

Let's manage the same helm chart we installed for Prometheus. Make the component directory.

Config Schema​

The prometheus and blackbox charts don't provide a good way to inject the blackbox host and port to both charts to integrate them together. Holos and CUE fill this gap. We'll define the schema and data in one place, then inject the validated values into both charts, integrating them together holistically.

Define the host and port in projects/blackbox.schema.cue. We'll inject these values into both charts to configure them in lock step.

mkdir -p projects
touch projects/blackbox.schema.cue
package holos

// Define the schema
#blackbox: {
	// host constrained to a lower case dns label
	host: string & =~"^[a-z0-9]([a-z0-9-]{0,61}[a-z0-9])?$"
	// port constrained to a valid range
	port: int & >0 & <=65535
}

// Concrete values that must validate against the schema.
_blackbox: #blackbox & {
	host: "blackbox"
	port: 9115
}
important

CUE allows us to define types and constraints. Validation in CUE is better than general purpose languages limited to type checking only.

Prometheus Chart​

Add the CUE configuration to manage the prometheus Helm Chart component.

mkdir -p projects/platform/components/prometheus
touch projects/platform/components/prometheus/prometheus.cue
package holos

// Produce a helm chart build plan.
_Helm.BuildPlan

_Helm: #Helm & {
	Chart: {
		name:    "prometheus"
		version: "25.27.0"
		repository: {
			name: "prometheus-community"
			url:  "https://prometheus-community.github.io/helm-charts"
		}
	}
}

Register the prometheus chart with the platform by adding the following file to the platform directory.

mkdir -p platform
touch platform/prometheus.cue
package holos

_Platform: Components: prometheus: {
	name:      "prometheus"
	component: "projects/platform/components/prometheus"
	cluster:   "local"
}

Render the platform to render the prometheus chart.

holos render platform ./platform

Blackbox Chart​

Add the CUE configuration to manage the blackbox Helm Chart component.

mkdir -p projects/platform/components/blackbox
touch projects/platform/components/blackbox/blackbox.cue
package holos

// Produce a helm chart build plan.
_Helm.BuildPlan

_Helm: #Helm & {
	Chart: {
		name:    "prometheus-blackbox-exporter"
		version: "9.0.1"
		repository: {
			name: "prometheus-community"
			url:  "https://prometheus-community.github.io/helm-charts"
		}
	}
}

Register the blackbox chart with the platform by adding the following file to the platform directory.

mkdir -p platform
touch platform/blackbox.cue
package holos

_Platform: Components: blackbox: {
	name:      "blackbox"
	component: "projects/platform/components/blackbox"
	cluster:   "local"
}

Render the platform to render both the prometheus and blackbox charts.

holos render platform ./platform

Now is a good time to commit so we can see the next changes clearly.

git add .
git commit -m 'prometheus and blackbox not integrated'

Unify Helm Values​

Inject the blackbox host and port fields into both charts to manage them in lock step. Holos and CUE offer a holistic integration layer unified across the whole platform.

We'll import the default chart values directly into CUE so we can work with them easily as data instead of plain text.

First for prometheus.

cue import --package holos \
  --path '_Helm: Values:' \
  --outfile projects/platform/components/prometheus/values.cue \
  projects/platform/components/prometheus/vendor/25.27.0/prometheus/values.yaml

Then for blackbox.

cue import --package holos \
  --path '_Helm: Values:' \
  --outfile projects/platform/components/blackbox/values.cue
  projects/platform/components/blackbox/vendor/9.0.1/prometheus-blackbox-exporter/values.yaml
important

Values in CUE are straight forward to integrate with the blackbox data structure we defined in the Config Schema section.

git add .
git commit -m 'import values'

Then make the following changes to both values files to manage the charts in lock step.

patching file 'projects/platform/components/blackbox/values.cue'
patching file 'projects/platform/components/prometheus/values.cue'

Now both charts use the same values in lock step.

git add .
git commit -m 'unify blackbox host configuration'

Render the platform to see the fully rendered manifests.

holos render platform ./platform
git diff
git add . && git commit -m 'render linked prometheus and blackbox'

Better Validation​

The following Helm command produces no error and renders invalid output, a common source of frustration and friction.

helm template \
  --set service.port=80 \
  --set fullnameOverride="this is not valid" \
  prometheus-blackbox-exporter \
  prometheus-community/prometheus-blackbox-exporter

Holos and CUE provide much better validation and early indication of the problem. Editors with LSP support benefit from immediate feedback. The holos render platform command fails immediately with a clear validation error.

patch -p1 <<EOF
--- a/projects/blackbox.schema.cue
+++ b/projects/blackbox.schema.cue
@@ -10,6 +10,6 @@ package holos
 
 // Concrete values that must validate against the schema.
 _blackbox: #blackbox & {
-	host: "blackbox"
+	host: "this is not valid"
 	port: 6115
 }
EOF
holos render platform ./platform

VS Code out of bound error

Undo the invalid change.

git restore projects/blackbox.schema.cue
rm -f projects/blackbox.schema.cue.orig

Httpbin Kustomization​

We need to manage httpbin so we can achieve the goal of probing a simple service with prometheus. The maintainers provide a Kustomize base in lieu of a Helm chart. Holos makes it easy to integrate software regardless of the distribution method.

We'll use Holos and CUE to add the prometheus.io/probe: "true" annotation using Kustomize without having to write any YAML, only well structured CUE.

Create the httpbin component similar to how we did the prometheus and blackbox components.

mkdir -p projects/platform/components/httpbin
touch projects/platform/components/httpbin/httpbin.cue
package holos

import "encoding/yaml"

// Produce a Kustomize BuildPlan for Holos
_Kustomize.BuildPlan

// https://github.com/mccutchen/go-httpbin/blob/v2.15.0/kustomize/README.md
_Kustomize: #Kustomize & {
	KustomizeConfig: Files: "resources.yaml": _
	KustomizeConfig: Kustomization: {
		commonLabels: "app.kubernetes.io/name": "httpbin"
		images: [{name: "mccutchen/go-httpbin"}]
		_patches: probe: {
			target: kind: "Service"
			target: name: "httpbin"
			patch: yaml.Marshal([{
				op:    "add"
				path:  "/metadata/annotations/prometheus.io~1probe"
				value: "true"
			}])
		}
		patches: [for x in _patches {x}]
	}
}

Register the component with the platform.

mkdir -p platform
touch platform/httpbin.cue
package holos

_Platform: Components: httpbin: {
	name:      "httpbin"
	component: "projects/platform/components/httpbin"
	cluster:   "local"
}

Render the platform to render the prometheus chart.

holos render platform ./platform
git add .
git commit -m 'add httpbin'

Local Cluster​

Let's apply the fully rendered manifests holos produces for prometheus, blackbox, and httpbin. Refer to the Local Cluster guide to follow along.

kubectl apply \
  -f deploy/clusters/local/components/prometheus \
  -f deploy/clusters/local/components/blackbox \
  -f deploy/clusters/local/components/httpbin

Port forward to the prometheus web interface.

kubectl wait --for=condition=Available deployment/prometheus-server --timeout=300s
kubectl -n default port-forward svc/prometheus-server 8081:80

Then browse to http://localhost:8081/targets?search=httpbin

Prometheus should report httpbin.default.svc:80 is UP:

Prometheus httpbin probe is UP

Conclusion​

In this guide we saw how Holos makes it easier to holistically integrate the prometheus and blackbox charts so they're configured in lock step with each other. If we relied on Helm alone, there is no good way to configure both charts to use the same service endpoint.

We also saw how Holos makes it easier to manage httpbin which is distributed as a Kustomize base. Holos offers a clear way to post process both Kustomize and Helm output, for example patching an annotation onto the httpbin Service.

Dive deeper with our guides, or learn more about Holos in our Technical Overview.