Kubernetes is a powerful cluster-wide management system for application containers that has become a standard for container orchestration. It allows developers to manage their deployments, including scaling, and monitoring, in a highly efficient and automated manner. But this flexibility has a price: complexity. There are many concepts to learn, different tools to work with it, and it may be difficult to feel that you really have control over it.
Many articles about Kubernetes present a hybrid mode of operation where resource creations and updates are mostly declarative, but deletions are imperative (or there are no deletions at all). The apply operation in Kubernetes only overwrites the specified resources, so if you rename something and apply the updated configuration, both the old and the new resources will be in the final state. To address this, the community developed a number of tools like Helm and Kapp to make it easier to manage and upgrade a set of resources.
This article explains how to deploy resources to a Kubernetes cluster in a way that creates, updates, and prunes resources so that the remote configuration matches exactly the applied configuration. It also discusses how to deal with Helm charts to include configuration from the community, and encrypt secrets to store them in a Git repository, since versioning the entire configuration is an important part of this workflow.
§Deploying with Kustomize
You can setup a Kubernetes test environment locally with tools like Minikube. Provided Docker is running, you should be able to start a test environment with this command:
Next, follow Kubectl's installation instructions. These commands should return some information about the server:
You should also create a Git repository to version your configuration files and allow rollbacks:
Do not hesitate to commit your changes as you follow along.
Kubectl can apply a set of configuration files that represent Kubernetes
resources from their path given as argument to the flag
A component called Kustomize can apply a group of resources listed in a
kustomization.yaml file. This is the same as bundling these resources into
one big YAML file, where each resource is separated by
---, and then applying
This feature can help organize the configuration files with a directory per
namespace, and inside them, a directory per "application", each having its own
Inside the top-level
kustomization.yaml, source the
directory as follows:
default directory, source the
file-server application directory:
In the directory
default/file-server, we will just add a basic
darkhttpd, also sourced from the associated
You can then create the
Kustomize is not limited to bundling configuration files. It allows "customization" by applying attributes like namespaces and labels to all the nested resources.
Because the resources in the
default directory should be deployed to the
default namespace, you can specify a namespace
default/kustomization.yaml that will apply to all nested resources:
You can inspect the configuration after it went through Kustomize, which corresponds to the previous Deployment with the namespace attribute added:
The next step is to apply this configuration on the server with
kubectl apply. The flag
-k will look for a
kustomization.yaml file in the
directory passed as argument to build the configuration, just like you did
kubectl kustomize, and apply it on the server:
You should have a
test-file-server pod running:
You can refer to Kubectl Cheat Sheet § Viewing, finding resources for additional commands that can help you inspect this Deployment. Try to print the server-side Deployment configuration in YAML format and compare it with the local version.
To connect to the file server, create a port forward from the host to the pod:
Then you can open the index page at
http://127.0.0.1:8080/ in a web browser.
Since the file server appears to be working, let's end our experimentation and
rename the Deployment from
Re-apply the configuration:
Although it isn't mentioned in the previous command output, the old test Deployment isn't cleaned up, so you end up with two Deployments of the file server:
kubectl apply only creates or updates resources. You could
cleanup this Deployment using an imperative
kubectl delete command:
But this is exactly what we want to avoid. You will see in the next section how you can prune resources declaratively.
Removing resources from your local configuration is not enough to remove them from the cluster configuration, and it wouldn't be appropriate for Kubectl to remove everything that is on the server but isn't in the local configuration.
Indeed, some resources are created dynamically by services running inside your Kubernetes cluster, so you will have to restrict pruning to the resources you actually created. This is done using labels and selectors.
Also, stateful resources like volumes, or the namespaces that contain different kind of resources, shouldn't be pruned without careful consideration. For that reason, Kubectl also requires us to be explicit about the type of resources we want to prune.
The allowlist prune feature built into Kubectl since Kubernetes v1.5 (2016-12-13) doesn't work with server-side applies and has some correctness issues, which explains why it never got out of the experimental stage.
Recently, it underwent a complete redesign tracked in KEP 3659, which introduces a new standard called ApplySet that records in a Secret the set of applied Kubernetes objects to provide fast and correct pruning.
The alpha implementation landed in Kubernetes v1.27 (2023-04-11). Once stabilized, I will update this section accordingly.
§Labels and selectors
Kubernetes supports labels, which are key/value pairs attached to resources.
You can use selectors to find a subset of resources with a given label. Do not
confuse them with annotations which aren't indexed (they cannot be selected
efficiently). Your previous Deployment of the file server uses labels
internally, since it only targets pods that have the label
To identify all the resources you deployed through Kustomize, and to scope all Kubectl's operations on them, you can add a dedicated label:
Using a fully-qualified domain name that you own is a simple way to avoid
or third-party labels. Try to print the final configuration with
kubectl kustomize . to check that the label is applied properly.
The prune command in Kubectl is similar to the apply command, but it has an
--prune option to remove Kubernetes resources that are no longer
defined in the local configuration. To use this option, you have to provide a
label selector to the option
-l, which specify the resources you want to
For example, if you pass
-l dzx.fr/managed-by=kustomize, the command will
apply and prune the resources that have the label
dzx.fr/managed-by=kustomize, leaving anything else untouched.
At the end of a CI pipeline, it is useful to select only the resources that are part of this application to ensure the deployment doesn't affect other resources. To that end, you can also add a label for the application name:
A selector can contain comma-separated values to select multiple labels for the
apply command. For instance,
-l dzx.fr/managed-by=kustomize,dzx.fr/app=echo-server will only apply to
resources that are both labelled with
dzx.fr/app=echo-server. Kubernetes provides multiple ways to select
If you try to apply this configuration, you will likely encounter the following error:
A Deployment manages a set of pods according to the
attribute. Being able to edit these labels while the Deployment is active
could result in orphaned pods that do not match the selector anymore (the
only safe action is to remove a label).
The only way update these immutable fields is to recreate the Deployment, which leaves two options:
- Delete the Deployment with
kubectl deleteand apply the configuration to recreate it.
kubectl applyto let Kubernetes recreate it (you cannot use
Recreating the Deployment ensures the associated pods get cleaned up and recreated with the appropriate labels.
At the time of writing, it is still possible to prune some resources without
specifying their type, but this behavior is deprecated. The apply command wants
you to be explicit about the resource types you want to prune with the option
--prune-allowlist, for example,
You can try to rename the file server Deployment as we did previously, and run the following command to see that the previous version gets pruned:
Supplying the allowed resource types by hand is enough for simple configurations, but it would be better if we could generate this list procedurally, especially since in addition to the builtin resources, Kubernetes supports the definition of custom resources. For example, this extension mechanism is used by Traefik and Cert Manager.
Collecting them directly from the configuration files wouldn't always work,
because if you remove all the resources of some type, they wouldn't get pruned
the next time you apply this configuration. Kubectl provides a command to query
the server for the list of all the supported resource types, from which you can
filter the resources that can be pruned (they must support the verb
The main issue with this command is that it doesn't list the resource types
according to the strict format required by the option
--prune-allowlist=<group/version/kind>: you have to join the
column with the
A further tweak is required for the core resources that have
v1. If you pass
--prune-allowlist=v1/ConfigMap, you would get an error
core group must be specified for these resources, as in
core/v1/ConfigMap. If you need a quick and dirty solution, the following
shell command generates the list of
--prune-allowlist options for all the
resource types supported by the server:
For a more robust solution, you could rely on Kubernetes' Go client library to generate the list of prunable resource names:
To keep the commands short in the remaining code snippets, I will only specify the resource types that we use.
Even with a selector and an explicit allowlist, pruning server resources shouldn't be taken lightly. Versioning your configuration files provides a way to review changes locally, but a modification that looks benign may have unintended effects on the generated configuration.
You could automate the pruning step on a subset of non-stateful resources, and
carefully prune the remaining resources. You can use the following command to
generate a diff between the server and the client configurations that shows the
updates and deletions that would be performed by
If you add an annotation on the file server Deployment:
Kubectl should generate the following diff:
The drawback of this command is that the final configuration is generated
implicitly for both the diff and the apply commands. Although unlikely, the
source files may change in-between. As a solution, you could generate the
configuration first with
kubectl kustomize, review it with
and then apply it:
That ensures the configuration that gets applied is exactly what you reviewed.
§Handling Helm charts
The leading tool to deploy complex applications on Kubernetes is Helm. A Secret records the state of each release. On upgrade, the new configuration is compared to the previous release, and only the components that changed are updated.
If you modify a resource controlled by Helm, the changes won't be reverted by subsequent upgrades unless you also modify the source configuration for that specific resource. This is because Helm upgrades depend on the recorded state instead of the actual state, which prevents it from working in a truly declarative way.
Fortunately, we can turn Helm charts into plain Kubernetes configuration files, that we can version in our repository and deploy with Kubectl. In this section, you will see how to configure and deploy the Helm chart for Traefik, a cloud native reverse proxy.
Follow Helm's installation instructions. On most Linux distributions, it should be available from your package manager:
Helm charts are commonly distributed as part of a repository. You will have to add the Traefik Labs repository:
Then, you can list the charts available for install:
In the next section, you see how to add
traefik/traefik to your
Let's create a directory for the deployment of Traefik:
Source this directly from the default namespace
Then, create the associated
Helm charts rely on the concept of value files. They contain the context that is injected into the resource template files. You can view the default values with the following command:
You can override any of these properties by creating your own
file, that you can source when generating the chart. For demonstration
purposes, configure new command line arguments to enable Traefik's access log
in JSON format, keeping the
kubectl kustomize, you can ask Helm to output a plain configuration
file corresponding to the entire Helm chart (we refer to this process as
"inflating" a Helm chart):
For our Traefik deployment, you can use this command to generate
default/traefik/traefik.yaml configured with the updated values:
You can grep the output to find them in the main Deployment:
There are a few extra arguments that need an explanation:
--namespace: defaultensures the proper namespace is passed to the template, as it can be used in an application-specific beyond the control of Kustomize.
--include-crdsincludes the Custom Resources Definitions, which are Kubernetes resource extension mechanism.
--skip-testsprevents the creation of test containers that are used by Helm to validate the deployment of a release.
-f default/traefik/values.yamladds the values to the rendering context.
Helm provide additional features such as hooks that can run when deploying a release. Obviously, this isn't supported outside of Helm. So if you have to use a chart that relies on these features, you may prefer to deploy it through Helm.
Otherwise, upgrading Helm charts in a Kustomize configuration is as simple as
helm template with the same options described
previously, inspecting the diff, and applying the changes.
If you skim through Kustomize's documentation, you will find that it
supports inflating Helm
behind the flag
The documentation states that these options were designed for experimentation, not for production. Here are a few reasons:
Such configuration doesn't capture the state in a fully declarative way because it depends on remote resources at generation time. The main risk is that you may not be able deploy or rollback if this URL becomes inaccessible.
- When you review changes with Git after changing this configuration, you
cannot inspect what changed in the Helm chart in terms of actual
Kubernetes resources without comparing the output of
kubectl kustomizebefore and after.
- The chart is pulled every time you generate the config. It is possible to commit it to the repository to avoid repulling every time, but these rules are implicit, and it becomes slightly more complicated to keep the local chart in sync with the specified version.
For all these reasons, whenever I want to upgrade these Helm charts, I prefer to inflate them explicitly and commit the result as a plain YAML configuration file, sourced like a regular Kustomize resource.
- When you review changes with Git after changing this configuration, you cannot inspect what changed in the Helm chart in terms of actual Kubernetes resources without comparing the output of
§Keeping secrets with KSOPS
Secrets are designed to hold sensitive data that you shouldn't embed in your Deployments. This section explains how to make them work with a fully declarative configuration while maintaining a reasonable level of security. We will have to ensure that updates to a Secret are propagated to all the Deployments (which isn't the default behavior), then we will see how to store them encrypted in the repository, and finally, we will rely on a plugin to decrypt the Secrets during the apply operation, which isn't natively supported by Kubectl.
Create a Secret named
password that contains the typical piece of information
you don't want to serve publicly, like a password:
Secret data is Base64-encoded. You can use the following command to encode to Base64:
And decode from Base64 as follows:
Source the Secret from the file server's
Then, edit the Deployment to mount this Secret as a volume in the webserver's root directory (of course, this isn't something you would typically do, this is only for demonstration purposes):
Apply these changes:
You can now create a port-forward to the file server:
And cURL the content of the secret:
Let's try to update the password:
Apply this change:
Check that the Secret was updated:
And... It didn't work. So much for a declarative deployment. You can try to restart the Deployment:
And now the Secret is up-to-date:
Kubernetes only restarts Deployments whose configuration changed. Because the Deployment itself didn't change, but only the content of the Secret it references, Kubernetes didn't restart this Deployment.
To force a restart after changing a Secret, you would have to generate them in a way that also requires a change to the Deployment. For example, you could add the hash of the Secret to its name, so the Deployment must be updated when you reference the new Secret.
There are two ways to achieve that with Kustomize:
Annotate the Secret with
Use a Secret generator from a
In the output of
kubectl kustomize, you can see that a new Secret is created
based on the hash of its content:
And the Secret name in the Deployment is automatically updated as well:
Whenever the content of the Secret changes, the hash will change, and Kubernetes will have to restart the Deployment.
Any resource that references another by name, like using ConfigMaps from a Deployment, is subject to the same lazy update behavior.
ConfigMaps are used to pass non-sensitive bits of configuration to a
Deployment, so Kustomize provides a
to make them declarative.
Unfortunately, nothing prevents a custom resource from referencing another one by name, and Kustomize won't be able to generate unique identifiers for them. That highlights one limitation of declarative deployments with Kubernetes.
Whether you use plain Secrets or generate them through Kustomize's Secret generator, the content is left unencrypted, which makes them unsuitable for pushing to a shared Git repository. To encrypt these files, we will use SOPS, a CLI tool to manage encrypted files, and Age, a simple alternative to GnuPG.
You can install these programs from your package manager:
Creating a key pair with Age is easy:
SOPS looks for Age keys in
~/.config/sops/age/keys.txt, so move the previously
generate key to this location:
Next, you have to indicate which public keys should be able to decrypt the
Secrets you will create in this repository by listing them in a
configuration file at the root:
unencrypted_regex attribute will make sure to keep the specified fields
unencrypted, so only the sensitive data is encrypted.
To create a new encrypted Secret inside the repository, ensure the
environment variable is set and run:
It opens a temporary file that contains the cleartext. Replace the content by the password Secret:
Then, save and close this file. You should try to reopen it with SOPS to ensure
you can decrypt it properly. If you inspect its encrypted content, you will see
that it includes a ciphertext for each public key specified in
The next step is to allow Kubectl to decrypt this file during the apply operation. Kubectl doesn't provide a way to manage encrypted Secrets, but it supports plugins such as KSOPS, which builds on SOPS to decrypt Secrets.
The plugin ecosystem is currently experimental, so the following instructions depend on which version of Kustomize you use, of which there are two:
The integration into Kubectl, which limits access to some features and requires the use of legacy plugins.
The standalone version, that provides a newer "plugin" mechanism called KRM functions.
At some point, the integration should catch up with the standalone version, but the following two sections contain the instructions for each version, choose whichever applies to you.
§Legacy exec plugin (
Install KSOPS using the legacy installation script that puts the binary to
To hook KSOPS into Kustomize, you have to create a generator resource that references the encrypted Secret file:
Add this generator to the file server Kustomization:
The use of Kubectl plugins is marked as experimental, so it is behind the flag
--enable-alpha-plugins. This flag cannot be used with the apply operation, so
you will have to apply in two steps, first by generating the configuration with
kubectl kustomize, and then piping the output into
kubectl apply -f -
-f - tells Kubectl to read the configuration from STDIN):
§Exec KRM function (standalone
You can install KSOPS by downloading the binary from the
releases and putting
it in your
$PATH (for example, in
To hook KSOPS into Kustomize, you have to create a generator resource:
Add this generator to the file server
The use of Kustomize plugins is marked as experimental, so it is behind the
--enable-alpha-plugins. You also have to allow exec plugins with
These flags cannot be used with the apply operation, so you will have to apply
in two steps, first by generating the configuration with
kustomize build, and
then piping the output into
kubectl apply -f - (where
-f - tells Kubectl to
read the configuration from STDIN):
There are a few challenges to overcome to manage Kubernetes resource in a fully declarative way:
- The pruning mechanism built into Kubectl requires explicit configuration involving labels, selectors, resource types, that isn't straightforward and is still experimental.
- Popular packages rely on third-party tools like Helm that provide other mechanisms to manage Kubernetes resources.
- Changes to resources like Secrets or ConfigMaps do not cause restarts of the Deployments that reference them.
- There are security considerations to versioning sensitive resources in a Git repository.
Kustomize and KSOP provide a way to overcome these issues to provide a first step on the way to GitOps, by allowing the management of Kubernetes resources in declarative way from a Git repository. The next step is to integrate these configuration into a continuous deployment pipeline: you could go a long way with a few Drone CI pipelines, or you could use tools like ArgoCD or Flux that rely on a Kubernetes controller.
This article demonstrated how Kustomize helps bundling multiple resources, how to apply cross-cutting fields to nested resources, and how to use generators. But this is only the tip of iceberg, as its true purpose is to patch resources to provide a template-free last-mile customization layer. A new standard based on KRM functions aims to create an ecosystem of transformations that generalizes the concepts of plugins and generators to make Kustomize even more versatile.