Helm, a package manager for kubernetes, went through some evolution in the past several years. It evolved from Helm 2 to Helm 3, where helm 2 went through end-of-life nearly a year ago so I would be pretty late to the party. Without going to deep into helm internals I would mention just main feature. The removal of Tiller, a component that acted as a middle man and caused many troubles (requiring cluster around for many helm commands, security as tiller run as Kubernetes RBAC cluster-admin, etc.) is now gone! And many more, if interested official pages provide a good summary of helm 3). In this short blog post, I would like to give a quick overview of how Helm 3 works from a high-level perspective and what are the potential Helm operational modes and risks associated.
Helm 3 architecture is lightweight (compared to helm 2) schematically described in the following picture.

• Helm binary installed on the client machine interacting with kubernetes cluster api. 
• Helm metadata objects stored either as ConfigMap or Secret kubernetes object (depends on the configuration options)

Helm binary provides a cli to helm. The main function is to render manifests based on the Helm manifests templates and apply them to kubernetes cluster with preserving the revision history for possible rollbacks via helm cli. In addition, helm metadata objects whose payload is a serialized protocol buffer contains all data to render requested kubernetes manifests based on helm package specification and provided value file which acts as variables to helm package. You can list a history of helm release via (e.g. prometheus deployment in namespace prometheus):

$ helm history prometheus -n prometheus
REVISION	UPDATED                 	STATUS    	CHART            	APP VERSION	DESCRIPTION
113     	Fri Apr 23 12:55:11 2021	superseded	prometheus-11.0.0	2.16.0     	Upgrade complete
114     	Wed Apr 28 13:18:29 2021	superseded	prometheus-11.0.0	2.16.0     	Upgrade complete
115     	Wed Apr 28 13:49:13 2021	superseded	prometheus-11.0.0	2.16.0     	Upgrade complete
116     	Wed Apr 28 15:23:38 2021	superseded	prometheus-11.0.0	2.16.0     	Upgrade complete
117     	Wed Apr 28 17:03:15 2021	superseded	prometheus-11.0.0	2.16.0     	Upgrade complete
118     	Fri Apr 30 16:50:13 2021	superseded	prometheus-11.0.0	2.16.0     	Upgrade complete
119     	Mon May  3 16:10:01 2021	superseded	prometheus-11.0.0	2.16.0     	Upgrade complete
120     	Fri May  7 11:49:50 2021	superseded	prometheus-11.0.0	2.16.0     	Upgrade complete
121     	Fri May 14 15:06:13 2021	superseded	prometheus-11.0.0	2.16.0     	Upgrade complete
122     	Thu May 20 10:45:56 2021	deployed  	prometheus-11.0.0	2.16.0     	Upgrade complete

and find corresponding secrets in the namespace where the package is being applied

$ kubectl get secrets -n prometheus
NAME                                                  TYPE                                  DATA   AGE
sh.helm.release.v1.prometheus.v113                    helm.sh/release.v1                    1      52d
sh.helm.release.v1.prometheus.v114                    helm.sh/release.v1                    1      47d
sh.helm.release.v1.prometheus.v115                    helm.sh/release.v1                    1      47d
sh.helm.release.v1.prometheus.v116                    helm.sh/release.v1                    1      47d
sh.helm.release.v1.prometheus.v117                    helm.sh/release.v1                    1      47d
sh.helm.release.v1.prometheus.v118                    helm.sh/release.v1                    1      45d
sh.helm.release.v1.prometheus.v119                    helm.sh/release.v1                    1      42d
sh.helm.release.v1.prometheus.v120                    helm.sh/release.v1                    1      38d
sh.helm.release.v1.prometheus.v121                    helm.sh/release.v1                    1      31d
sh.helm.release.v1.prometheus.v122                    helm.sh/release.v1                    1      25d

When you upgrade or install package, Helm render the manifests (performs manifest api validation) and apply them to kubernetes cluster api. Kubernetes api performs upgrades, fill in missing default options, may silently ignore unrecognised or wrong settings (depends on configuration). Lower level kubernetes objects are derived from higher-level objects, e.g. replica sets from deployments. All this results in a manifest that is actually running in the kubernetes cluster as depicted in the picture. When you ask Helm to provide a manifest via

helm get manifest release-name -n namespace

Will provide deployed kubernetes manifest. Requested kubernetes manifest that reside in kubernetes Helm secret metadata. That is not exactly what is running in the cluster. To get the manifests that are actually running in the cluster.

kubectl get all -n namespace

This command will provide all kubernetes objects, including derived that are running in the cluster. By comparing those two, you can see differences. If you consider that kubernetes cluster is over time upgraded as well that you realize that what is actually running in the cluster can be actually surprising. That surprise usually manifests during disaster recovery. That is the reason why is it is highly preferred to eliminate Helm abstractions for the deployment chain. Kubernetes natively supports versioning via the revision history feature, which is specific to kubernetes object, e.g. deployment.

kubectl rollout history deployment deployment_name

This command captures only actual differences in the deployment manifest, so the revision count might not be the same as the number of revisions from Helm. Also, Helm revision history bundles all the kubernetes objects into a single revision.
To move to deployments of native kubernetes manifests, Helm offers a feature to render the manifests using a package and value file via helm template command. That completes the picture of how Helm operates. A simplified view could be summarised into:
1. render manifests
2. kubectl apply rendered manifests
3. store the helm revision metadata

Helm 3 provides good flexibility for deployments and leaves important decisions to SREs while keeping access to community-maintained packages.

TIP: If you develop some helm packages in-house, adding a helm lint command to your PR checks allows discovering issues during the PR review process. Don’t forget to check helm advanced features

If you like the content or have some experiences or questions don’t forget to leave a comment bellow and follow me on Twitter.