--- title: "Automating deploys to Kubernetes using ArgoCD and jsonnet" url: "https://www.bekk.christmas/post/2020/16/automating-deploys-to-kubernetes-using-argocd-and-jsonnet" authors: ["Gustav Karlsson"] published: "2020-12-16" updated: "2024-11-07T13:05:28Z" language: "en-US" type: "article" tags: ["Sky"] --- # Automating deploys to Kubernetes using ArgoCD and jsonnet So you have a Kubernetes-cluster and a bunch of applications. You trigger deploys by manually pushing templated yaml to your cluster. The codebase is large and the manual processes time-consuming and error-prone. You need a better way. Well, let me help you with an opinionated suggestion. ## In short ArgoCD can use Git as the source of truth for Kubernetes deployments, pull changes into the cluster, and coordinate multi-step rollouts with hooks and sync waves. Combined with jsonnet, it becomes a code-driven way to generate reusable Kubernetes manifests and bootstrap an entire stack through an app-of-apps setup. - **For:** Kubernetes users managing deployments with ArgoCD and jsonnet - **Good to know beforehand:** Basic familiarity with Kubernetes and Git-based deployment workflows _AI-generated_ ## ArgoCD [ArgoCD](https://argoproj.github.io/argo-cd/) is a continuous delivery tool for Kubernetes that uses Git repos as the source-of-truth for the desired state of the cluster (following the [GitOps](https://www.weave.works/technologies/gitops/) philosophy). It runs inside the cluster and continuously monitor the Git repos for changes, rendering them into Kubernetes manifests and applying them to the cluster. So, contrary to the common pipeline approach where changes are pushed in, ArgoCD pulls them. As shown below, it comes with a UI for visualizing resources and their relations, which is particularly handy for newcomers to Kubernetes. (The UI _may_ also be used to administer applications, repositories etc, but then you are engaging in ClickOps rather than GitOps). ![The echoserver Application](https://cdn.sanity.io/images/ah2n1vfr/bekk-blogg-prod/887a2a0b8c24c9adfb0134c28c59e2ae3ba5e809-2754x864.png) The central component is the custom resource `Application` which basically is a pointer to a Git repo containing code that can be rendered into Kubernetes manifests. You may specify branch/tag, path, whether it should be automatically synced, inject parameters etc. See the doc (which is excellent btw) for [an example](https://argoproj.github.io/argo-cd/operator-manual/application.yaml) showing all the knobs. ## Deployment orchestration Applications often consist of multiple components that need to be updated during a deploy, and sometimes these components have an internal ordering that must be honored. I.e. "first do this and then do that". Tools that were not built specifically for deploys are often lacking here. Luckily, ArgoCD solves this elegantly using [resource hooks](https://argoproj.github.io/argo-cd/user-guide/resource_hooks/) and [sync waves](https://argoproj.github.io/argo-cd/user-guide/sync-waves/). ## App-of-apps Finally, one of the gems of ArgoCD is the pattern which is referred to as [App-of-apps](https://argoproj.github.io/argo-cd/operator-manual/cluster-bootstrapping/#app-of-apps-pattern): An `Application` may point to a Git repo that renders additional `Application` CRs (again pointing to other Git repo), continuing in as long a chain as necessary. So you can effectively bootstrap your entire stack by seeding ArgoCD with a single root `Application` that transitively renders all your `Applications` and underlying resources. No clicking required. You can of course also propagate relevant parameters such as `` to all applications as needed. ## Jsonnet When it comes to selecting a [tool/language](https://argoproj.github.io/argo-cd/user-guide/application_sources/) for describing the Kubernetes-manifests there will be lots of opinions. This is currently mine: I am a developer and I like to write code. I like to find the right abstractions and create generic, reusable pieces. I have always struggled with templating languages like Jinja2 or Go-templates because they are so limited and always "get in my way". After reading [Using Jsonnet does not have to be complex](https://medium.com/@prune998/using-jsonnet-does-not-have-to-be-complex-54b1ad9b21db) and [Why the f**k are we templating yaml?](https://leebriggs.co.uk/blog/2019/02/07/why-are-we-templating-yaml.html) I tried out jsonnet. I had heard it was a bit complex, but my initial skepticism was blown away after trying it. It felt much more familiar than awkward templating languages and factoring the code into reusable components was very easy (see for example [bitnami/kube-libsonnet](https://github.com/bitnami-labs/kube-libsonnet)). In jsonnet, it is quite simple to build a library allowing you to generate the full stack of Kubernetes objects (deployment, service, network policies, ingress, service monitor, etc) just by declaring the stack's config. Something similar to what you would do in a Helm values-file, but achieved with code rather than templating. Below is an example of what that _might_ look like. ```json local lib = import 'lib/v2/lib.libsonnet'; # top-level arguments (tlas) can be injected here: function(name='echo-server', namespace='default', env) local vars = lib.loadVars(env); (import 'lib/v2/app.libsonnet') { # here we are patching the default _config: _config+: { name: name, namespace: namespace, imageName: 'hashicorp/http-echo', imageTag: vars.tag, userPort: 8080, replicas: vars.replicas, ingress+: { path: '/echo/(.*)', rewrite: '/$1', tlsSecret: 'tls-secret-name', }, }, }.newAppAsList() ``` If done right, all resources can still be fully patchable. And since the jsonnet code is rendered into JSON which Kubernetes speaks natively, it will (like YAML) still allow you to do stuff like: - Validate the resources: - Dry-run against a cluster: - Diff against a cluster: This is of course typically handled by ArgoCD, but it might still be useful as development tools. ## Putting the pieces together Using the app-of-apps pattern referred to above, it is possible to seed the ArgoCD-installation with a single `Application` which will transitively render all `Applications` for the system. It might look something like: ``` argocd-root # git repo apps/ lib/ # libraries, managed by jsonnet-bundler or via git submodules vars/ vars/test.libsonnet # per-env vars for tracking-branch, autosync, tlas etc apps.jsonnet # renders all Applications according to vars/.libsonnet, injecting tla for env etc ``` The root-repo will render the echoserver `Application` and propagate relevant tlas such as ``. _That_ repo will in turn render the app-specific Kubernetes-manifests. It might look something like: ``` argocd-apps # git repo apps/ echoserver/ lib/ # libraries, managed by jsonnet-bundler or via git submodules vars/ vars/test.libsonnet # per-env vars for imageTag, replicas etc config/ # static configuration that goes into configmaps etc echoserver.jsonnet otherapp/... anotherapp/... ``` New `Applications` are then added by pushing changes to `argocd-root` and deployments are triggered by pushing changes to `argocd-apps`. Voilà! No more manual deploys and hopefully a tidier codebase 🎉. ## Relevant resources recommended by the author - [Why are we templating yaml?](https://leebriggs.co.uk/blog/2019/02/07/why-are-we-templating-yaml.html) - [Using Jsonnet does not have to be complex](https://medium.com/@prune998/using-jsonnet-does-not-have-to-be-complex-54b1ad9b21db) - [Jsonnet tutorial (official)](https://jsonnet.org/learning/tutorial.html) - [The State of Kubernetes Configuration Management: An Unsolved Problem](https://blog.argoproj.io/the-state-of-kubernetes-configuration-management-d8b06c1205)