<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Brian Grant on Medium]]></title>
        <description><![CDATA[Stories by Brian Grant on Medium]]></description>
        <link>https://medium.com/@bgrant0607?source=rss-ccdbe9a99319------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*IswKlFXcN7a_0bsXVi8l9A.jpeg</url>
            <title>Stories by Brian Grant on Medium</title>
            <link>https://medium.com/@bgrant0607?source=rss-ccdbe9a99319------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 02:08:21 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@bgrant0607/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com" rel="hub"/>
        <item>
            <title><![CDATA[Managing variant drift with generic Helm templates]]></title>
            <link>https://itnext.io/managing-variant-drift-with-generic-helm-templates-f7b8433dd0b8?source=rss-ccdbe9a99319------2</link>
            <guid isPermaLink="false">https://medium.com/p/f7b8433dd0b8</guid>
            <category><![CDATA[gitops]]></category>
            <category><![CDATA[config-as-code]]></category>
            <category><![CDATA[helm]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Brian Grant]]></dc:creator>
            <pubDate>Mon, 17 Aug 2026 16:58:12 GMT</pubDate>
            <atom:updated>2026-08-17T19:28:16.587Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TWCxnULfUhslXmt5fe_73w.png" /><figcaption>Image generated by Gemini</figcaption></figure><p>“Configuration drift” <a href="https://itnext.io/kubernetes-configuration-and-infrastructure-as-code-taxonomy-62ab4dd7e82f">usually refers</a> to the <a href="https://itnext.io/why-configuration-drift-is-so-hard-to-avoid-in-practice-443248cafc9c">drift between the source of truth for a configuration and the live state</a>. But there can also be unintended or undesired drift between different <a href="https://itnext.io/variant-generation-is-the-primary-capability-of-infrastructure-as-code-773e6b404b49">variants of a configuration</a>, such as deployments of an application in different clusters or environments. This is another flavor of the “works on my laptop” problem — works in dev, staging, some regions, etc.</p><p>How can this kind of drift be prevented? There’s an even more basic issue: how do people determine what the configuration should be for each deployment target? The answer to that is often <strong>parameter values</strong>.</p><p>I’ve written about over-parameterization of Helm templates many times, such as in the <a href="https://github.com/kubernetes/design-proposals-archive/blob/main/architecture/declarative-application-management.md#parameterization-pitfalls">original document explaining the thinking behind kustomize</a>. It happens both with <a href="https://itnext.io/i-shouldnt-have-to-read-installer-code-every-day-4dc1e5f9ee1a">off-the-shelf charts</a> and with <a href="https://itnext.io/infrastructure-as-code-cargo-culting-patterns-b714f680db4b">charts for custom applications</a>.</p><p>Some people have fully embraced the “parameterize everything” approach. I call this the “One Chart” approach, after <a href="https://www.youtube.com/watch?v=HzJ9ycX1h0c&amp;t=563s">this Michael Goodness talk</a> and <a href="https://github.com/gimlet-io/onechart/tree/master/charts/common/templates">Gimlet’s OneChart</a>. The pattern is also used by the <a href="https://github.com/companyinfo/helm-charts/tree/main/charts/helmet">Helmet library chart</a>.</p><p>Such generic charts often have template code that looks like this:</p><pre>  {{- if .Values.foo.enabled }}<br>  foo: {{- omit .Values.foo &quot;enabled&quot; | toYaml | nindent 4}}<br>  {{- end }}</pre><p>Or it might just check whether .Values.foo exists, just reference a single value, explicitly reference several keys and values, iterate over a dict, etc. But the pattern is essentially: if this key is in the values’ YAML, and, optionally, is enabled, then insert the corresponding value(s) into the template’s YAML here.</p><p>This approach doesn’t provide much or any <a href="https://itnext.io/abstraction-is-the-wrong-way-to-simplify-configuration-81ac4bad02ab">abstraction over the raw Kubernetes resources</a>, so what’s the point of doing this rather than just writing Kubernetes manifests directly?</p><ul><li>Default values can be provided.</li><li>It formats the YAML for Kubernetes resources. The values.yaml structure is simpler — just nested maps.</li><li>Because values files are just maps of maps, they are easily mergeable, so values can be factored into multiple files and “stacked”.</li><li>In a few cases, the same value may be reused in multiple places, such as in both resource names and labels.</li></ul><p>Compared to a simpler chart that exposes fewer parameters, it’s very <a href="https://itnext.io/the-tension-between-flexibility-and-simplicity-in-infrastructure-as-code-6cec841e3d16">flexible</a>. It doesn’t hardcode decisions into the template and mitigates the need for patching via kustomize or other mechanism. Simpler charts can be created by wrapping a generic chart. Helmet, for example, was written with that intent. But even with umbrella charts generic enable values can be exposed.</p><p>Obviously, with the generic template approach, just by looking at the template one can’t infer what value(s) is likely to be inserted or, importantly, where (i.e., which clusters/environments) and why. Instead, the intent for what values should be set where is encoded in directories and files.</p><p>The values might be organized something like:</p><pre>globals/<br>  global.yaml<br>  environments/<br>    dev.yaml<br>    staging.yaml<br>    prod.yaml<br>  regions/<br>    na.yaml<br>    eu.yaml<br>    ...<br>  clusters/<br>    staging1.yaml<br>    ...<br>    prod-naeast1.yaml<br>    ...<br>charts/<br>  cert-manager/<br>    base.yaml<br>    environments/<br>      prod.yaml<br>    clusters/<br>      prod-naeast1.yaml<br>...    <br></pre><p>There could also be values files used as more specific mixins, sometimes called traits (<a href="https://github.com/oam-dev/spec/blob/master/6.traits.md">OAM</a>, <a href="https://openchoreo.dev/docs/concepts/developer-abstractions/">OpenChoreo</a>) or aspects, such as:</p><pre>traits/<br>  foo/<br>    no-foo.yaml<br>    default-foo.yaml<br>    special-foo.yaml</pre><p>Each file represents a different configuration of that trait. These types of mixins can be used across multiple applications in the case of a unified generic chart like OneChart. Off-the-shelf charts, on the other hand, don’t generally share template libraries.</p><p>With multiple layers of values file overlays, one would likely build a tool to select and enumerate the correct values files in the correct order for each cluster, to implement the desired inheritance and override model. Like a poor man’s <a href="https://github.com/puppetlabs/hiera">Hiera</a>. Over time it requires a fair amount of discipline and oversight to eliminate special cases or move them up the hierarchy or factor them out into mixins. When that isn’t or can’t be done, then the same values may be set in multiple files, and may be different in others. That’s when variant drift becomes challenging to manage.</p><p>This kind of structure can address the <em>what</em> and <em>where</em>, but what about the <em>why</em>? If you’re lucky, there are comments in the values files. Failing that, you can do a git blame and track back to a commit message and/or PR discussion. If the value is set in multiple files, you may need to track down multiple commits, potentially across multiple git repositories.</p><p>After doing that, you may discover the reasons why the values are set the way they are, assuming the information does not conflict. And also assuming it matches the state in the clusters. Anyway, while YAML comments, commit messages, and PR discussions can provide context for AI agents, they are not usable by conventional tests and tools. When someone makes a change later, how can the correctness of the change be validated?</p><p>An intent-oriented mixin/trait approach might help some, but this approach doesn’t appear to be commonly implemented in public charts. The <a href="https://argo-cd.readthedocs.io/en/stable/operator-manual/installation/">ArgoCD installation manifests</a> are an example of splitting configuration by high-level intent: namespaced vs. cluster-scoped and HA vs. single-replica. A similar approach could be used with Helm values. Instead, <a href="https://github.com/argoproj/argo-helm/blob/main/charts/argo-cd/README.md">combinations of values are documented</a>.</p><p>Higher-level intent is not generally encoded explicitly in a machine-readable form: high availability vs. not, persistent vs. not, scalable vs. low cost, PCI-DSS compliant, etc. The values are set the way they are, spread across many values files factored by commonality, sometimes with up-to-date comments for humans and/or AI agents.</p><p>So generic Helm charts do not convey intent, and push the variant drift problem into values files. The approach transforms the configuration schemas from standard Kubernetes resources and well known CRDs to templates that are impossible to reason about locally and that have non-standard values schemas, which hopefully are specified with <a href="https://helm.sh/docs/topics/charts/#schema-files">schema files</a>, but have no widely understood semantics, at least outside a single organization and type of workload. This lack of standardization, combined with <a href="https://itnext.io/the-unidirectionality-of-infrastructure-as-code-creates-asymmetry-40c9f5eed959">unidirectional configuration generation</a>, makes configuration as code not interoperable with reusable tools: <a href="https://itnext.io/building-imperative-operational-tools-on-a-declarative-platform-cec24f8efe05">operational tools</a>, <a href="https://itnext.io/configuration-as-code-is-a-liability-for-security-d0e53727cd17">security tools</a>, <a href="https://itnext.io/there-should-be-an-app-for-that-1f4d23286a9e">access-control tools</a>, and <a href="https://itnext.io/building-tools-to-manage-cross-cutting-concerns-in-kubernetes-by-using-confighub-6bd64afcdd69">other tools</a>.</p><p>We’ve been living with this tradeoff for so long that it’s easy to forget that it is a tradeoff. It’s time to consider alternatives not just to Helm, but to configuration as code in general. <a href="https://itnext.io/generating-transforming-and-patching-kubernetes-configuration-with-kustomize-fb7b02476a1b">Kustomize</a> showed that there are <a href="https://itnext.io/kubernetes-configuration-transformation-orchestrators-b998809eda17">other ways</a> to implement <a href="https://itnext.io/how-are-variants-managed-in-confighub-061bc2f89cff">configuration inheritance</a> and traits. <a href="https://itnext.io/introducing-confighub-b127736641c5">ConfigHub</a> builds on lessons learned from <a href="https://itnext.io/kustomize-kpt-porch-and-confighub-28b781336885">kustomize, kpt, and porch</a> to not just provide <a href="https://itnext.io/what-is-configuration-as-data-210b0c4be324">an alternative approach</a>, but an <a href="https://itnext.io/configuration-needs-an-api-b36f08b92551">API for configuration</a> that enables interoperable tools.</p><p>Intent attached to workloads in a well known form can be respected by tools, which are decoupled from the configuration data itself. In ConfigHub, expressions (called <a href="https://docs.confighub.com/background/concepts/filters/">filters</a>) that select what data operations (e.g., <a href="https://docs.confighub.com/guide/functions/">functions</a>) should be applied to can be decoupled from the code, as well. I’ll write another post about how these mechanisms enable management of variant drift based on intent and properties of the configuration data. I’m excited about being able to develop a first-class solution to this problem.</p><p>Have you discovered undesirable variant drift? How do you organize your Helm values files? How many layers of values overlays do you have? Did you build a tool to generate the specification of which values files to use, or just use ArgoCD ApplicationSet templates with conditionals? Do you use non-trivial conditional logic in your templates rather than just generic enable checks? Do you have a way to indicate higher-level intent, such as by using values mixins, or by annotations on the workloads themselves? Do you have any way of validating that rendered values match higher-level intent?</p><p>Reply here, or send me a message on <a href="https://www.linkedin.com/in/bgrant0607/">LinkedIn</a>, <a href="https://x.com/bgrant0607">X/Twitter</a>, or <a href="https://bsky.app/profile/bgrant0607.bsky.social">Bluesky</a>, where I plan to crosspost this.</p><p>You could also <a href="https://auth.confighub.com/sign-up">try out ConfigHub</a>, which is now in preview.</p><p>If you found this interesting, you may be interested in <a href="https://medium.com/@bgrant0607/">my other posts</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f7b8433dd0b8" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/managing-variant-drift-with-generic-helm-templates-f7b8433dd0b8">Managing variant drift with generic Helm templates</a> was originally published in <a href="https://itnext.io">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Abstractions as views over configuration instead of just generators of configuration]]></title>
            <link>https://itnext.io/abstractions-as-views-over-configuration-instead-of-just-generators-of-configuration-8ec6ea8aa746?source=rss-ccdbe9a99319------2</link>
            <guid isPermaLink="false">https://medium.com/p/8ec6ea8aa746</guid>
            <category><![CDATA[gitops]]></category>
            <category><![CDATA[platform-engineering]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[config-as-data]]></category>
            <dc:creator><![CDATA[Brian Grant]]></dc:creator>
            <pubDate>Wed, 12 Aug 2026 16:00:56 GMT</pubDate>
            <atom:updated>2026-08-12T20:53:41.015Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J7lWSvGys4a6SE5TPlmiWg.png" /><figcaption>Image generated by Gemini</figcaption></figure><p>I previously wrote about <a href="https://itnext.io/abstraction-is-the-wrong-way-to-simplify-configuration-81ac4bad02ab">abstraction being the wrong way to simplify configuration</a>. What I really meant was that <a href="https://itnext.io/the-unidirectionality-of-infrastructure-as-code-creates-asymmetry-40c9f5eed959">unidirectional configuration generators</a> were the wrong way to implement abstractions.</p><p>Internal developer platforms and other <a href="https://itnext.io/kubernetes-and-application-platforms-7a36aa5c3c08">application platforms built on Kubernetes</a> present simplified abstractions that don’t expose all Kubernetes functionality, but that also <a href="https://itnext.io/the-tension-between-flexibility-and-simplicity-in-infrastructure-as-code-6cec841e3d16">decreases flexibility</a> and couples <a href="https://itnext.io/building-tools-to-manage-cross-cutting-concerns-in-kubernetes-by-using-confighub-6bd64afcdd69">operational concerns</a> and <a href="https://itnext.io/configuration-as-code-is-a-liability-for-security-d0e53727cd17">security concerns</a> with development concerns. Many such abstractions are also not as widely used standards as the Kubernetes APIs.</p><p>One application abstraction gaining a fair bit of adoption is <a href="https://score.dev/">Score</a>. Score looks a little like Docker Compose, but also contains some prominent Kubernetes features, such as <a href="https://docs.score.dev/docs/score-specification/score-spec-reference/">probes and resources</a>. The <a href="https://github.com/score-spec/score-k8s">score-k8s tool</a> will generate Kubernetes resources from a Score spec. The documentation says “Iterate by changing the score files, and re-running generate.” That’s the typical unidirectional generation approach.</p><p>Since the Score spec doesn’t cover all operational concerns, they must be added somehow. The <a href="https://github.com/score-spec/score-k8s#how-do-i-configure-the-number-of-replicas-or-security-context-for-the-workload-deployment">documentation recommends</a> patching the generated configuration using <a href="https://itnext.io/generating-transforming-and-patching-kubernetes-configuration-with-kustomize-fb7b02476a1b">kustomize</a> or <a href="https://github.com/score-spec/score-k8s#how-do-i-modify-the-generated-manifests-or-add-and-remove-from-them">built-in JSON patch operations</a>. That’s quite a few patches: replicas, security context, pod anti-affinity, sidecar containers, various controller annotations and labels, …</p><p>When using ConfigHub, ConfigHub is the authoritative record of the configuration data. The resources don’t need to be patched over and over again on the fly. Instead, <a href="https://docs.confighub.com/guide/functions/">functions</a> and <a href="https://itnext.io/there-should-be-an-app-for-that-1f4d23286a9e">tools</a> can be used to customize the configuration, and those changes are persisted. However, that means we don’t want some external files to be persisted as the source of truth.</p><p>Instead, we want to regenerate the Score spec from the data in ConfigHub. I built a tool to do just that, <a href="https://github.com/confighub/examples/tree/main/k8s-to-score">k8s-to-score</a>. Point it at a space and it identifies the workloads and generates Score files for them. For example:</p><pre>% k8s-to-score --space nonprod-c3agent --out-dir /tmp/agent-score</pre><p>One of the components it generated looks like this:</p><pre>apiVersion: score.dev/v1b1<br>metadata:<br>  name: c3agent-gateway<br>containers:<br>  gateway:<br>    image: c3agent-gateway:sha-eaedf4dfa33ef25dd1e8b06e2f31bd2d007ce412<br>    resources:<br>      limits:<br>        cpu: 500m<br>        memory: 512Mi<br>      requests:<br>        cpu: 10m<br>        memory: 128Mi<br>    variables:<br>      CONTROL_PLANE_ADDR: c3agent-controlplane:50050<br>      CP_GRPC_PORT: &quot;50050&quot;<br>      CP_HTTP_PORT: &quot;8080&quot;<br>      CP_SCHEDULE_INTERVAL_SECS: &quot;7&quot;<br>      DEFAULT_MAX_BUDGET_USD: &quot;10.00&quot;<br>      GATEWAY_ADDR: c3agent-gateway:50052<br>      GW_GRPC_PORT: &quot;50052&quot;<br>      MAX_CONCURRENT_TASKS: &quot;5&quot;<br>      TASK_NAMESPACE: c3agent<br>      TASK_PVC: c3agent-taskdata<br>service:<br>  ports:<br>    grpc:<br>      port: 50052</pre><p>This obviously doesn’t reflect all of the details in the Kubernetes Deployment, Service, Ingress, PodDisruptionBudget, HorizontalPodAutoscaler, and other related resources. However, when score-k8s generateis run on this, it roundtrips to a subset of the original resources.</p><p>If the Score spec is changed and then the manifests are regenerated, the output can be <em>merged</em> back into ConfigHub’s full representations of the resources. For example, if the cpu request is changed to 100m, that’s also updated in the output:</p><pre>...<br>            requests:<br>              cpu: 100m<br>              memory: 128Mi<br>...</pre><p>And that change can be merged back into the copy of the resource in ConfigHub.</p><p>ConfigHub supports several patterns where configuration changes can be merged into other units, including <a href="https://docs.confighub.com/guide/variants/#merging-external-sources-into-confighub">merging changes from external sources</a>. <a href="https://itnext.io/a-new-kubernetes-package-installer-324db6f615ac">The installer</a> supports this, as well, so that configuration can be re-rendered if inputs or other choices or facts change, and then the changes in the rendered output can be merged back into ConfigHub.</p><p>Through this approach the fully rendered configuration data can serve as the authoritative record, with the external platform abstraction as just one temporary, partial view of the resources. The abstraction can then interoperate with other users, agents, functions, or tools that modify the resources. For example, operators can use views and tools that expose details that are relevant to their responsibilities.</p><p>This kind of interoperability is typical and expected of API-based tools. For example, one expects to see changes made with a CLI through a web UI for the same service, and vice versa. However, this is not the case with configuration as code — changes made through other tools cause configuration drift. In this case, all changes are being made to the same authoritative copy of the configuration data, so there is no drift.</p><p>Configuration as code has made us accustomed to sacrificing the benefits of APIs in the name of repeatability (i.e., the ability to <a href="https://itnext.io/variant-generation-is-the-primary-capability-of-infrastructure-as-code-773e6b404b49">create additional variants</a>). That tradeoff was expedient, but unnecessary, and suboptimal. I hope to see wider adoption of view-based abstractions in the future, as opposed to just generator-based abstractions.</p><p>Have you run into the challenge where some team needs one more API feature exposed through your platform abstraction, and then that happens repeatedly for different features? How did you handle that? Did you expose them and erode your abstraction? Did you support patches? Have you found developer-oriented abstractions to hamper operational changes due to the use of configuration generators like Score, Helm, or Crossplane compositions? Have you built platform abstractions using the view/projection approach rather than generators? If so, were they built directly on Kubernetes APIs, or on top of configuration files in git, or on top of another storage system?</p><p>Reply here, or send me a message on <a href="https://www.linkedin.com/in/bgrant0607/">LinkedIn</a>, <a href="https://x.com/bgrant0607">X/Twitter</a>, or <a href="https://bsky.app/profile/bgrant0607.bsky.social">Bluesky</a>, where I plan to crosspost this.</p><p>You could also <a href="https://auth.confighub.com/sign-up">try out ConfigHub</a>, which is now in preview.</p><p>If you found this interesting, you may be interested in <a href="https://medium.com/@bgrant0607/">my other posts</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8ec6ea8aa746" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/abstractions-as-views-over-configuration-instead-of-just-generators-of-configuration-8ec6ea8aa746">Abstractions as views over configuration instead of just generators of configuration</a> was originally published in <a href="https://itnext.io">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building imperative operational tools on a declarative platform]]></title>
            <link>https://itnext.io/building-imperative-operational-tools-on-a-declarative-platform-cec24f8efe05?source=rss-ccdbe9a99319------2</link>
            <guid isPermaLink="false">https://medium.com/p/cec24f8efe05</guid>
            <category><![CDATA[crossplane]]></category>
            <category><![CDATA[gitops]]></category>
            <category><![CDATA[config-as-code]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[config-as-data]]></category>
            <dc:creator><![CDATA[Brian Grant]]></dc:creator>
            <pubDate>Fri, 07 Aug 2026 20:36:04 GMT</pubDate>
            <atom:updated>2026-08-09T17:45:06.876Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tx9CSH2uTaFZTfrGNiNLbw.png" /><figcaption>Image generated by Gemini</figcaption></figure><p>Can internal developer platforms and imperative operational tools coexist? I believe they can, if one builds on the right model.</p><p>I have recently been writing about building tools to perform Kubernetes management tasks (<a href="https://itnext.io/there-should-be-an-app-for-that-1f4d23286a9e">1</a>, <a href="https://itnext.io/configuration-as-code-is-a-liability-for-security-d0e53727cd17">2</a>, <a href="https://itnext.io/building-tools-to-manage-cross-cutting-concerns-in-kubernetes-by-using-confighub-6bd64afcdd69">3</a>). Performing tasks with imperative tools can be more goal-oriented, direct, concise, and ergonomic than editing large, complex configuration-as-code files that must accommodate all conceivable tasks, and can help perform common tasks quickly, correctly, and safely. Imperative tools can also be learned quickly by both humans and AI agents, without reading pages of reference documentation, especially if they have intuitive command names and decent help text. They can also be relatively efficient in terms of token usage, and well scoped, hierarchical commands naturally provide progressive disclosure. Autocompletion is supported in cases where humans are still typing the commands.</p><p>I also believe that such tools capture expertise more effectively and in a more reusable form than configuration-as-code templates and modules. For some of the tools, a fair bit of the creation time was spent on “product” research and product definition, and I’m already very familiar with <a href="https://itnext.io/the-technical-history-of-kubernetes-2fe1988b522a">Kubernetes</a> and <a href="https://medium.com/@bgrant0607/is-the-kubernetes-cncf-ecosystem-too-big-complex-and-fragmented-350d52f37e31">its ecosystem</a>.</p><p>What makes such tools possible is storing <a href="https://itnext.io/what-is-configuration-as-data-210b0c4be324">configuration as data</a> in <a href="https://itnext.io/configuration-belongs-in-a-database-be6d31f46d3d">a database</a> with <a href="https://itnext.io/configuration-needs-an-api-b36f08b92551">an API</a>, as opposed to <a href="https://itnext.io/the-12-anti-factors-of-infrastructure-as-code-acb52fba3ae0">configuration as code</a>. This enables us to build imperative tools, like, say, kubectl. Tools that can <a href="https://itnext.io/configuration-editing-is-imperative-fa9db379fbe4">imperatively change</a> the configuration data. A lot of functionality in kubectl, such as kubectl edit, is <a href="https://itnext.io/infrastructure-as-code-is-too-generic-a3d267c5d0eb">relatively generic</a>. But tools can be as general or as specific, such as the tools for managing Kubernetes RBAC and NetworkPolicy, as we like.</p><p>As with Kubernetes itself, the configuration data stored in ConfigHub is (primarily, for now, other than <a href="https://itnext.io/the-challenge-of-configmap-rollouts-in-kubernetes-1431398c0866">application configuration</a>) <a href="https://itnext.io/on-using-the-kubernetes-resource-model-for-declarative-configuration-a5dd11b99546">Kubernetes resources, or rather KRM resources, which are declarative</a>. Using controllers like <a href="https://www.crossplane.io/">Crossplane</a>, <a href="https://docs.aws.amazon.com/eks/latest/userguide/ack.html">ACK</a>, and <a href="https://docs.cloud.google.com/config-connector/docs/overview">Config Connector</a>, KRM resources can also <a href="https://itnext.io/using-the-kubernetes-resource-model-to-provision-cloud-infrastructure-dc8b75a4e328">provision cloud resources</a>.</p><p>This <a href="https://itnext.io/reflections-on-declarative-configuration-c2fe1c1e50d5">serialized representation of the desired state</a> can be changed <a href="https://itnext.io/validating-scoring-and-gating-kubernetes-configuration-after-every-change-a11fbab14110">and validated</a> prior to being applied. A <a href="https://itnext.io/is-gitops-actually-useful-a1c851ba99d8">GitOps tool</a>, such as ArgoCD or Flux, can perform the apply by pulling the configuration from <a href="https://docs.confighub.com/guide/gitops/">ConfigHub’s OCI endpoint</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iB5i13UeevjwOmXi9hgFXg.png" /></figure><p>Unlike resources stored by the Kubernetes control plane, the configuration data in ConfigHub is automatically versioned. ConfigHub also <a href="https://itnext.io/how-are-variants-managed-in-confighub-061bc2f89cff">supports approaches</a> to keep <a href="https://itnext.io/variant-generation-is-the-primary-capability-of-infrastructure-as-code-773e6b404b49">multiple variants of a configuration</a> in sync. These are the two main capabilities that would otherwise normally <a href="https://itnext.io/what-drives-users-to-infrastructure-as-code-848e8640a506">drive users to configuration as code tools</a>. Instead, we have a declarative system of record that we can operate upon with tools in a repeatable, scalable manner.</p><p>As another tool-building demonstration, I created a tool to manage EKS clusters inspired by <a href="https://docs.aws.amazon.com/eks/latest/eksctl/what-is-eksctl.html">eksctl</a>. eksctl provides a simpler experience for creating EKS clusters compared to, say, Terraform or OpenTofu, with developer-friendly imperative commands. To use it, one doesn’t need to know all of the details of the EKS APIs. For example, you can create a cluster with eksctl create cluster.</p><p>Under the hood, eksctl generates CloudFormation. Instead of generating CloudFormation configuration, cub-eks creates, updates, reads, and validates <a href="https://docs.crossplane.io/latest/managed-resources/managed-resources/">Crossplane managed resources</a> in ConfigHub. The code is here:</p><p><a href="https://github.com/confighub/examples/tree/main/eks-manager">https://github.com/confighub/examples/tree/main/eks-manager</a></p><p>Why managed resources rather than <a href="https://docs.crossplane.io/latest/composition/composite-resources/">composite resources</a>? Like configuration as code, <a href="https://docs.crossplane.io/latest/composition/compositions/">Crossplane compositions</a> <a href="https://itnext.io/abstraction-is-the-wrong-way-to-simplify-configuration-81ac4bad02ab">abstract the underlying managed resources</a>, and are implemented using <a href="https://itnext.io/the-unidirectionality-of-infrastructure-as-code-creates-asymmetry-40c9f5eed959">unidirectional configuration generators</a>. Crossplane compositions expect that all changes (with a few special exceptions) to the generated resources will be made by changing the composition input values or the <a href="https://docs.crossplane.io/latest/composition/compositions/#install-a-composition-function">composition functions</a>. This is called <a href="https://itnext.io/kubernetes-configuration-and-infrastructure-as-code-taxonomy-62ab4dd7e82f"><em>exclusive actuation</em></a> (similar to <a href="https://itnext.io/what-is-it-with-template-catalogs-7637c24d5200">template catalogs</a>), and obstructs composable, interoperable tools that read-modify-write the resources, which is what we’re trying to enable. Any out-of-band changes are considered <a href="https://itnext.io/why-configuration-drift-is-so-hard-to-avoid-in-practice-443248cafc9c">configuration drift</a>. Furthermore, I typically see additional generators, such as Helm and/or <a href="https://backstage.io/docs/features/software-templates/writing-templates/">Backstage scaffolder templates</a>, layered on top of composite resources, which makes the managed resources even harder to change when operationally necessary.</p><p>That doesn’t mean the tools can’t provide opinionated experiences over the underlying resources. For example, I decided to only support <a href="https://docs.aws.amazon.com/eks/latest/userguide/automode-get-started-eksctl.html">EKS auto mode</a>. But it is <a href="https://platformengineering.org/blog/what-are-golden-paths-a-guide-to-streamlining-developer-workflows">a golden path</a>, or part of one, as opposed to an <a href="https://itnext.io/the-tension-between-flexibility-and-simplicity-in-infrastructure-as-code-6cec841e3d16">inflexible constraint</a>.</p><p>Hard and soft guardrails are still possible to enforce, via <a href="https://docs.confighub.com/guide/validation-and-policies/">Triggers</a>. This command:</p><pre>% cub-eks guardrails install</pre><p>creates a number of default guardrails to catch changes that would violate policies. A guardrail was also added regarding auto mode, at least for now, but it’s just a warning. You can see what it created by listing the Triggers and what they check (expressed using <a href="https://cel.dev/">CEL</a>):</p><pre>% cub trigger list --space eks-policy -o yq=&#39;[.[] | {(.Trigger.Slug): .Trigger.Arguments[0].Value}]&#39;<br>- eks-automode-invariant: &#39;!(r.kind == &#39;&#39;Cluster&#39;&#39; &amp;&amp; has(r.spec.forProvider.computeConfig)) || (has(r.spec.forProvider.storageConfig) &amp;&amp; has(r.spec.forProvider.kubernetesNetworkConfig))&#39;<br>- eks-control-plane-logging: &#39;!(r.kind == &#39;&#39;Cluster&#39;&#39;) || (has(r.spec.forProvider.enabledClusterLogTypes) &amp;&amp; &#39;&#39;audit&#39;&#39; in r.spec.forProvider.enabledClusterLogTypes)&#39;<br>- eks-no-latinit: &#39;!(r.kind == &#39;&#39;NodeGroup&#39;&#39; &amp;&amp; has(r.spec.managementPolicies)) || !(&#39;&#39;LateInitialize&#39;&#39; in r.spec.managementPolicies)&#39;<br>- eks-pinned-version-extended-support: &#39;!(r.kind == &#39;&#39;Cluster&#39;&#39; &amp;&amp; has(r.spec.forProvider.version)) || (has(r.spec.forProvider.upgradePolicy) &amp;&amp; r.spec.forProvider.upgradePolicy.supportType == &#39;&#39;EXTENDED&#39;&#39;)&#39;<br>- eks-private-endpoint: &#39;!(r.kind == &#39;&#39;Cluster&#39;&#39; &amp;&amp; has(r.spec.forProvider.vpcConfig) &amp;&amp; has(r.spec.forProvider.vpcConfig.endpointPublicAccess) &amp;&amp; r.spec.forProvider.vpcConfig.endpointPublicAccess) || (has(r.spec.forProvider.vpcConfig.publicAccessCidrs) &amp;&amp; !(&#39;&#39;0.0.0.0/0&#39;&#39; in r.spec.forProvider.vpcConfig.publicAccessCidrs))&#39;<br>- eks-secrets-encryption: &#39;!(r.kind == &#39;&#39;Cluster&#39;&#39;) || has(r.spec.forProvider.encryptionConfig)&#39;</pre><p>We can also check where they are registered and whether they have surfaced any gates or warnings:</p><pre>% cub-eks guardrails status -o table<br>SPACE             UNIT     GATES  WARNINGS<br>eks-prod-usw2     cluster  0      3<br>eks-staging-use1  cluster  0      1</pre><p>(The output is JSON by default for AI agents.)</p><p>We can discover those and some additional issues are with cub-eks findings:</p><pre>% cub-eks findings -o table<br>SEVERITY  ANALYZER             CLUSTER       KIND       NAME          MESSAGE<br>HIGH      exposure             prod-usw2     Cluster    prod-usw2     the Kubernetes API endpoint is publicly accessible with publicAccessCidrs 0.0.0.0/0<br>MEDIUM    autoscaler-conflict  demo-use1     NodeGroup  system        scalingConfig.desiredSize (3) is set under forProvider with Update management: Crossplane will reconcile it on every pass, fighting Cluster Autoscaler or Karpenter<br>MEDIUM    autoscaler-conflict  demo-use1     NodeGroup  system-v2     scalingConfig.desiredSize (3) is set under forProvider with Update management: Crossplane will reconcile it on every pass, fighting Cluster Autoscaler or Karpenter<br>MEDIUM    autoscaler-conflict  prod-usw2     NodeGroup  main          scalingConfig.desiredSize (4) is set under forProvider with Update management: Crossplane will reconcile it on every pass, fighting Cluster Autoscaler or Karpenter<br>MEDIUM    exposure             prod-usw2     Cluster    prod-usw2     no encryptionConfig: Kubernetes Secrets are not encrypted with a KMS key<br>MEDIUM    exposure             staging-use1  Cluster    staging-use1  no encryptionConfig: Kubernetes Secrets are not encrypted with a KMS key<br>LOW       version-skew         demo-use1     NodeGroup  batch         node group is on 1.34, 1 minor version(s) behind the control plane (1.35)<br>LOW       version-skew         demo-use1     NodeGroup  system        node group is on 1.34, 1 minor version(s) behind the control plane (1.35)<br>LOW       version-skew         demo-use1     NodeGroup  system-v2     node group is on 1.34, 1 minor version(s) behind the control plane (1.35)<br>LOW       exposure             prod-usw2     Cluster    prod-usw2     control-plane logging is missing: audit, authenticator</pre><p>(The demo-use1 cluster was created to test the guardrails.)</p><p>cub-eks plan can flag changes that would cause disruptions, such as node re-creation, or cause Crossplane’s provider implementation to not be able to update the cluster or nodepool due to immutable AWS API fields.</p><pre>% cub-eks plan -o table<br>DISRUPTION           CLUSTER    UNIT       REV   RESOURCE   CHANGED<br>replace              demo-use1  ng-system  1-&gt;3  system     instanceTypes.0, scalingConfig.maxSize<br>rolling              demo-use1  ng-batch   2-&gt;3  batch      version<br>in-place-disruptive  demo-use1  cluster    1-&gt;2  demo-use1  version<br><br>  eks-demo-use1/ng-system — REPLACE<br>    instanceTypes.0: instanceTypes is immutable<br>    This cannot be reconciled in place. Crossplane will refuse the update and<br>    retry forever; the Unit will read as applied while AWS is never changed.<br>    -&gt; this replaces the node group — use `replace-nodegroup` for a blue/green swap, or revert the field<br><br>3 Unit(s) with pending changes: 1 blocking, 1 rolling, 1 in-place.<br>43 Unit(s) have never been applied (no baseline to compare against).</pre><p>In order to do these kinds of checks prior to deployment when using Crossplane compositions, one would need to render the compositions client-side, such as in CI.</p><p>We can also just inspect the state with domain-specific commands — snapshot, versions, list, and get:</p><pre>% cub-eks snapshot -o table<br>CLUSTER       VERSION  REGION     MODE     NODEGROUPS  ADDONS  NETWORK  IAM  UNITS  GATED  UNAPPLIED<br>demo-use1     1.35     us-east-1  auto     3           1       2        1    8      0      8<br>prod-usw2     1.33     us-west-2  classic  1           0       0        0    2      0      2<br>staging-use1  1.34     us-east-1  auto     1           1       24       9    36     0      36<br><br>3 clusters, 5 nodegroups, 2 addons, 46 units (0 gated, 46 unapplied)<br><br><br>% cub-eks versions -o table<br>CLUSTER       ENV      VERSION  SUPPORT   MODE     NODEGROUPS  MAX SKEW  ADDONS<br>demo-use1     dev      1.35     EXTENDED  auto     3           1 behind  1<br>prod-usw2     prod     1.33     EXTENDED  classic  1           -         0<br>staging-use1  staging  1.34     EXTENDED  auto     1           -         1<br><br>versions: 1.35 x1, 1.34 x1, 1.33 x1<br>3 clusters, 3 distinct versions, 1 with node-group skew, 0 pinned under STANDARD support<br><br><br>% cub-eks list -o table<br>CLUSTER       GROUP  KIND                   NAME                                                       UNIT<br>...<br>staging-use1  ec2    EIP                    staging-use1-nat-a-eip                                     eip-nat-a<br>staging-use1  ec2    InternetGateway        staging-use1-igw                                           igw<br>staging-use1  ec2    NATGateway             staging-use1-nat-a                                         nat-a<br>staging-use1  ec2    Route                  staging-use1-private-a-default                             route-private-a-nat<br>staging-use1  ec2    Route                  staging-use1-private-b-default                             route-private-b-nat<br>staging-use1  ec2    Route                  staging-use1-private-c-default                             route-private-c-nat<br>staging-use1  ec2    Route                  staging-use1-public-default                                route-public-igw<br>staging-use1  ec2    RouteTable             staging-use1-private-a-rt                                  rt-private-a<br>staging-use1  ec2    RouteTable             staging-use1-private-b-rt                                  rt-private-b<br>staging-use1  ec2    RouteTable             staging-use1-private-c-rt                                  rt-private-c<br>staging-use1  ec2    RouteTable             staging-use1-public-rt                                     rt-public<br>staging-use1  ec2    RouteTableAssociation  staging-use1-private-a-assoc                               rta-private-a<br>staging-use1  ec2    RouteTableAssociation  staging-use1-private-b-assoc                               rta-private-b<br>staging-use1  ec2    RouteTableAssociation  staging-use1-private-c-assoc                               rta-private-c<br>staging-use1  ec2    RouteTableAssociation  staging-use1-public-a-assoc                                rta-public-a<br>staging-use1  ec2    RouteTableAssociation  staging-use1-public-b-assoc                                rta-public-b<br>staging-use1  ec2    RouteTableAssociation  staging-use1-public-c-assoc                                rta-public-c<br>staging-use1  ec2    Subnet                 staging-use1-private-a                                     subnet-private-a<br>staging-use1  ec2    Subnet                 staging-use1-private-b                                     subnet-private-b<br>staging-use1  ec2    Subnet                 staging-use1-private-c                                     subnet-private-c<br>staging-use1  ec2    Subnet                 staging-use1-public-a                                      subnet-public-a<br>staging-use1  ec2    Subnet                 staging-use1-public-b                                      subnet-public-b<br>staging-use1  ec2    Subnet                 staging-use1-public-c                                      subnet-public-c<br>staging-use1  ec2    VPC                    staging-use1-vpc                                           vpc<br>staging-use1  eks    Addon                  aws-efs-csi-driver                                         addon-aws-efs-csi-driver<br>staging-use1  eks    Cluster                staging-use1                                               cluster<br>staging-use1  eks    NodeGroup              batch                                                      nodegroup-batch<br>staging-use1  iam    Role                   staging-use1-cluster-role                                  cluster-role<br>staging-use1  iam    Role                   staging-use1-node-role                                     node-role<br>staging-use1  iam    RolePolicyAttachment   staging-use1-cluster-role-AmazonEKSBlockStoragePolicyV2    cluster-role-amazoneksblockstoragepolicyv2<br>staging-use1  iam    RolePolicyAttachment   staging-use1-cluster-role-AmazonEKSClusterPolicy           cluster-role-amazoneksclusterpolicy<br>staging-use1  iam    RolePolicyAttachment   staging-use1-cluster-role-AmazonEKSComputePolicy           cluster-role-amazonekscomputepolicy<br>staging-use1  iam    RolePolicyAttachment   staging-use1-cluster-role-AmazonEKSLoadBalancingPolicy     cluster-role-amazoneksloadbalancingpolicy<br>staging-use1  iam    RolePolicyAttachment   staging-use1-cluster-role-AmazonEKSNetworkingPolicy        cluster-role-amazoneksnetworkingpolicy<br>staging-use1  iam    RolePolicyAttachment   staging-use1-node-role-AmazonEC2ContainerRegistryPullOnly  node-role-amazonec2containerregistrypullonly<br>staging-use1  iam    RolePolicyAttachment   staging-use1-node-role-AmazonEKSWorkerNodeMinimalPolicy    node-role-amazoneksworkernodeminimalpolicy<br><br>46 resources<br><br><br>% cub-eks get demo-use1 -o table<br>CLUSTER demo-use1<br><br>  control plane    demo-use1<br>  apiVersion       eks.aws.upbound.io/v1beta2<br>  version          1.35<br>  region           us-east-1<br>  compute          auto mode (general-purpose,system)<br>  endpoint         private=yes public=no<br>  logging          api,audit,authenticator<br>  encryption       yes<br>  auth mode        API<br>  support type     EXTENDED<br>  deletion policy  Orphan<br>  mgmt policies    Observe,Create,Update,Delete<br>  unit             eks-demo-use1/cluster<br><br>NODE GROUPS<br>  NAME       VERSION  INSTANCE TYPES         CAPACITY   MIN  MAX  DESIRED   UNIT<br>  batch      1.34     m6i.xlarge,m6a.xlarge  SPOT       0    20   2 (init)  ng-batch<br>  system     1.34     m6i.xlarge             ON_DEMAND  3    12   3         ng-system<br>  system-v2  1.34     m6i.2xlarge            ON_DEMAND  3    12   3         nodegroup-system-v2<br><br>ADDONS<br>  NAME     VERSION             ON UPDATE  PRESERVE  UNIT<br>  vpc-cni  v1.19.5-eksbuild.1  OVERWRITE  yes       addon-vpc-cni<br><br>NETWORKING<br>  Subnet  1  demo-use1-private-a<br>  VPC     1  demo-use1-vpc<br><br>IAM<br>  Role  1  demo-use1-cluster-role</pre><p>Remember, this is a fleet-wide view. Using the default JSON output, AI agents can also build summary tables (e.g., using python) based on what aspects you’d like to see.</p><p>But it’s the commands that update the configuration that reach beyond what we could do with configuration as code.</p><p>These clusters and nodepools were created with cub-eks create, using the typical CLI approach of specifying values using command-line flags rather than a configuration file. Unlike configuration as code, we don’t need to regenerate the resources in order to make changes, so we don’t need to save an input configuration file as a source of truth. We can just change the resources. For safety, the mutating commands, such as scale and upgrade, use dry run by default. When “committing” changes (without dry run), I required change descriptions and asked the agent to convey what it was asked to do, but you could do differently if you preferred.</p><pre>% cub-eks scale nodegroup eks-demo-use1/ng-system --nodes-max 15 -o table<br>PATH                   VALUE  DISRUPTION<br>scalingConfig.maxSize  15     in-place<br><br>Dry run — 1 resource(s) would change in eks-demo-use1/ng-system. Re-run with --commit --change-desc &quot;…&quot; to write.<br><br><br>% cub-eks upgrade nodegroup eks-demo-use1/ng-system --to 1.35 -o table<br>PATH     VALUE  DISRUPTION<br>version  1.35   rolling<br><br>  warning: EKS will drain and replace every node in this group; check updateConfig.maxUnavailable and PodDisruptionBudgets first<br><br>Dry run — 1 resource(s) would change in eks-demo-use1/ng-system. Re-run with --commit --change-desc &quot;…&quot; to write.<br><br><br>% cub-eks upgrade cluster demo-use1 --to 1.36 -o table<br>Upgrade demo-use1: 1.35 -&gt; 1.36<br><br>  #  STAGE              RESOURCE   FROM                TO                      DISRUPTION           STATUS<br>  1  nodegroup-catchup  batch      1.34                1.35                    rolling              REQUIRED FIRST<br>  2  nodegroup-catchup  system     1.34                1.35                    rolling              REQUIRED FIRST<br>  3  nodegroup-catchup  system-v2  1.34                1.35                    rolling              REQUIRED FIRST<br>  4  control-plane      demo-use1  1.35                1.36                    in-place-disruptive  pending<br>  5  addon              vpc-cni    v1.19.5-eksbuild.1  (compatible with 1.36)  in-place-disruptive  pending<br>  6  nodegroup          batch      1.34                1.36                    rolling              pending<br>  7  nodegroup          system     1.34                1.36                    rolling              pending<br>  8  nodegroup          system-v2  1.34                1.36                    rolling              pending<br><br>  1. batch: behind the current control plane; catch up before the control plane moves<br>  2. system: behind the current control plane; catch up before the control plane moves<br>  3. system-v2: behind the current control plane; catch up before the control plane moves<br>  4. demo-use1: multi-minute; EKS allows only one cluster update at a time<br>  5. vpc-cni: check the addon version compatible with 1.36 before setting it<br>  6. batch: drains and replaces every node in the group<br>  7. system: drains and replaces every node in the group<br>  8. system-v2: drains and replaces every node in the group<br><br>BLOCKED — node groups are behind the current control plane and must be caught up<br>first. Moving the control plane now would strand them further behind than EKS<br>supports, and node groups only advance one minor version at a time.<br><br>Run these, apply each, and wait for the nodes to roll:<br>  bin/cub-eks upgrade nodegroup eks-demo-use1/ng-batch --to 1.35 --commit --change-desc &quot;…&quot;<br>  bin/cub-eks upgrade nodegroup eks-demo-use1/ng-system --to 1.35 --commit --change-desc &quot;…&quot;<br>  bin/cub-eks upgrade nodegroup eks-demo-use1/nodegroup-system-v2 --to 1.35 --commit --change-desc &quot;…&quot;<br><br>Then re-run this command.</pre><p>Note that the last upgrade command was blocked by the prerequisite checks. The system-v2 nodegroup was created by cub-eks replace-nodegroup, which is used to create a new nodegroup that changes some immutable field of an existing nodegroup. eksctl doesn’t have a command to do this, but <a href="https://docs.aws.amazon.com/eks/latest/userguide/migrate-stack.html">has instructions</a>.</p><p>We could also implement more sophisticated orchestration and workflows in such tools, though we might also want to implement it as a server, such as a MCP server, if we did that. Such a server could be run with the ability to access the cluster, so that it could orchestrate the <a href="https://kubernetes.io/docs/tasks/administer-cluster/safely-drain-node/">draining of the nodes</a>, for example. Note that the <a href="https://github.com/confighub/examples/tree/main/workload-manager">workload-manager</a> tool is responsible for ensuring that the workloads have PodDisruptionBudget and proper anti-affinity constraints.</p><p>There’s also a fleet-edit command for making <a href="https://itnext.io/making-mass-changes-to-infrastructure-as-code-b1edea2f7c48">mass changes</a> across multiple clusters — again, starting with dry run by default. It automatically chooses an appropriate <a href="https://docs.confighub.com/guide/functions/">function</a> to run to change the specified fields based on their data types. These changes are validated by Triggers and can be reviewed before being released to GitOps tools like ArgoCD and Flux.</p><p>In the case that clusters and nodepools should have similar properties, they can be created as <a href="https://itnext.io/how-are-variants-managed-in-confighub-061bc2f89cff">variants</a> of the same base configuration. One would then first update the base and <em>upgrade</em> the downstream variants. The cub-eks promote command will do that, but so will the standard cub variant promote. This enables similar inheritance as compositions, templates, or modules, but with full transparency and granular control over <a href="https://itnext.io/modularity-and-blast-radius-in-infrastructure-as-code-9a1f6432196e">blast radius</a>.</p><p>I only spent about a day on this tool. I’m sure a lot more could be done, such as creating the MCP server version of the tool, or adding support for Karpenter, or (v2) namespace-scoped Crossplane resources, or ACK, or being able to reuse an existing VPC. I also didn’t yet create agent skills the way I did for the other tools. Would you simplify the cluster and/or nodegroup creation options, and make it more opinionated? What would be most useful?</p><p>What I find compelling about this is that we can use AI to build deterministic, testable, ergonomic, domain-specific tools that can inspect, modify, and validate the desired state based on standard, stable APIs. As opposed to AI agents directly editing layers of bespoke snowflake templates that are then rendered at apply time and generally would not have this level of domain-specific validation and impact analysis. The ability to orchestrate changes, similar to Kubernetes controllers, but with pre-deployment validation, versioning, etc. is also enticing, and I plan to explore that more in the future.</p><p>What kinds of operational changes do you “break glass” to make directly to the live state because of the complexity of making the changes through configuration as code or infrastructure as code? What kinds of operational tools would you build if it were feasible to do without creating configuration drift? Do you often need to make similar changes to different compositions, templates, or modules? Can you query your input values across your fleet? How about the rendered output? Do you generally make changes to input values by hand or using tools? How do you decouple operational concerns from developer concerns? What do you do when one-size-fits-all abstractions block valid use cases? Are there other CloudFormation-based tools that you wish could target Crossplane or ACK resources?</p><p>Reply here, or send me a message on <a href="https://www.linkedin.com/in/bgrant0607/">LinkedIn</a>, <a href="https://x.com/bgrant0607">X/Twitter</a>, or <a href="https://bsky.app/profile/bgrant0607.bsky.social">Bluesky</a>, where I plan to crosspost this.</p><p>You could also <a href="https://auth.confighub.com/sign-up">try out ConfigHub</a>, which is now in preview.</p><p>If you found this interesting, you may be interested in <a href="https://medium.com/@bgrant0607/">my other posts</a>.</p><p>Thanks to <a href="https://www.linkedin.com/in/ilyadmitrichenko/">Ilya Dmitrichenko</a>, <a href="https://github.com/eksctl-io/eksctl/commits?author=errordeveloper">creator of </a><a href="https://github.com/eksctl-io/eksctl/commits?author=errordeveloper">eksctl</a>, for providing valuable feedback on this post.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cec24f8efe05" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/building-imperative-operational-tools-on-a-declarative-platform-cec24f8efe05">Building imperative operational tools on a declarative platform</a> was originally published in <a href="https://itnext.io">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kubernetes configuration transformation orchestrators]]></title>
            <link>https://itnext.io/kubernetes-configuration-transformation-orchestrators-b998809eda17?source=rss-ccdbe9a99319------2</link>
            <guid isPermaLink="false">https://medium.com/p/b998809eda17</guid>
            <category><![CDATA[kustomize]]></category>
            <category><![CDATA[config-as-data]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Brian Grant]]></dc:creator>
            <pubDate>Thu, 06 Aug 2026 03:58:17 GMT</pubDate>
            <atom:updated>2026-08-06T10:29:21.830Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g5VeMFI_ubc-xZQt_SrjMQ.png" /><figcaption>Image generated by Gemini</figcaption></figure><p>I recently wrote about the backgrounds of <a href="https://medium.com/itnext/generating-transforming-and-patching-kubernetes-configuration-with-kustomize-fb7b02476a1b">kustomize</a> and <a href="https://medium.com/itnext/kustomize-kpt-porch-and-confighub-28b781336885">kpt</a>, and about a <a href="https://medium.com/itnext/a-new-kubernetes-package-installer-324db6f615ac">new package installer</a> I created that uses kustomize and <a href="https://docs.confighub.com/guide/functions/">ConfigHub functions</a> as kustomize transformer plugins under the hood. These tools manipulate <a href="https://itnext.io/what-is-configuration-as-data-210b0c4be324">configuration as data</a> using sequences of transformations on the data. They are Kubernetes configuration transformation orchestrators.</p><p>The orchestrator makes it possible to compose multiple function executions without generating code to call each function on the same configuration data payload(s). The combination of the orchestrator and <a href="https://github.com/confighub/sdk/blob/main/function-impl/README.md">function SDK</a> help with loading and storing configuration data, unmarshaling and re-marshaling the data, sequencing function executions (e.g., invoking mutating functions before validating functions), either fetching the function implementations (in the case of containers) or calling out to function servers (in the case of <a href="https://github.com/confighub/examples/tree/main/custom-workers/kyverno">ConfigHub custom workers</a>), selecting resources to invoke functions on, normalizing the configuration data, preserving comments, recording metadata, computing function arguments, validating function arguments, etc.</p><p>Kustomize, kpt, and the installer execute linear pipelines of transformation functions. They take Kubernetes YAML (<a href="https://medium.com/itnext/on-using-the-kubernetes-resource-model-for-declarative-configuration-a5dd11b99546">KRM</a>) as input, along with some parameter values from kustomization.yaml, Kptfile, or installer inputs and/or facts, and produce Kubernetes YAML as output. Kustomize and the installer transform <a href="https://itnext.io/kubernetes-configuration-and-infrastructure-as-code-taxonomy-62ab4dd7e82f">DRY</a> configuration out of place, with the output distinct from the input, and kpt and ConfigHub transform WET configuration in place.</p><p>Kpt generally takes function inputs from ConfigMaps. For example:</p><pre>pipeline:<br>  mutators:<br>    - name: set-tier-label<br>      image: set-labels:latest<br>      configMap:<br>        tier: mysql</pre><p>This works fine for the common case where values are provided as inputs and propagated by functions to all destinations in the configuration where they are needed, independently by every pipeline step.</p><p>You may be wondering how that is better than template variables, <a href="https://github.com/kubernetes-sigs/kustomize/issues/2052">kustomize vars</a>, or <a href="https://catalog.kpt.dev/apply-setters/v0.1/">kpt setters</a>.</p><p>First, that’s not the main point of the transformation approach. The objective is to unblock the capabilities that <a href="https://medium.com/itnext/a-concrete-example-of-configuration-as-data-575115e2ba46">configuration as data</a> enables, such as the ability to <a href="https://itnext.io/there-should-be-an-app-for-that-1f4d23286a9e">build interoperable applications on top of the configuration data</a> as easily as for live clusters, make <a href="https://medium.com/itnext/making-mass-changes-to-infrastructure-as-code-b1edea2f7c48">mass changes to configuration</a>, <a href="https://medium.com/itnext/what-is-bidirectional-gitops-ce0ced75fa1c">automatically incorporate changes made to the live state</a>, and <a href="https://docs.confighub.com/guide/authoring-config/">generate configuration without being wedded to a specific generation format</a> forever. For these approaches to work, it’s important for the configuration to be expressed using standard Kubernetes APIs (and popular CRDs), with <a href="https://github.com/kptdev/kpt/issues/3131">no special markup syntax</a>.</p><p>Second, input values are often not pure inputs, but come from somewhere else that are the authoritative sources, which may change independently of the configuration fields that consume the values. Restating them as a independent variable values creates a another source of truth and a drift risk. The values may also be derived from other values rather than just copied verbatim. And the values may be needed in multiple locations that could be determined programmatically, instead of a person needing to carefully wire them throughout the configuration or review the work of an AI agent that did that. Transformation functions are particularly suitable for such cases.</p><p>Third, I find that the transformation approach does have a number of direct benefits, such as mitigating <a href="https://github.com/kubernetes/design-proposals-archive/blob/main/architecture/declarative-application-management.md#parameterization-pitfalls">parameter proliferation</a>, making the <a href="https://medium.com/itnext/i-shouldnt-have-to-read-installer-code-every-day-4dc1e5f9ee1a">configuration more readable</a>, enabling it to be <a href="https://medium.com/itnext/vetting-kubernetes-configuration-with-kyverno-prior-to-deployment-131ee33d315b">validated directly</a>, and decoupling phases of customization, such as configuration authoring-time decisions, installation-time decisions, <a href="https://github.com/confighub/installer/blob/main/docs/principles.md#3-two-layers-of-override-with-clear-lifetimes">post-installation customization</a>, making cross-cutting changes like <a href="https://catalog.kpt.dev/examples/starlark-inject-sidecar/readme/">injecting sidecar containers</a>, <a href="https://medium.com/itnext/how-are-variants-managed-in-confighub-061bc2f89cff">variant creation</a>, and operational changes, like performing rollouts and tuning resources and probe thresholds. Of course, there’s also the common use case for kustomize of patching upstream configuration without forking it. And the case of <a href="https://github.com/kubernetes-sigs/kustomize/blob/master/examples/configGeneration.md">generating ConfigMaps</a>. Embedding application configuration files as multi-line strings in ConfigMaps and templating that has many disadvantages: inability to directly validate the configuration, lack of IDE syntax highlighting and indentation, lack of ability to use it directly in local execution, obfuscation, etc.</p><p>With respect to the second point, in order to extract, transform, and propagate values, it can be useful to be able to chain multiple functions together so that customization logic can be decomposed into more reusable functions. In kustomize and kpt, transformer functions don’t pass values to each other directly, but it is possible to write values to known locations in one function and read them from another function. An example would be to hash ConfigMaps, store the values in annotations, and then propagate the hashes to uses of the ConfigMaps in order to <a href="https://itnext.io/the-challenge-of-configmap-rollouts-in-kubernetes-1431398c0866">trigger rollouts on changes</a>. ConfigHub has built-in functions that do this (e.g., set-hash), but functions could be written to do this in kpt by using Starlark, Go, or Typescript. For brevity, I’ll omit the function code, but the pipeline would look something like this:</p><pre>  pipeline:<br>    mutators:<br>      # Stage 1: derive the value and park it where stage 2 can read it.<br>      - name: extract-config-hash<br>        image: gcr.io/kpt-fn/starlark:v0.5<br>        configPath: extract-config-hash.yaml<br>      # Stage 2: relational propagation.<br>      - name: stamp-consumers<br>        image: gcr.io/kpt-fn/starlark:v0.5<br>        configPath: stamp-consumers.yaml</pre><p>To propagate values between functions that weren’t written to be paired in this way, one would need to insert <a href="https://catalog.kpt.dev/apply-replacements/v0.1/">ApplyReplacements functions</a> in between them to transfer values from the fields where values were written to fields from which they would be read.</p><p><a href="https://docs.crossplane.io/latest/packages/functions/">Crossplane composition functions</a>, which were at least partly <a href="https://github.com/crossplane/crossplane/issues/2524">inspired by KRM functions</a>, similarly execute in a pipeline, though its <a href="https://docs.crossplane.io/latest/guides/function-patch-and-transform/">patch-and-transform function</a> and <a href="https://blog.crossplane.io/function-kro-yaml-cel/">krofunction</a> are pretty expressive. The patch-and-transform function is more powerful than apply-replacements, such as being able to combine multiple values in formatted string expressions. The kro function supports <a href="https://cel.dev">CEL</a> expressions that refer to resource fields, including runtime status fields, in function arguments.</p><p>Typical <a href="https://github.com/crossplane-contrib/function-kro#usage">AWS example</a> where many resources require the VPC ID:</p><pre>...<br>  pipeline:<br>  - step: kro<br>    functionRef:<br>      name: function-kro<br>    input:<br>      apiVersion: kro.fn.crossplane.io/v1alpha1<br>      kind: ResourceGraph<br>      status:<br>        vpcId: ${vpc.status.atProvider.id}<br>      resources:<br>...<br>      - id: subnet<br>        template:<br>          apiVersion: ec2.aws.upbound.io/v1beta1<br>          kind: Subnet<br>          spec:<br>            forProvider:<br>              region: ${schema.spec.region}<br>              vpcId: ${vpc.status.atProvider.id}<br>              cidrBlock: &quot;10.0.1.0/24&quot;</pre><p>Crossplane compositions aren’t applied to configuration data outside the cluster, so this is not strictly a <em>configuration</em> transformation, but I included them due to the similarity and because the patterns could be adapted to kustomize, kpt, and ConfigHub.</p><p>Like the Crossplane functions, <a href="https://medium.com/itnext/introducing-confighub-b127736641c5">ConfigHub</a>’s <a href="https://docs.confighub.com/guide/dependencies/#transformpaths-links">TransformPaths Links</a> can combine and transform configuration values, using Go templates or CEL expressions. They can also pass values between functions. <a href="https://docs.confighub.com/guide/functions/">ConfigHub functions</a> can extract values as well as transform configuration data, and those two types of functions can be used together, as well as with path specifications. That can be useful in situations requiring complex resource matching or path searching, for instance.</p><p>Here’s a simpler example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VdsfMXpnzvwLVu1Yb385iw.png" /><figcaption>Diagram generated by Claude</figcaption></figure><p>In an AWS environment the account ID and Region should be authoritative in one place — say, an AWSProfile custom resource the platform team maintains per environment — and many resources need values constructed from them. A workload’s container image lives in that account’s ECR registry (&lt;account&gt;.dkr.ecr.&lt;region&gt;.amazonaws.com/&lt;repo&gt;) and its ServiceAccount is bound to an IRSA role (arn:aws:iam::&lt;account&gt;:role/&lt;name&gt;).</p><pre>apiVersion: cloud.example.com/v1<br>kind: AWSProfile<br>metadata:<br>  name: prod<br>  annotations:<br>    config.kubernetes.io/local-config: &quot;true&quot;<br>spec:<br>  accountID: &quot;012345678901&quot;<br>  region: us-east-1</pre><p>A downstream orders unit contains the Deployment and its ServiceAccount. The link reads the account ID and Region and writes three derived values: a Region label (a per-path write), the ECR repository URI (preserving the existing tag), and the IRSA role-ARN annotation:</p><pre>UpstreamPaths:<br>  - Name: accountID<br>    Path: spec.accountID<br>    Resource:<br>      ResourceName: /prod<br>      ResourceType: cloud.example.com/v1/AWSProfile<br>  - Name: region<br>    Path: spec.region<br>    Resource:<br>      ResourceName: /prod<br>      ResourceType: cloud.example.com/v1/AWSProfile<br>DownstreamPaths:<br>  - Path: metadata.labels.aws-region<br>    Resource:<br>      ResourceName: default/orders<br>      ResourceType: apps/v1/Deployment<br>    Expression: &quot;{{.Params.region}}&quot;<br>    Evaluator: template<br>    Parameters: [region]<br>    DataType: string<br>DownstreamSetters:<br>  # Point the container at this environment&#39;s ECR registry, keeping its tag.<br>  - Parameters: [accountID, region]<br>    FunctionInvocation:<br>      FunctionName: set-container-repository-uri<br>      WhereResource: &quot;ConfigHub.ResourceType = &#39;apps/v1/Deployment&#39;&quot;<br>      Arguments:<br>        - Value: orders<br>        - Value: &quot;{{.Params.accountID}}.dkr.ecr.{{.Params.region}}.amazonaws.com/orders&quot;<br>          Evaluator: template<br>  # Bind the ServiceAccount to its IRSA role, illustrating the set-yq setter.<br>  - Parameters: [accountID]<br>    FunctionInvocation:<br>      FunctionName: set-yq<br>      WhereResource: &quot;ConfigHub.ResourceType = &#39;v1/ServiceAccount&#39;&quot;<br>      Arguments:<br>        - Value: &#39;.metadata.annotations[&quot;eks.amazonaws.com/role-arn&quot;] = $params.arn&#39;<br>        - Value: &quot;arn=arn:aws:iam::{{.Params.accountID}}:role/orders&quot;<br>          Evaluator: template</pre><p>More generally, <a href="https://docs.confighub.com/background/entities/link/">Links</a> in ConfigHub can propagate and transform configuration data in whole or in part from one <a href="https://docs.confighub.com/background/entities/unit/">Unit</a> containing Kubernetes resources or <a href="https://docs.confighub.com/guide/app-config/">application configuration</a> (INI, TOML, JSON, Env, etc.) to another. For instance, we use Upsert links with the render-configmap transformation function to generate ConfigMaps for application configuration, and Insert links to insert AWS IAM policy JSON into <a href="https://aws-controllers-k8s.github.io/docs/">ACK resources</a>.</p><p><a href="https://docs.confighub.com/background/entities/trigger/">Triggers</a> in ConfigHub are another mechanism that executes functions on configuration. Triggers are similar to <a href="https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/">Kubernetes dynamic admission control</a>. They can execute functions after every configuration change. Mutating functions are executed before validating functions and other readonly functions. Mutating function Triggers can maintain invariants. Validating function Triggers can <a href="https://itnext.io/validating-scoring-and-gating-kubernetes-configuration-after-every-change-a11fbab14110">verify correctness and completeness</a> of the configuration, and can <a href="https://itnext.io/vetting-kubernetes-configuration-with-kyverno-prior-to-deployment-131ee33d315b">enforce policies</a>. <a href="https://docs.confighub.com/guide/filters-and-views/#record-configuration-values-with-triggers">Readonly function Triggers can extract values</a> for filters and views and ensure they are kept up to date. ConfigHub ensures that mutating functions are executed first and that functions are executed by the <a href="https://docs.confighub.com/guide/custom-workers/">correct worker</a>.</p><p>Hopefully that gives you a sense of the capabilities and limitations of current Kubernetes configuration transformation orchestrators. Anything that could be done in a <a href="https://itnext.io/generating-configuration-using-general-purpose-programming-languages-19230a2c2573">general-purpose language</a>, such as using <a href="https://cdk8s.io/">cdk8s</a>, or <a href="https://itnext.io/can-configuration-languages-dsls-solve-configuration-complexity-eee8f124e13a">configuration DSL</a>, such as <a href="https://jsonnet.org/">jsonnet</a>, can be done using transformation functions, but they are composed through data instead of through code, and can interoperate with other tools that read and write the configuration data. Also, in the case of kpt, porch, and ConfigHub, because the data is stored persistently, it does not need to be regenerated from scratch for every change. That enables transformations to be decoupled across time as well.</p><p>We’ll share more examples as we find good ones that better illustrate how the approach changes the experience of maintaining Kubernetes configuration.</p><p>Have you written a kustomize transformer plugin, KRM function, or Crossplane composition function? What do you like or not like about writing transformations? Have you used AI agents to write transformations? What do you like or not like about using them? Have you tried to use yq as a transformation function? Do you use transformation functions so that you can keep the configuration fully rendered / <a href="https://itnext.io/kubernetes-configuration-and-infrastructure-as-code-taxonomy-62ab4dd7e82f">WET</a> in order to enable some other capability?</p><p>Reply here, or send me a message on <a href="https://www.linkedin.com/in/bgrant0607/">LinkedIn</a>, <a href="https://x.com/bgrant0607">X/Twitter</a>, or <a href="https://bsky.app/profile/bgrant0607.bsky.social">Bluesky</a>, where I plan to crosspost this.</p><p>You could also <a href="https://auth.confighub.com/sign-up">try out ConfigHub</a>, which is now in preview.</p><p>If you found this interesting, you may be interested in other posts in my <a href="https://medium.com/@bgrant0607/list/kubernetes-8b0b8930195b">Kubernetes series</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b998809eda17" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/kubernetes-configuration-transformation-orchestrators-b998809eda17">Kubernetes configuration transformation orchestrators</a> was originally published in <a href="https://itnext.io">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Rolling out new Kubernetes policies with Initiatives]]></title>
            <link>https://itnext.io/rolling-out-new-kubernetes-policies-with-initiatives-74ee4c2d463d?source=rss-ccdbe9a99319------2</link>
            <guid isPermaLink="false">https://medium.com/p/74ee4c2d463d</guid>
            <category><![CDATA[config-as-data]]></category>
            <category><![CDATA[gitops]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[policy-as-code]]></category>
            <dc:creator><![CDATA[Brian Grant]]></dc:creator>
            <pubDate>Mon, 03 Aug 2026 16:15:49 GMT</pubDate>
            <atom:updated>2026-08-03T17:50:45.524Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dmJCHfpKSP1Voa9wQgsmjQ.png" /></figure><p>I <a href="https://itnext.io/validating-scoring-and-gating-kubernetes-configuration-after-every-change-a11fbab14110">previously wrote about</a> validating Kubernetes configuration and <a href="https://docs.confighub.com/guide/validation-and-policies/">enforcing policies</a> after every change, prior to deployment.</p><p>Introducing a new validation check or policy when you have existing configuration is likely to find violations. To help roll out a new validation or policy check in a non-disruptive manner, we created an Initiatives workflow. It initially focuses on policies supported by <a href="https://itnext.io/vetting-kubernetes-configuration-with-kyverno-prior-to-deployment-131ee33d315b">Kyverno</a>, including <a href="https://kubernetes.io/docs/reference/access-authn-authz/validating-admission-policy/">Kubernetes </a><a href="https://kubernetes.io/docs/reference/access-authn-authz/validating-admission-policy/">ValidatingAdmissionPolicy</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0mwC7gPvRLmmheuFYwGaeA.png" /></figure><p>The workflow supports creating triggers initially in Warn mode, which will evaluate the validating functions and attach ApplyWarnings for failed checks, but not block apply/release of the configuration data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*brqq3t7dq02z_S0kgcdAxg.png" /></figure><p>It then reports configuration units that fail (and pass) the new check.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pcbTwfFpP6eQ9IUQdYeAYQ.png" /></figure><p>The violations can be fixed using <a href="https://docs.confighub.com/guide/functions/">ConfigHub functions</a>, or by direct edits. Claude Code appears to like using yq.</p><p>Once all of the violations are fixed, the trigger can be changed to enforced mode, and the initiative can be marked completed, as in a kanban board.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y10jhesgkMjLhMdApymjmA.png" /></figure><p>It’s simple, but provides some structure around the process. We also created a <a href="https://www.youtube.com/watch?v=9ERXS7togRI">video to walk you through it</a> for the first time.</p><p>Kubernetes policy tools, including Kyverno, typically have ways to both <a href="https://kyverno.io/docs/guides/reports/">report violations</a> and to convert them to enforced checks, but those workflows don’t necessarily work when running checks in CI, prior to deployment, because that’s partly specific to how each CI tool works. <a href="https://kyverno.io/docs/kyverno-cli/reference/kyverno_apply/">Kyverno’s CLI</a> also handles that differently than in-cluster <a href="https://kyverno.io/docs/introduction/how-kyverno-works/">admission control and report controllers</a>. Such mechanisms also are different for every tool, and different tools cover different checks. Initiatives provides a workflow to do this through ConfigHub that can be extended to work across any validation and policy tools integrated using validating functions.</p><p>In the future I also plan to explore integrating some of the <a href="https://itnext.io/configuration-as-code-is-a-liability-for-security-d0e53727cd17">security</a> and <a href="https://itnext.io/building-tools-to-manage-cross-cutting-concerns-in-kubernetes-by-using-confighub-6bd64afcdd69">other management tools</a> I built with the initiatives mechanism to make it easier to coordinate fixing the findings of those tools across multiple teams.</p><p>Do you have a way to orchestrate the introduction of new policy checks in CI? Does it work across multiple policy tools? Does it work across all repos containing Kubernetes configuration in your organization? Does it work across all Kubernetes configuration tools in use? How do you fix issues one environment at a time? How do you handle policy exceptions by environment or by workload?</p><p>Reply here, or send me a message on <a href="https://www.linkedin.com/in/bgrant0607/">LinkedIn</a>, <a href="https://x.com/bgrant0607">X/Twitter</a>, or <a href="https://bsky.app/profile/bgrant0607.bsky.social">Bluesky</a>, where I plan to crosspost this.</p><p>You could also <a href="https://auth.confighub.com/sign-up">try out ConfigHub</a>, which is now in preview.</p><p>If you found this interesting, you may be interested in other posts in my <a href="https://medium.com/@bgrant0607/list/kubernetes-8b0b8930195b">Kubernetes series</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=74ee4c2d463d" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/rolling-out-new-kubernetes-policies-with-initiatives-74ee4c2d463d">Rolling out new Kubernetes policies with Initiatives</a> was originally published in <a href="https://itnext.io">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Validating, scoring, and gating Kubernetes configuration after every change]]></title>
            <link>https://itnext.io/validating-scoring-and-gating-kubernetes-configuration-after-every-change-a11fbab14110?source=rss-ccdbe9a99319------2</link>
            <guid isPermaLink="false">https://medium.com/p/a11fbab14110</guid>
            <category><![CDATA[config-as-code]]></category>
            <category><![CDATA[config-as-data]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[gitops]]></category>
            <dc:creator><![CDATA[Brian Grant]]></dc:creator>
            <pubDate>Wed, 29 Jul 2026 14:45:24 GMT</pubDate>
            <atom:updated>2026-07-29T16:12:30.371Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TFvgr8ua-dqtzbGBx4N0vA.png" /><figcaption>An example of validating Triggers using Kyverno in ConfigHub</figcaption></figure><p>I previously wrote about <a href="https://itnext.io/vetting-kubernetes-configuration-with-kyverno-prior-to-deployment-131ee33d315b">integrating Kyverno with ConfigHub</a> in order to evaluate Kyverno policies after every configuration change and block deployment in the case of violations.</p><p>The purpose was to find policy violations and configuration errors earlier, especially prior to deployment. As I mentioned, it is possible to do this by rendering configuration and running validation checks in CI, but that requires many more steps (e.g., the git commit and push dance, rendering), can be slow (minutes), and can be hard to manage centrally.</p><p>I encountered another issue with the CI approach: AI agents. <a href="https://itnext.io/using-ai-to-generate-kubernetes-configuration-5ba954acd97a">AI agents are getting better and better at generating</a> and changing Kubernetes YAML, but they still make mistakes and they also aren’t aware of organizational policies and conventions. Even with AI agent skills, I have found agents to not 100% reliably follow instructions from documents, so that is not an adequate method of conveying policies. That said, skills do have their place, and I <a href="https://github.com/confighub/confighub-skills">created some agent skills for ConfigHub</a>.</p><p>AI agents don’t know what they don’t know and work by trial and error to some degree. They can iterate (“hill climb”) on their own when their work is verified, can iterate faster when the verification is faster, and consume fewer tokens when the output is concise and in a consistent format, as opposed to typical logs and human-oriented tool output.</p><p>So, I integrated more <a href="https://itnext.io/kubernetes-configuration-linting-tools-699ddeedaeec">validation, scoring, and policy tools</a> with ConfigHub.</p><p>So far I implemented support for:</p><ul><li>Kyverno (<a href="https://github.com/confighub/examples/tree/main/custom-workers/kyverno">CLI</a>, <a href="https://github.com/confighub/examples/tree/main/custom-workers/kyverno-server">admission webhook</a>): <a href="https://cel.dev/">CEL</a>-based policies</li><li><a href="https://github.com/confighub/examples/tree/main/custom-workers/opa-gatekeeper">Open Policy Agent Gatekeeper</a>: <a href="https://www.openpolicyagent.org/docs/policy-language">Rego</a>-based policies</li><li><a href="https://github.com/confighub/examples/tree/main/custom-workers/kubescape">Kubescape</a>: also Rego-based, with a <a href="https://github.com/kubescape/regolibrary">large policy library</a></li><li><a href="https://github.com/confighub/examples/tree/main/custom-workers/kube-linter">kube-linter</a>: vets production readiness</li><li><a href="https://github.com/confighub/examples/tree/main/custom-workers/kube-score">kubescore</a>: also checks production readiness and network policy</li><li><a href="https://github.com/confighub/examples/tree/main/custom-workers/kubepug">kubepug</a>: checks for use of deprecated APIs</li><li><a href="https://github.com/confighub/sdk/blob/main/function-impl/kubernetes/standard_functions.go#L537">kubeconform</a>: validates resources against schemas</li></ul><p>These integrations convert the various tool output to a consistent JSON data structure containing messages from the tools, resource types and names, failing configuration paths where reported, and scores where provided. Each example includes a demo script so you can see them in action.</p><p>When used in <a href="https://docs.confighub.com/background/entities/trigger/">triggers</a>, failures automatically attach corresponding Apply Gates to the configurations. It’s also possible to set the trigger to just warn instead of blocking deployment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yRkOsRUTPXrLBGCBHTmBLg.png" /></figure><p>With details regarding the failure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*__zoBPEO5xfveWC68JRQIA.png" /></figure><p>The apply gates and warnings are reported by the API and can be accessed by the CLI as well, for use by AI agents. The triggers can be invoked on demand, as well, to return the full responses.</p><p>Other than kubeconform, which is integrated as a library, some of these tools are CLIs and some are <a href="https://docs.confighub.com/guide/admission-webhook-functions/">admission webhooks</a>. Kyverno supported both, so I integrated both. The CLIs are pretty slow — up to multiple seconds, even without spinning up a container. So a server is a more viable way to provide interactive performance, though it does cost something to run, unless it can scale to zero, which would also incur some cold-start overhead.</p><p>I was asked a good question on the Kyverno post about how integrating with admission webhooks was different from kubectl apply — dry-run=server — validate=true. A few differences:</p><ul><li>Cluster credentials aren’t required</li><li>It provides more control over what checks are performed, and eliminates other potential causes of failure, which I wanted to report separately</li><li>Failure details are easier to extract</li></ul><p>It’s also possible to <a href="https://docs.confighub.com/guide/validation-and-policies/">implement your own policies and checks in CEL, Starlark, and Go</a>. The <a href="https://itnext.io/there-should-be-an-app-for-that-1f4d23286a9e">RBAC manager</a> uses <a href="https://github.com/confighub/examples/blob/main/rbac-manager/setup.sh#L68">CEL</a>. There are also some additional built-in validators, such as to check whether there are duplicate merge keys (e.g., environment variables) and whether immutable fields are changed.</p><p>Once I created a validation function that implemented a particular pattern, I found AI pretty good at applying the pattern to a different tool or check.</p><p>I just started to iterate on the agent skills and am interested in your feedback on what you’d like to use them to do with ConfigHub. I used the triggers-and-applygates skill on the demo data shown above in Claude Code.</p><pre>Here&#39;s the full diagnosis of initiatives-demo — 19 Units.<br><br>  Policy in force — 5 vet-kyverno Triggers (worker kyverno-cli-worker)<br><br>  ┌─────────────────────────────────────┬───────────────┬──────────────────────┐<br>  │               Trigger               │     Tier      │ Currently firing on  │<br>  ├─────────────────────────────────────┼───────────────┼──────────────────────┤<br>  │ image-registry-restriction-check    │ Gate (blocks) │ 8 Units              │<br>  ├─────────────────────────────────────┼───────────────┼──────────────────────┤<br>  │ disallow-host-ports-check           │ Gate (blocks) │ 0 Units              │<br>  ├─────────────────────────────────────┼───────────────┼──────────────────────┤<br>  │ run-as-nonroot-enforcement-check    │ Warn          │ ~all 17 warned Units │<br>  ├─────────────────────────────────────┼───────────────┼──────────────────────┤<br>  │ liveness-and-readiness-probes-check │ Warn          │ 14 Units             │<br>  ├─────────────────────────────────────┼───────────────┼──────────────────────┤<br>  │ resource-limits-enforcement-check   │ Warn          │ 2 Units              │<br>  └─────────────────────────────────────┴───────────────┴──────────────────────┘<br><br>  Tier 1 — Blocking gates (8 Units, apply refused)<br><br>  All 8 are blocked by the same gate, image-registry-restriction-check (requires images to start with ghcr.io/acme/):<br><br>  aichat-postgres, aichat-redis, docs-search, eshop-postgres,<br>  eshop-redis, kyverno-cli-worker-unit, portal-postgres, website-postgres<br><br>  These are off-the-shelf images (postgres, redis, the kyverno CLI, a search component) from public registries — exactly the &quot;non-compliant registry&quot; case the policy<br>  exists to stop. disallow-host-ports-check is armed but nothing violates it, so it produces zero gates.<br><br>  Tier 2 — Advisory warnings (17 Units, still appliable)<br><br>  Warnings don&#39;t block; they&#39;re the &quot;tech-debt&quot; tier. Distribution:<br><br>  - run-as-nonroot-enforcement-check — nearly universal (missing runAsNonRoot security context across the fleet).<br>  - liveness-and-readiness-probes-check — 14 Units lack probes. The three without this warning — eshop-api, portal-api, portal-frontend — already have probes.<br>  - resource-limits-enforcement-check — only 2: kyverno-cli-worker-unit and website-cms (the only multi-warning Unit, carrying all three).<br><br>  Tier 3 — Fully clean (2 Units, no gates, no warnings)<br><br>  aichat-api, aichat-frontend<br><br>  Want me to take any of this further — e.g. fix one gated Unit end-to-end to demo the release loop, or pull the actual kyverno failure messages for a specific Unit?</pre><p>I asked the agent to look at the failures of one unit, and it proposed fixes.</p><pre>Here are the full failure messages for docs-search (a apps/v1/StatefulSet, namespace currently the placeholder confighubplaceholder). It has 3 failing checks — 1<br>  blocking gate + 2 advisory warnings:<br><br>  🔴 Gate (blocks apply) — image-registry-restriction-check<br><br>  ▎ policy restrict-image-registries rule restrict-image-registries failed: Images must come from ghcr.io/acme/.<br><br>  The container image isn&#39;t in the ghcr.io/acme/ registry. This is the one that&#39;s actually refusing the apply.<br><br>  🟡 Warning (advisory) — liveness-and-readiness-probes-check<br><br>  ▎ policy require-probes rule validation failed: All containers must define a livenessProbe.<br><br>  🟡 Warning (advisory) — run-as-nonroot-enforcement-check<br><br>  ▎ policy require-run-as-nonroot rule validation failed: Pods or all containers must set runAsNonRoot to true.<br>  <br>  Notes<br><br>  - The resource name shows as confighubplaceholder/docs-search — the namespace is still a placeholder, so once you sort the registry you may also want ensure-namespaces /<br>  a real namespace before apply (a vet-placeholders gate would catch that too, if one were wired).<br>  - Fix loop to demo: set-container-image → ghcr.io/acme/docs-search:&lt;tag&gt; clears the gate; adding a livenessProbe and runAsNonRoot: true (e.g.<br>  set-pod-container-security-context-defaults, set-container-probe-defaults) clears the two warnings.</pre><p>With changes made by functions, there’s no risk of typos, indentation issues, incorrect field names, etc., and functions can be applied to multiple resources in a single API call. Of course, the agent could also make changes directly. Claude Code likes to use yq, which it can invoke via set-yq without pulling all of the configuration into its context.</p><p>The configuration data in ConfigHub is shared state, so any changes can be observed immediately by other team members and agents, but they aren’t automatically released to the cluster. The agent can iterate until all checks pass.</p><p>Validating triggers provide the kind of guardrails that AI agents need in order to verify their work and guide their act-and-fix loops. I found them to be effective in forcing the AI agent to address gaps and inappropriate choices in configuration it generated with only high-level instructions. And much more quickly than waiting for CI.</p><p>Do you validate your Kubernetes configuration prior to deployment? What validation and/or policy tools do you use? Do you render and validate it only in CI? How long does it typically take to get a result? Does your AI agent understand how to find the results? Or do you have a way to render and run all the checks locally? How are the tools and checks packaged to ensure you’re running the correct, up-to-date set of checks?</p><p>Reply here, or send me a message on <a href="https://www.linkedin.com/in/bgrant0607/">LinkedIn</a>, <a href="https://x.com/bgrant0607">X/Twitter</a>, or <a href="https://bsky.app/profile/bgrant0607.bsky.social">Bluesky</a>, where I plan to crosspost this.</p><p>You could also <a href="https://auth.confighub.com/sign-up">try out ConfigHub</a>, which is now in preview.</p><p>If you found this interesting, you may be interested in other posts in my <a href="https://medium.com/@bgrant0607/list/kubernetes-8b0b8930195b">Kubernetes series</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a11fbab14110" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/validating-scoring-and-gating-kubernetes-configuration-after-every-change-a11fbab14110">Validating, scoring, and gating Kubernetes configuration after every change</a> was originally published in <a href="https://itnext.io">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using AI to generate Kubernetes configuration]]></title>
            <link>https://itnext.io/using-ai-to-generate-kubernetes-configuration-5ba954acd97a?source=rss-ccdbe9a99319------2</link>
            <guid isPermaLink="false">https://medium.com/p/5ba954acd97a</guid>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Brian Grant]]></dc:creator>
            <pubDate>Mon, 27 Jul 2026 15:27:33 GMT</pubDate>
            <atom:updated>2026-07-27T16:04:19.784Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*idKdB8QMvJU-cu8sIBEOZw.png" /><figcaption>I asked Gemini to create an image of a robot writing a Kubernetes Deployment. I love the Kubernetes swag it spontaneously added to the image, like the Kubernetes lamp.</figcaption></figure><p>I’ve been experimenting with various approaches to generate Kubernetes configuration using AI. Is it better than writing the configuration by hand? Better than kubectl create? Better than traditional UI forms? Better than <a href="https://itnext.io/infrastructure-as-code-cargo-culting-patterns-b714f680db4b">starting from an existing configuration</a> and changing it?</p><p>On the last question, it definitely helps tremendously to have some content to start from, whether Kubernetes resources, schemas, or other information about the workload. A Helm chart was <a href="https://itnext.io/converting-helm-charts-to-agent-skills-2a92829d418c">my starting point for vLLM</a>, and that required very little prompting. But most of the work was already done when creating the Helm chart, so that’s cheating, really. I’ll discuss other sources of information in a bit.</p><p>In my first attempt, I just asked Claude Code to generate a Kubernetes Service and Deployment for my application. It created a minimalistic pair of resources with dummy values that needed to be edited. That was not better than kubectl create.</p><p>I created <a href="https://github.com/confighub/confighub-skills/blob/main/skills/kubernetes-resources/SKILL.md">an agent skill for creating resources</a>. It generated <a href="https://github.com/confighub/confighub-skills/blob/main/references/yaml-patterns.md">some example resources</a> to start from, but they were mostly similarly fairly minimalistic. For example:</p><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: my-app<br>  namespace: confighubplaceholder<br>  labels:<br>    app: my-app<br>spec:<br>  replicas: 2<br>  selector:<br>    matchLabels:<br>      app: my-app<br>  template:<br>    metadata:<br>      labels:<br>        app: my-app<br>    spec:<br>      containers:<br>        - name: main<br>          image: ghcr.io/myorg/my-app:v1.2.3<br>          ports:<br>            - containerPort: 8080</pre><p>The main benefits of the skill were that I added instructions for how to use ConfigHub functions to set additional default values, as shown below, and how to use ConfigHub functions, such as vet-schemas, to validate the resources.</p><pre>cub function set --space myapp-dev --where &quot;Slug = &#39;my-app&#39;&quot; \<br>  set-container-resources-defaults --change-desc &quot;Fill resource defaults&quot;<br>cub function set --space myapp-dev --where &quot;Slug = &#39;my-app&#39;&quot; \<br>  set-container-probe-defaults --change-desc &quot;Fill probe defaults&quot;<br>cub function set --space myapp-dev --where &quot;Slug = &#39;my-app&#39;&quot; \<br>  set-pod-container-security-context-defaults --change-desc &quot;Fill security context defaults&quot;</pre><p>I concluded that the example resources themselves weren’t of much value unless a user were able to curate their own customized versions of them beyond what Claude would generate on its own, so I created <a href="https://github.com/confighub/installer/tree/main/packages/kubernetes-resources/bases/default">an installer package to upload the examples</a> into ConfigHub for customization. That still left the user to edit the resources to fill in the many details after copying the examples, however.</p><p>I then started over from scratch using Claude Code’s AskUserQuestion tool, and it was like a game of 20 questions. The first couple of questions were fairly high-level, but already required significant Kubernetes knowledge:</p><pre>What kind of workload is this application?<br><br>❯ 1. Stateless web/API ✔<br>     A standard Deployment — replicas are interchangeable, no per-pod persistent identity or storage. Most web servers and APIs fit here.<br>  2. Stateful service<br>     A StatefulSet — stable network identity and per-pod persistent volumes. Databases, queues, and clustered systems fit here.<br>  3. Batch / scheduled job<br>     A Job (run-to-completion) or CronJob (scheduled). For data migrations, ETL, periodic tasks.<br>  4. Node-level daemon<br>     A DaemonSet — one pod per node. For log collectors, monitoring agents, CNI components.<br>  5. Type something.<br>  6. Chat about this<br><br>How should the application be exposed?<br><br>❯ 1. Internal only (ClusterIP)<br>     Reachable only inside the cluster via a Service. Good for backend services other pods call.<br>  2. External via Ingress<br>     ClusterIP Service plus an Ingress with a hostname/path for HTTP(S) traffic from outside.<br>  3. External via LoadBalancer<br>     A cloud LoadBalancer Service that gets a public IP. Good for non-HTTP or direct exposure.<br>  4. No network exposure<br>     No Service at all — for workers/jobs that only make outbound calls.<br>  5. It&#39;s external. We use Traefik IngressRoute for ingress. ✔<br>  6. Chat about this<br><br>How is the app configured and what does it need for secrets/config?<br><br>❯ 1. [ ] Environment variables<br>  Plain env vars set directly in the pod spec.<br>  2. [ ] ConfigMap<br>  Non-sensitive config (files or env) mounted/injected from a ConfigMap.<br>  3. [ ] Secret<br>  Sensitive values (API keys, passwords) from a Secret.<br>  4. [ ] None<br>  No external configuration needed.<br>  5. [ ] Type something: We want to use a ConfigMap for environment variables and ExternalSecretOperator to pull secrets from AWS SSM<br>  6. Chat about this</pre><p>From there, it dove into details. In the interest of space, I won’t present all of the options it offered or my responses.</p><pre>What resource and scaling profile do you want?<br>What&#39;s the app name, namespace, and container image (with tag)?<br>What port does the container listen on, and what hostname should the IngressRoute match?<br>For the ExternalSecret: which SecretStore and AWS SSM paths?<br>How should the ConfigMap and Secret reach the container?<br>I also need the actual values. These are free-form, so just type them in your reply.</pre><p>At that point, I was copy/pasting lines from a local env file.</p><p>It then asked about probe endpoints and certificate details. I also requested the addition of an emptyDir volume, and it took multiple iterations to get that exactly the way I needed. One advantage of a tool is that it can explicitly require or guide you to provide the necessary details. I stopped before getting to the command line, security best practices, RBAC, and <a href="https://itnext.io/variant-generation-is-the-primary-capability-of-infrastructure-as-code-773e6b404b49">variants for multiple environments</a>.</p><p>There are a lot of details in Kubernetes configuration, but, other than security-related properties, which could have different defaults if not for backward compatibility and usability concerns, most settings depend on details of the application and underlying infrastructure. That did not seem much better than a guided set of UI forms, except that it was able to handle some non-standard choices I requested, such as the use of <a href="https://external-secrets.io/latest/">External Secrets</a>. It also allowed me to change my mind without backtracking, though it didn’t call me out on that either, so I’m not sure that was a good thing, since it could have indicated a mistake or led to inconsistencies.</p><p>In order to address the issue of providing details about the application, I provided <a href="https://itnext.io/application-interfaces-revisited-10-years-later-0ba1bfc43835">specifications of an application’s interfaces</a>: commands and flags, environment variables, ports, probe endpoints, and filesystem paths written, with the agent skill installed. I also asked it to use security best practices. Because I was generating configuration for a ConfigHub component, it inferred other details from the content of other skills. Other kinds of technical documents describing the component could have played a similar role. Claude tried to validate the resources using dry run applies with a cluster, but I hadn’t given it up-to-date credentials for any.</p><p>Claude correctly added pod-security labels to the Namespace, created a ConfigMap with valid environment variables, wired the Deployment to the ConfigMap and to a Secret, set the security context, added liveness and readiness probes pointed at the correct endpoints, mounted an emptyDir volume for /tmp, set minimal resource requests, set the port consistently in multiple fields and resources including a ConfigMap value, and created a NetworkPolicy that allowed egress to DNS and to ConfigHub’s server and ingress allowing monitoring of metrics. The result was a much more complete starting point with no back and forth required. An excerpt from the Deployment’s pod spec template follows:</p><pre>      # Pod-level hardening (satisfies the &quot;restricted&quot; Pod Security Standard).<br>      securityContext:<br>        runAsNonRoot: true<br>        runAsUser: 65532<br>        runAsGroup: 65532<br>        fsGroup: 65532<br>        seccompProfile:<br>          type: RuntimeDefault<br>      # Give the worker time to drain in-flight work after SIGTERM<br>      # (GRACE_PERIOD_DELAY + shutdown timeout, with headroom).<br>      terminationGracePeriodSeconds: 30<br>      containers:<br>        - name: worker<br>          image: ghcr.io/confighubai/confighub-worker:v0.1.90<br>          imagePullPolicy: IfNotPresent<br>          ports:<br>            - name: http<br>              containerPort: 9090<br>              protocol: TCP<br>          envFrom:<br>            - configMapRef:<br>                name: confighub-worker<br>          env:<br>            - name: CONFIGHUB_WORKER_ID<br>              valueFrom:<br>                secretKeyRef:<br>                  name: confighub-worker<br>                  key: CONFIGHUB_WORKER_ID<br>            - name: CONFIGHUB_WORKER_SECRET<br>              valueFrom:<br>                secretKeyRef:<br>                  name: confighub-worker<br>                  key: CONFIGHUB_WORKER_SECRET<br>          livenessProbe:<br>            httpGet:<br>              path: /internal/ok<br>              port: http<br>            initialDelaySeconds: 10<br>            periodSeconds: 15<br>            timeoutSeconds: 3<br>            failureThreshold: 3<br>          readinessProbe:<br>            httpGet:<br>              path: /internal/ready<br>              port: http<br>            initialDelaySeconds: 5<br>            periodSeconds: 10<br>            timeoutSeconds: 3<br>            failureThreshold: 3<br>          resources:<br>            requests:<br>              cpu: 50m<br>              memory: 64Mi<br>            limits:<br>              # Intentionally no CPU limit (avoids throttling); memory limit caps blast radius.<br>              memory: 256Mi<br>          # Container-level hardening.<br>          securityContext:<br>            allowPrivilegeEscalation: false<br>            privileged: false<br>            readOnlyRootFilesystem: true<br>            runAsNonRoot: true<br>            capabilities:<br>              drop:<br>                - ALL<br>            seccompProfile:<br>              type: RuntimeDefault<br>          volumeMounts:<br>            # Root FS is read-only; /tmp is the worker&#39;s required writable scratch space.<br>            - name: tmp<br>              mountPath: /tmp<br>      volumes:<br>        - name: tmp<br>          emptyDir: {}</pre><p>Additionally providing the Dockerfile would have enabled the agent to choose the expected user and group IDs. They could also be added to the <a href="https://github.com/confighub/sdk/blob/main/core/function/api/runtime_spec.go">RuntimeSpec</a>. If the agent had used the set-pod-container-security-context-defaults function, as the skill recommended, it also would have set the correct values, since I adopted the same values in our Dockerfile as reasonable defaults. Agents still don’t always do as they are told.</p><p>From there, I could use <a href="https://itnext.io/building-tools-to-manage-cross-cutting-concerns-in-kubernetes-by-using-confighub-6bd64afcdd69">the policy and operational tools and corresponding agent skills I created</a> to configure the RBAC, autoscaling, PodDisruptionBudget, scheduling constraints, and so on. For opinionated operational environments it should be possible to create skills that convey more of the necessary decisions up front.</p><p>For this particular component, I used AI to create <a href="https://github.com/confighub/installer/tree/main/packages/worker">an installer package</a>. The package captured those decisions and included the application specs for context and for validation. It also added a <a href="https://itnext.io/a-new-kubernetes-package-installer-324db6f615ac">“fact collector”</a> to discover some necessary variable values, including the current image release, the ability to change the namespace, and deterministic validation steps, further automating customization of the configuration. (Such packages are really only necessary for components that will be installed by multiple teams or organizations for multiple use cases. ConfigHub has other ways to <a href="https://itnext.io/how-are-variants-managed-in-confighub-061bc2f89cff">manage variants of a configuration</a>.)</p><p>I could imagine going a step farther, by <a href="https://itnext.io/there-should-be-an-app-for-that-1f4d23286a9e">generating a management tool</a> just for a specific component, though this particular workload is simple enough that general operational tools and schema-driven tools would be sufficient. However, for an application with many more configuration options, I could imagine some guidance, progressive disclosure, consistency checking, and other early validation would be helpful. Using <a href="https://docs.confighub.com/background/concepts/filters/">ConfigHub’s filters</a>, it could find all instances of configurations for the same component, and provide <a href="https://itnext.io/abstraction-is-the-wrong-way-to-simplify-configuration-81ac4bad02ab">an opinionated, selected view</a> on top of them. Let me know if you are interested in an experiment along those lines.</p><p>I haven’t seen much discussion of this topic in any forums or blog posts yet, and I found just a handful of agent skills with simple manifest examples, so it’s still early in the development of best practices in this area. I found that a combination of information extracted from application source code or generated from the application, architectural documents, infrastructure context, deterministic AI-generated tools, and agent skills was most effective so far.</p><p>Have you used AI agents to generate Kubernetes configuration for any workloads? What approaches did you try? Which ones worked best? What information did you provide to the agent? What information did it, perhaps unexpectedly, discover on its own? How many iterations were necessary? Do you think the process was faster or resulted in fewer errors than non-AI-based approaches?</p><p>Reply here, or send me a message on <a href="https://www.linkedin.com/in/bgrant0607/">LinkedIn</a>, <a href="https://x.com/bgrant0607">X/Twitter</a>, or <a href="https://bsky.app/profile/bgrant0607.bsky.social">Bluesky</a>, where I plan to crosspost this.</p><p>You could also <a href="https://auth.confighub.com/sign-up">try out ConfigHub</a>, which is now in preview.</p><p>If you found this interesting, you may be interested in other posts in my <a href="https://medium.com/@bgrant0607/list/kubernetes-8b0b8930195b">Kubernetes series</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5ba954acd97a" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/using-ai-to-generate-kubernetes-configuration-5ba954acd97a">Using AI to generate Kubernetes configuration</a> was originally published in <a href="https://itnext.io">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Application interfaces revisited 10 years later]]></title>
            <link>https://itnext.io/application-interfaces-revisited-10-years-later-0ba1bfc43835?source=rss-ccdbe9a99319------2</link>
            <guid isPermaLink="false">https://medium.com/p/0ba1bfc43835</guid>
            <category><![CDATA[config-as-data]]></category>
            <category><![CDATA[config-as-code]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Brian Grant]]></dc:creator>
            <pubDate>Tue, 21 Jul 2026 18:42:48 GMT</pubDate>
            <atom:updated>2026-07-21T19:56:49.091Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0DhN7WXuTGOAmLQhaC6SPA.png" /><figcaption>Image generated by Gemini</figcaption></figure><p>Ten years ago, I co-authored a <a href="https://kubernetes.io/blog/2016/09/cloud-native-application-interfaces/">blog post about application interfaces</a>. The lack of well defined, discoverable application interfaces impedes automation and management of applications, and obstructs configuration validation and generation in particular.</p><p>Since then, for user-facing interfaces, specifications like <a href="https://swagger.io/specification/">Swagger/OpenAPI</a> have matured and evolved. Now there are agent-facing interfaces that support interface discovery like <a href="https://modelcontextprotocol.io/specification/2025-11-25">MCP</a>, as well.</p><p>Unfortunately, for the most part, the management-oriented surfaces that the post was about have not received the same amount of investment and standardization.</p><p>One exception to that is <a href="https://opentelemetry.io/">OpenTelemetry</a>, which I <a href="https://github.com/cncf/toc/pull/233#issuecomment-490152831">helped bring into the CNCF</a>. OpenTelemetry is now <a href="https://opentelemetry.io/ecosystem/vendors/">widely supported</a> by observability tools.</p><p>Partly <a href="https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/">thanks to Kubernetes</a>, service discovery is largely through DNS still.</p><p><a href="https://itnext.io/benefits-and-challenges-of-infrastructure-from-code-d59c6e004642">Infrastructure from Code</a> tools automate infrastructure provisioning, but I’m not aware of any interface specifications coming out of that space.</p><p>Developer-oriented specs like <a href="https://score.dev/">score.dev</a> and Docker Compose describe how to run an application in a specific configuration, not the interfaces it exposes.</p><p><a href="https://itnext.io/application-configuration-and-feature-flags-356f38712120">Application configuration</a> continues to be very fragmented, with many formats: INI, TOML, YAML, JSON, etc. Some applications, such as Grafana, even consume multiple formats. Also typically, there are no schemas published to specify valid configuration fields and values. This is especially true for environment variables read by an application. Hopefully that information is at least documented, but such rigor is hard to maintain for internal applications. In many cases it’s an afterthought. Often the only way to validate configuration is to run the application.</p><p>One low-hanging fruit of publishing a schema for application configuration is that it becomes easier to validate it. For example, ConfigHub supports a vet-jsonschema <a href="https://docs.confighub.com/guide/functions/">function</a> that can use JSON Schema to validate application configuration in any supported format: JSON, YAML, INI, TOML, Properties, and even Env. For our <a href="https://docs.confighub.com/guide/workers/">worker</a> component, I used a <a href="https://github.com/confighub/sdk/blob/main/cmd/cub-worker/args.go#L31">Go struct</a> to specify all supported environment variables, including default values and description text, and then generated a <a href="https://github.com/confighub/installer/blob/main/packages/worker/validation/env.schema.json">JSON schema</a> from that, which is published in the <a href="https://github.com/confighub/installer/tree/main/packages/worker">worker’s installer package</a>.</p><pre>{<br>  &quot;properties&quot;: {<br>...<br>    &quot;CONFIGHUB_URL&quot;: {<br>      &quot;description&quot;: &quot;Base URL (scheme and host) of the ConfigHub API.&quot;,<br>      &quot;type&quot;: &quot;string&quot;<br>    },<br>    &quot;CONFIGHUB_WORKER_FUNCTIONS&quot;: {<br>      &quot;description&quot;: &quot;Comma-separated list of additional worker function names to register (e.g. \&quot;vet-kyverno-server,vet-opa-gatekeeper\&quot;).&quot;,<br>      &quot;type&quot;: &quot;string&quot;<br>    },<br>    &quot;CONFIGHUB_WORKER_HTTP_SERVER_PORT&quot;: {<br>      &quot;description&quot;: &quot;Port for the worker&#39;s local HTTP server (Prometheus metrics, pprof, /internal/ok and /internal/ready). When unset, the HTTP server is not started.&quot;,<br>      &quot;type&quot;: &quot;string&quot;<br>    },<br>...</pre><p>Many server applications also still take configuration via command-line flags, though I don’t think it is a very good way to configure servers in Kubernetes. One reason is that it’s harder for tools to determine what the configuration is compared to other mechanisms. For instance, in general, it’s impossible to determine whether a value following a flag is that flag’s value or just a positional argument. Supporting <a href="https://github.com/spf13/viper">multiple ways to configure the same property</a>, such as by flag and by environment variable, makes it even harder. Also, in Kubernetes, command-line arguments are a positional array, so Kubernetes tools can’t <a href="https://medium.com/itnext/kubernetes-strategic-merge-patch-4bdd19b48789">merge</a> new flags into the list in a standard way.</p><p>Our worker uses <a href="https://cobra.dev/">cobra.dev</a> as its command framework, which supports generating <a href="https://github.com/confighub/installer/blob/main/packages/worker/validation/command.yaml">help text as YAML</a>, so we can at least produce a spec for its supported commands and flags.</p><p>In addition to application configuration, there are a number of runtime facets that are important to managing applications, such as exposed ports, probe endpoints, and parts of the filesystem written. For this, I created a <a href="https://github.com/confighub/sdk/blob/main/core/function/api/runtime_spec.go">runtime spec format</a>:</p><pre>type RuntimeSpec struct {<br> Ports  []Port  `yaml:&quot;Ports,omitempty&quot;`<br> Paths  []Path  `yaml:&quot;Paths,omitempty&quot;`<br> Probes []Probe `yaml:&quot;Probes,omitempty&quot;`<br>}<br><br>// Port describes a network port the binary listens on.<br>type Port struct {<br> // Name is a stable identifier for this port. Probes reference it by<br> // name. It is also a good candidate for containers[].ports[].name.<br> Name string `yaml:&quot;Name&quot;`<br> // Description is a human-readable summary of what is served on this port.<br> Description string `yaml:&quot;Description,omitempty&quot;`<br> // Default is the default port number used when no Source resolves.<br> // Zero (or unset) means there is no default and the port is bound<br> // only when one of Sources resolves.<br> Default int `yaml:&quot;Default,omitempty&quot;`<br> // Protocol is &quot;TCP&quot; (default) or &quot;UDP&quot;.<br> Protocol string `yaml:&quot;Protocol,omitempty&quot;`<br> // Sources, if non-empty, lists locations from which the port number<br> // may be overridden. Evaluated in priority order; the first hit wins.<br> Sources []ValueSource `yaml:&quot;Sources,omitempty&quot;`<br>}<br><br>...</pre><p>The <a href="https://github.com/confighub/installer/blob/main/packages/worker/validation/runtime.yaml">worker’s spec</a> is:</p><pre>Ports:<br>  - Name: http<br>    Description: Local HTTP server. Bound only when CONFIGHUB_WORKER_HTTP_SERVER_PORT is set. Serves /internal/metrics (Prometheus), /internal/pprof/* (pprof), /internal/routes (route listing), /internal/ok (liveness), and /internal/ready (readiness).<br>    Protocol: TCP<br>    Sources:<br>      - Type: Env<br>        EnvVar: CONFIGHUB_WORKER_HTTP_SERVER_PORT<br>Paths:<br>  - Name: tmp<br>    Description: Scratch space for schema caching and transient work. Required to be writable; safe to back with emptyDir.<br>    Path: /tmp<br>    Access: ReadWrite<br>    Persistence: Ephemeral<br>Probes:<br>  - Name: liveness<br>    Type: Liveness<br>    HTTPGet:<br>      Path: /internal/ok<br>      PortName: http<br>  - Name: readiness<br>    Type: Readiness<br>    HTTPGet:<br>      Path: /internal/ready<br>      PortName: http</pre><p>After creating this spec, I <a href="https://www.reddit.com/r/platformengineering/comments/1sea01y/trying_to_understand_if_theres_a_layer_beyond/">discovered</a> <a href="https://github.com/TrianaLab/pacto">Pacto</a> for defining a <a href="https://www.reddit.com/r/platformengineering/comments/1run7p6/how_do_you_define_the_contract_between_a_service/">runtime contract</a>. It looks similar in spirit and is worth checking out. However, it appears to have <a href="https://trianalab.github.io/pacto/contract-reference/#environment-specific-values-files">some concrete values mixed in</a>, whereas I was aiming to decouple development decisions, baked into the application, from operational choices, specified in concrete configuration, similar to <a href="https://score.dev/">score.dev</a>.</p><p>Besides configuration validation, of what use are these such interface specifications?</p><p>Instead of writing an enormous template that contains every supported configuration property, environment variable, and/or command-line flag, one can provide the specs to an <a href="https://itnext.io/converting-helm-charts-to-agent-skills-2a92829d418c">AI agent to generate a valid configuration</a>. At this point, that’s likely to yield a better experience than generating a large GUI or TUI form, for example. I’ll write another post that compares the experience of doing that with other approaches.</p><p>The agent could try to read all of the application source code, but that would consume more tokens and, depending on how well organized and how large or small the codebase is, could omit some properties or make other mistakes. If the application doesn’t have straightforward ways of generating such specs currently, I might use an AI agent to generate one. For instance, I generated a <a href="https://github.com/confighub/sdk/blob/main/cmd/fctl/test-data/grafana-ldap-schema.json">JSON Schema for Grafana LDAP configuration</a> from its documentation.</p><pre>...<br>      &quot;servers&quot;: {<br>        &quot;type&quot;: &quot;array&quot;,<br>        &quot;minItems&quot;: 1,<br>        &quot;description&quot;: &quot;List of LDAP servers to authenticate against&quot;,<br>        &quot;items&quot;: {<br>          &quot;type&quot;: &quot;object&quot;,<br>          &quot;required&quot;: [&quot;host&quot;, &quot;bind_dn&quot;, &quot;search_filter&quot;, &quot;search_base_dns&quot;, &quot;attributes&quot;],<br>          &quot;properties&quot;: {<br>            &quot;host&quot;: {<br>              &quot;type&quot;: &quot;string&quot;,<br>              &quot;description&quot;: &quot;LDAP server hostname or IP address. Can specify multiple hosts separated by space&quot;,<br>              &quot;minLength&quot;: 1,<br>              &quot;examples&quot;: [&quot;ldap.example.com&quot;, &quot;ldap1.example.com ldap2.example.com&quot;]<br>            },<br>            &quot;port&quot;: {<br>              &quot;type&quot;: &quot;integer&quot;,<br>              &quot;description&quot;: &quot;LDAP server port. Default 389 for non-SSL, 636 for SSL&quot;,<br>              &quot;minimum&quot;: 1,<br>              &quot;maximum&quot;: 65535,<br>              &quot;default&quot;: 389<br>            },<br>            &quot;use_ssl&quot;: {<br>              &quot;type&quot;: &quot;boolean&quot;,<br>              &quot;description&quot;: &quot;Use LDAPS (LDAP over SSL/TLS) for connections&quot;,<br>              &quot;default&quot;: false<br>            },<br>...</pre><p>Once a configuration has been generated, when it needs to be subsequently modified, it should be just changed directly, as I advocated in <a href="https://itnext.io/a-new-kubernetes-package-installer-324db6f615ac">my installer post</a>.</p><p>The runtime spec could be passed to a function that deterministically adds the corresponding settings to a Kubernetes Deployment, StatefulSet, or DaemonSet configuration. If you think such a function would be useful, let me know.</p><p>Because it’s so easy to generate boilerplate code using AI, one could even <a href="https://itnext.io/there-should-be-an-app-for-that-1f4d23286a9e">generate a tool</a> to manage an application. That would <a href="https://itnext.io/abstraction-is-the-wrong-way-to-simplify-configuration-81ac4bad02ab">not be an abstraction</a> in the same sense as a template, but would be a <em>view </em>over the underlying configuration.</p><p>Now that AI agents are being used to automate all kinds of tasks, they need sufficient context to perform the tasks correctly. In the absence of any information, LLMs will still just guess. For example, I’ve observed that Claude Code will just try to execute hallucinated commands rather than use --help, read documentation, ask me for clarification, or even load relevant AI agent skills. In order to correctly run and configure server applications, application interface specifications can provide the necessary context.</p><p>Do you generate or maintain specifications for your applications’ interfaces? Do you have some other way of validating your application’s configuration other than just executing the application? Do you use AI agents to generate and/or update your applications’ configurations? How often do they hallucinate? What did you do to prevent it?</p><p>Reply here, or send me a message on <a href="https://www.linkedin.com/in/bgrant0607/">LinkedIn</a>, <a href="https://x.com/bgrant0607">X/Twitter</a>, or <a href="https://bsky.app/profile/bgrant0607.bsky.social">Bluesky</a>, where I plan to crosspost this.</p><p>You could also <a href="https://auth.confighub.com/sign-up">try out ConfigHub</a>, which is now in preview.</p><p>If you found this interesting, you may be interested in other posts in my <a href="https://medium.com/@bgrant0607/list/kubernetes-8b0b8930195b">Kubernetes series</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0ba1bfc43835" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/application-interfaces-revisited-10-years-later-0ba1bfc43835">Application interfaces revisited 10 years later</a> was originally published in <a href="https://itnext.io">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Putting Agentic Ops into context: the Evolution of Operational Tools over the past 40 years]]></title>
            <link>https://itnext.io/putting-agentic-ops-into-context-the-evolution-of-operational-tools-over-the-past-40-years-b764b7785dc2?source=rss-ccdbe9a99319------2</link>
            <guid isPermaLink="false">https://medium.com/p/b764b7785dc2</guid>
            <category><![CDATA[infrastructure-as-code]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[ai-agent]]></category>
            <dc:creator><![CDATA[Brian Grant]]></dc:creator>
            <pubDate>Mon, 20 Jul 2026 15:26:50 GMT</pubDate>
            <atom:updated>2026-07-20T21:11:58.392Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VnvOSdgFBoaZbUxEY7DCrA.png" /><figcaption>Image generated by Gemini</figcaption></figure><p>This topic came up as we, the ConfigHub cofounders, were speculating the future of operational tooling back in 2024. I didn’t find an article like this, so I decided to research the topic and relate the history to my own experience. References are at the end. I’m glad I waited to write this post, because the role of AI in operational tasks in the future, while not yet clear, has started to take shape.</p><p>I’m going to cover the generations of tools that have overlapped with my career. Like human generations, they overlap, but I’ll break them up approximately by decade. I have first-hand experience with some of the tools, but not with most of them. In some cases the tools were adjacent to what I was working on at the time, and Google developed its own internal tools. I’m most familiar with the Cloud Native era that arose around Kubernetes, Terraform, and the cloud.</p><p>There are clear recurring patterns: increased automation to cope with increased scale of machines and deployments, increased version-control complexity to cope with increased scale of collaboration, adoption of new technologies to save costs, waves of fragmentation and consolidation, and waves of services and self-hosting.</p><p>What do you think will come next?</p><p>This is kind of long since it covers four decades. If you aren’t interested in the history, you can skip to the AI section (search for “2023-today”) near the end.</p><h3>Late 1980s: the batch-and-mainframe-to-Unix transition and ITIL</h3><p>This era is dominated by tools that assumed a modest number of expensive, long-lived machines managed by hand. In this era, I used Apple computers, PCs, and Digital VAX minicomputers through attached terminals and dial-up modem network connections.</p><p><strong>Version control</strong> was already a generation old. SCCS, developed in 1972, was nearly the first version control tool. RCS was released in 1982 at Purdue, where I went to college, as a free alternative. RCS was, naturally, the first version-control system I used. RCS was used as an ops tool as well as a dev tool. It was used to version Unix config files and scripts, not just source code. CVS was created in 1986 as a set of shell scripts layered on top of RCS to add repository-level tracking and a client-server model, but I didn’t use it in that form.</p><p><strong>Batch scheduling / queuing</strong> in high-performance computing (HPC) traces to this period. The Network Queueing System (NQS) is considered the progenitor of essentially all later batch schedulers — it originated at NASA Ames around 1986. Condor (now HTCondor) started around 1988 at the University of Wisconsin, built around the idea of cycle-scavenging idle desktop workstations. Great name. I became exposed to this area at the beginning of the next decade, but both of these tools were influential.</p><p><strong>Process/service management</strong> meant mainly init and cron . There was no &quot;process supervisor&quot; concept yet AFAIK, just rc scripts. Different systems used different distributions of Unix based on SysV Unix (IRIX, AIX) and BSD Unix (Vax, SunOS, Ultrix). Purdue had all of those systems, and more.</p><p><strong>Process/ITSM frameworks</strong> were born at this time. The British government’s CCTA released the original ITIL in 1989, and the first ITIL books were Service Level Management, Help Desk (incorporating Incident Management), Contingency Planning, and Change Management. The CMDB concept came directly out of this framework — the ITIL principles from the late 1980s called for a database to track IT services, making the CMDB idea nearly as old as ITIL itself. On the commercial side, Remedy Corporation shipped its Action Request System in 1991, which became the dominant help-desk/ITSM ticketing engine for over a decade.</p><h3>Early-to-mid 1990s: commodity Unix, Linux, and clusters</h3><p>The defining shift was cheap commodity hardware and the arrival of Linux, which suddenly meant there were <em>many</em> machines to manage. During this decade I used PCs, minicomputers, workstations, and supercomputers.</p><p><strong>CVS matured</strong> into the version-control tool of choice. It was rewritten in C, and CVS 1.0 was submitted to the Free Software Foundation in November 1990. CVS, which supported merging changes, definitely made it easier to collaborate on software projects compared to RCS, which locked files. I used both version-control systems over NFS, AFS, and other network filesystems rather than with a central hub service.</p><p><strong>Configuration management tools were invented</strong>. CFEngine was created in 1993 to automate management of a group of workstations. Its lasting contributions were the ideas of <em>declarative desired state</em> and <em>convergence</em> (self-healing toward a defined state rather than running imperative scripts once) — the intellectual foundation for everything that followed. Mark Burgess’s 1995 USENIX paper “Cfengine: a site configuration engine” is the canonical primary source. I didn’t have the need to use it at the time.</p><p><strong>Clustering captured the zeitgeist.</strong> The title of a conference talk/paper from Lawrence Livermore National Lab (LLNL) in 1990 was <a href="https://en.wikipedia.org/wiki/Killer_micro">“Attack of the Killer Micros”</a>. Beowulf originally referred to a specific machine built in 1994 at NASA — a 16-node cluster of Intel DX4 processors on channel-bonded Ethernet, built at CESDIS at Goddard under the HPCC/ESS project. The term “cluster of workstations” became prevalent. The concept, if not yet the reality, was commodity off-the-shelf (COTS) hardware plus open-source software replacing supercomputers. I interned at LLNL in the the early 90s, and NASA Ames was a 45-minute drive from there. I evaluated <a href="https://www.csm.ornl.gov/pvm/">PVM</a>, though mostly on distributed-memory <a href="https://www.admin-magazine.com/HPC/Articles/A-Brief-History-of-Supercomputers">supercomputers</a> rather than workstations, and participated in the creation of <a href="https://en.wikipedia.org/wiki/Message_Passing_Interface">MPI</a>, particularly the collective communication model and libraries. I was a user rather than an operator of these systems.</p><p><strong>Workload management was commercialized.</strong> Platform Computing was founded in 1992, and its LSF product was based on the Utopia research project at the University of Toronto. LSF became the standard commercial scheduler in finance and semiconductor HPC. In the mid-1990s NASA Ames developed the Portable Batch System (PBS) as an NQS replacement, spawning the OpenPBS → PBS Pro → TORQUE lineage. These queueing systems had similar CLI interfaces to submit jobs: bsub, qsub, etc. They were convenient for launching and managing long-running (multi-hour) tests, simulations, and batch jobs on shared pools of machines, even if just a handful of machines. Some supercomputers <a href="https://www.nas.nasa.gov/assets/nas/pdf/techreports/1995/nas-95-006.pdf">had similar interfaces</a>, but for parallel workloads.</p><p><strong>Package managers appeared.</strong> dpkg was created for Debian in January 1994, first as a shell script, later rewritten in Perl and then C. RPM was written in 1997 by Erik Troan and Marc Ewing, cofounder of Red Hat— though a primitive RPM shipped with Red Hat Linux 2.0 back in 1995. I interned with Marc at IBM in 1991. We built a system for distributing files to a network of AIX workstations because the local network, and network filesystem, went down frequently. I don’t know if that was part of the inspiration for RPM, but a package manager would have been useful.</p><p><strong>Network monitoring got its first graphing tool.</strong> MRTG (Multi Router Traffic Grapher) was created in 1995 to monitor router traffic via SNMP, producing time series bandwidth graphs. Big Brother, one of the first tools to give a red/green web dashboard of host and service state, appeared around 1996–1997. The purpose wasn’t exactly the same, but at LLNL in the early 1990s I used ParaGraph and Pablo profiling/tracing tools to visualize supercomputer communication overheads over time, so similar ideas were emerging in multiple areas.</p><p><strong>ssh was developed</strong>. In 1995. The network tools I had used up to that point — telnet, ftp, etc. — did not use encryption. We could telnet to SMTP servers to send email as anyone. Simpler times, if less secure. The Internet was just entering the public awareness with the emergence of the World Wide Web.</p><h3>Late 1990s: monitoring, supervision, and dependency-aware packaging</h3><p>I was in grad school focusing on compilers at the time rather than working at a Dot-Com, and not paying much attention to operational tools, so this period was filled in by searching. Let me know if I missed something important or got some details wrong.</p><p><strong>Nagios arrived (as NetSaint).</strong> NetSaint was released in 1999, and renamed to Nagios in 2002. Its plugin architecture, a core scheduler running arbitrary check plugins, made it the default open-source monitoring system for over a decade, and it was the ancestor of Icinga, Shinken, and Naemon. I think I encountered Nagios, or at least heard of it, at Transmeta in the early 2000s.</p><p><strong>RRDtool reshaped metrics storage.</strong> RRDtool was released in 1999 as a spin-off of MRTG, using a round-robin (fixed-size, circular-buffer) database so storage stayed constant over time. It became the storage/graphing backend for nearly every monitoring tool of the 2000s. Storage was a lot more limited and expensive then compared to now, so bounded storage was a necessity, unlike today’s scalable timeseries databases.</p><p><strong>Process supervision became a named concept.</strong> daemontools (with svscan and supervise) established the modern pattern — a master process that stays parent to each service and restarts it immediately on crash, with reliable logging. It inspired runit and other process supervisors.</p><p><strong>APT added dependency management.</strong> The Advanced Package Tool was developed for Debian in 1998, as a high-level front-end to dpkg that resolved dependencies and fetched from centralized repositories. yum brought the same capability to RPM systems a couple of years later. Dependency management is so essential now, that it’s hard to imagine that packages didn’t always provide it, though containers later repeated a similar pattern.</p><h3>2000s: DevOps groundwork and the VM explosion</h3><p>The combination of multi-core chips, hardware acceleration, and the Great Recession ushered in the <a href="https://queue.acm.org/detail.cfm?id=1017000">virtual machine (VM)</a> era, to save costs through consolidation, creating even more “machines” to manage. Steve Herrod was my manager at Transmeta for a week, before he left for VMWare. Containers in Linux (cgroups) were developed for similar reasons (i.e., binpacking) around the time I joined Google. Google skipped VMs and adopted containers on bare metal. Go was developed at Google late in the decade, also.</p><p><strong>Version control split into centralized-vs-distributed.</strong> Subversion (SVN) was created in 2000 to be “CVS done right,” with atomic commits and efficient branching; version 1.0 shipped in February 2004. That was the main version-control tool I used immediately prior to joining Google. Meanwhile BitKeeper (a proprietary distributed VCS founded around 2000) was used by the Linux kernel from 2002 to 2005. When the free BitKeeper license was revoked in April 2005, Git and Mercurial were both created in response. Linus had already left Transmeta at that point to focus on Linux, so I wasn’t immediately aware of git. I used Mercurial a little, and the fully distributed model was challenging to keep sane in the absence of a central hub like Github. Google used Perforce, which had a centralized model.</p><p><strong>Configuration management became mainstream.</strong> Puppet was released in 2005. Puppet adopted CFEngine’s declarative desired-state idea and added a Ruby-based DSL with a client-server model. Chef followed in 2009, and was the predominant alternative alongside Puppet and CFEngine, taking a more imperative Ruby &quot;recipes&quot; approach. CFEngine itself was rewritten as CFEngine 3 in 2008 around Promise Theory. GitHub was founded around the same time and found early traction in the Ruby community. Google had its own internal solution to this problem, at least for its datacenter machines, designed to manage its massive fleet of systems underlying <a href="https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43438.pdf">Borg</a>.</p><p><strong>Monitoring fragmented.</strong> Cacti was started in 2001 as a friendlier PHP front-end to RRDtool, aiming for more ease of use than RRDtool and more flexibility than MRTG. Ganglia targeted large HPC clusters with a lightweight multicast agent. Zabbix was released in 2001 as an integrated monitoring platform, and Munin (~2003) offered RRD-based node graphing. The Nagios fork Icinga appeared in 2009. Within Google, the BorgMon monitoring tool was developed, which later served as the inspiration for Prometheus.</p><p><strong>Init systems evolved.</strong> Ubuntu’s Upstart (Canonical, 2006) introduced event-based booting as a SysV-init successor, and runit (~2004) carried the daemontools approach forward.</p><p><strong>Continuous integration emerged as tooling.</strong> CruiseControl, produced by ThoughtWorks, was the venerable early CI server (2001), preceded by Mozilla’s Tinderbox (~1998). Hudson was created in summer 2004 at Sun Microsystems and first released in February 2005. It overtook CruiseControl in popularity in 2008 and by 2010 held an estimated 70% of the CI market. We implemented continuous, or at least daily, build-and-test automation at Transmeta and PeakStream, but I don’t recall using a CI tool. I think it was just cron and a queuing system. Again, Google had its own “continuous build” mechanism, developed to work with its <a href="https://research.google/pubs/why-google-stores-billions-of-lines-of-code-in-a-single-repository/">massive monorepo codebase</a>.</p><p><strong>Service management went SaaS.</strong> ITIL v2 (2000–2001) consolidated the framework into the well-known Service Support and Service Delivery volumes, and ITIL v3 in 2007 renamed the process to Service Asset and Configuration Management, cementing the CMDB’s central role. Self-hosted ITSM products like BMC Remedy and HP Service manager dominated in this decade, but ServiceNow was founded in 2004 and would go on to dominate ITSM/CMDB tools in the future.</p><p><strong>Infrastructure as a Service</strong>. AWS launched in 2006, which was essentially the beginning of the public cloud.</p><p><strong>Platform as a Service</strong>. Heroku launched in 2007, and AppEngine launched in 2008. CloudFoundry was conceived in 2009. By the early 2010s, there were many PaaS offerings, which were soon <a href="https://itnext.io/kubernetes-and-application-platforms-7a36aa5c3c08">disrupted by Kubernetes</a>.</p><p><strong>Cloud Management Platforms</strong>. Rightscale was founded in 2007. The formula of a centralized management control plane plus policy enforcement and cost management was carried forward into both the infrastructure-as-code world (“<a href="https://itnext.io/spice-up-your-infrastructure-as-code-with-tacos-1a9c179e0783">TACOS</a>”) and Kubernetes world.</p><h3>2010–2013: consolidation into “DevOps”</h3><p><strong>Jenkins forked from Hudson.</strong> After Oracle acquired Sun and claimed the Hudson trademark, the community voted to rename the project Jenkins. The first Jenkins release shipped in 2011 under the MIT license, and roughly 85% of the community migrated within months. When Kubernetes started, Jenkins was still extremely popular, and the project adopted it, until the scale became too large to manage.</p><p><strong>systemd rewrote Linux init.</strong> Introduced in 2010, systemd absorbed process supervision, socket activation, logging, timers, and mount management into PID 1 — controversially subsuming much of what daemontools, Upstart, and separate cron/inetd tooling had done. This was the last major process management change before the container era. Systemd was used together with Docker initially.</p><p><strong>New configuration management tools, and YAML. </strong>Ansible arrived in 2012, with an agentless SSH-plus-YAML approach — a reaction to the agent-and-DSL complexity of Puppet and Chef. Ansible was growing rapidly in popularity around the time we started working on Kubernetes, and was commonly used to manage the nodes of Kubernetes clusters, even though we initially implemented node setup using Salt, another new config-management tool. Popular CI SaaS products, such as TravisCI and CircleCI, also adopted YAML. YAML became pervasive in the DevOps tools space.</p><p><strong>Infrastructure as Code</strong>. AWS CloudFormation launched in 2010–2011. It later served as the inspiration for Terraform.</p><h3>2013–2022: Cloud Native, Containers, Kubernetes, GitOps, X as Code, and the later ZIRP era</h3><p>As the original lead architect of Kubernetes, Kubernetes Steering Committee member, and CNCF Technical Oversight Committee member, I was heavily involved in this era. I already wrote about the <a href="https://itnext.io/the-technical-history-of-kubernetes-2fe1988b522a">history of Kubernetes</a> and <a href="https://medium.com/@bgrant0607/is-the-kubernetes-cncf-ecosystem-too-big-complex-and-fragmented-350d52f37e31">about the CNCF landscape</a>, so I won’t rehash those topics in this post.</p><p><strong>Containers. </strong>Docker was first shown publicly March 2013. It bundled cgroups, namespaces, LXC, union filesystems, and other Linux primitives into a developer-friendly image format and runtime. That obviously had a huge impact on the operational landscape. The Open Container Initiative formed in June 2015 to standardize image and runtime formats, and containerd was contributed to the CNCF in 2017.</p><p><strong>Container orchestration</strong>. An effort to build a container-based platform started within Google later in 2013. We open-sourced Kubernetes at the first DockerCon in June 2014. The Cloud Native Computing Foundation (CNCF) was kicked off in July 2015, coinciding with Kubernetes 1.0. Kubernetes was the leading “container orchestrator” by the end of 2017. Not only did the cloud Kubernetes services (GKE, EKS, etc.) become popular, but Kubernetes also was widely adopted “on prem”.</p><p><strong>Whitebox monitoring and multi-faceted observability. </strong>Prometheus was developed in 2012, inspired by Google’s Borgmon, and was open-sourced in January 2015. It was the second project to join CNCF, in 2016. Its whitebox pull-based, label-based, dimensional metrics model became the de facto monitoring standard for Kubernetes workloads. OpenTelemetry, also a CNCF project, formed in 2019 by merging OpenTracing and Google’s OpenCensus into one vendor-neutral standard for metrics, logs, and traces, which was adopted by major observability vendors. Grafana, released in 2014, became popular for visualization. Timeseries databases were developed for metric storage. Elasticsearch became popular for searching logs.</p><p><strong>Service mesh appeared.</strong> Linkerd (2016) coined the term “service mesh”. Service meshes initially used sidecar containers to proxy traffic. Envoy was open-sourced (2016), serving as a data plane. Istio launched in 2017 as a service-mesh control plane. All of these projects eventually landed in CNCF.</p><p><strong>GitHub Actions</strong> disrupted the SaaS CI tool space in 2019.</p><p><strong>Kubernetes configuration as code. </strong>Helm (2015–2016; discussed some in <a href="https://itnext.io/a-new-kubernetes-package-installer-324db6f615ac">this post</a>) became the de facto Kubernetes package manager and templating tool. <a href="https://itnext.io/generating-transforming-and-patching-kubernetes-configuration-with-kustomize-fb7b02476a1b">Kustomize</a> was created in 2017. <a href="https://itnext.io/kubernetes-configuration-in-2024-434abc7a5a1b">Dozens upon dozens of other configuration generation tools</a> were developed.</p><p><a href="https://itnext.io/is-gitops-actually-useful-a1c851ba99d8"><strong>GitOps</strong></a><strong> was coined</strong> by my cofounder, Alexis Richardson, in 2017 — Git as the source of truth, with a controller continuously reconciling actual state to the desired state. Weaveworks’ <strong>Flux</strong> (2016) and Intuit’s <strong>Argo CD</strong> (~2018), both now in CNCF, became the two most popular GitOps tools.</p><p><strong>Infrastructure as Code</strong> was widely adopted, starting with Terraform (using HCL) in 2014, and then Pulumi (<a href="https://itnext.io/generating-configuration-using-general-purpose-programming-languages-19230a2c2573">using general-purpose programming languages</a>) and Crossplane (<a href="https://itnext.io/using-the-kubernetes-resource-model-to-provision-cloud-infrastructure-dc8b75a4e328">using KRM</a>) in 2018. Terraform had the largest <a href="https://itnext.io/infrastructure-as-code-landscape-overview-2024-a066124e5989">ecosystem of tools</a> built around it. A lot of those tools shifted to OpenTofu later.</p><p><strong>Configuration management (the OS/filesystem variety) faded</strong>. All of the major tools, Puppet, Chef, Ansible, and Salt, were acquired, and their popularity diminished with the rise in popularity of API-driven operations.</p><p><a href="https://itnext.io/what-are-state-based-policy-constraints-good-for-019b70f0b698"><strong>Policy as Code</strong></a> <strong>arrived.</strong> Open Policy Agent<strong> / </strong>OPA (2016) with Gatekeeper for admission control, later joined by Kyverno (2020).</p><p><a href="https://itnext.io/the-challenge-presented-by-secrets-in-declarative-configuration-193a4d12b08e"><strong>Secret management</strong></a><strong> became essential</strong>. Vault was very popular, at least until the license change to BSL, but there were many other tools.</p><p><a href="https://itnext.io/infrastructure-as-code-cloud-asset-management-a729222e0126"><strong>Asset management tools</strong></a><strong> came to the cloud</strong>. To help manage security and cost.</p><p><strong>Platform engineering tools and practices developed mindshare. </strong>Kubernetes users built internal developer platforms almost from the beginning, but the concept really took off around 2018.<strong> </strong>Backstage was open-sourced in 2020 and donated to CNCF, seeding the developer-portal trend. Part of the motivation of both movements was to hide some of the complexity caused by the tool proliferation of the era.</p><p><strong>ITSM partially connected with DevOps</strong>. Jira Service Desk (2013) integrated with Atlassian’s developer-oriented tools, like Bitbucket Pipelines and Confluence. ServiceNow acquired and developed some DevOps capabilities, also, but I’m unsure how they fared.</p><h3>2023-today: Post-ZIRP and AI</h3><p>Obviously AI isn’t the only recent development. eBPF is replacing service-mesh sidecar containers, for example. OpenTofu was forked from Terraform due to the BSL license change, which was part of a bigger trend of open-source license reversals. ClickHouse’s column storage is now used to <a href="https://matduggan.com/clickhouse-is-winning-the-observability-wars/">store observability data</a>.</p><p>It’s already clear that AI will define the current era, though.</p><p><strong>Tools and services built to run AI workloads</strong>. Inference gateways, <a href="https://modelplane.ai/blog/open-control-plane-for-inference">Modelplane</a>, <a href="https://github.com/nvidia/aicr">AICR</a>, inference services, neoclouds, etc. Kubernetes was extended with AI-oriented features, such as DRA, also. That this area is getting attention is a no-brainer, but it is still in the early stages.</p><p><strong>Git is under pressure</strong>. There have been shifts in version-control tools before. And similar to past responses to increases in scale, the large volume of changes made by AI agents is motivating challenges to git, or at least its current implementation, such as <a href="https://x.com/bstaples/status/2054227478093468135">GitLab’s “act 2”</a>.</p><p><strong>Terminal-based AI agents</strong> have made it possible for AI to use current tools (especially CLIs), which the LLMs were also trained on. Due to the prevalence and increasing effectiveness of AI agents, <a href="https://georgefa.com/blog/ax-is-the-new-dx/">agent experience is starting to supersede developer experience</a> in importance. On the other hand, AI agents may also be able to <a href="https://itnext.io/converting-helm-charts-to-agent-skills-2a92829d418c">replace some tools</a> or at least some use cases, and can also be used to <a href="https://itnext.io/there-should-be-an-app-for-that-1f4d23286a9e">rapidly develop new tools</a>. It’s still too early to guess how this will play out, but I’d guess that tools built by AI, for AI will win eventually. <a href="https://github.com/swamp-club/swamp">SWAMP</a> is an example of that — AI-generated workflows for AI agents to execute, using an AI-authored tool.</p><p>New tools aren’t in the models’ training sets, however, so that’s where <a href="https://agentskills.io/home">agent skills</a> and/or custom harnesses, like <a href="https://github.com/stakpak/agent">Stakpak</a>, need to fill the gap. Where CLIs aren’t available, MCP servers can provide access to tools, and agents can use browsers as well, though sometimes they get blocked. Tool selection, context management, memory, sharing information across individuals, and safety are current challenges.</p><p>A number of recent reddit threads indicate that many people are not ready to let agents make changes to production environments yet, which is a significant contrast with software development, where autonomous “loops” are a hot topic. Approaches I’ve seen to constrain AI agents include using credentials with read-only permission, MCP servers to evaluate CLI commands and other requests and/or make read-only API requests, secret proxies, network sandboxes with policy gates, deny-tools command lists, PreToolUse hooks that vet command patterns, and <a href="https://github.com/borisdali/helpdesk">specialized harnesses</a> with <a href="https://github.com/borisdali/helpdesk/blob/main/docs/AIGOVERNANCE.md#3-policy-engine">policy layers</a>. The patterns to separate “safe” from “unsafe” changes appear coarse-grained so far. Presumably coding agents are being used to make changes to configuration as code in git, with human review, but I haven’t seen a lot of discussion about it yet, other than for creating new Helm charts and Kubernetes manifests. ConfigHub provides a governance layer that can validate changes automatically as well as gate changes on human review, and we are working on agent-based techniques as well.</p><p><strong>Root-cause analysis (RCA), troubleshooting, incident response, and on-call agents </strong>are a current trend, collecting and interpreting multiple data sources quickly<strong>:</strong></p><ul><li><a href="http://anyshift.io">Anyshift.io</a>: context graph for RCA</li><li><a href="https://neubird.ai/home">Neubird</a>: autonomous production operations agent</li><li><a href="http://wildmoose.ai">wildmoose.ai</a>: AI first responder</li><li><a href="https://vibraniumlabs.ai/">Vibranium labs</a>: oncall</li><li><a href="http://resolve.ai">resolve.ai</a>: oncall, RCA</li><li><a href="http://sre.ai">sre.ai</a></li><li><a href="https://cleric.ai/">Cleric</a>: RCA, oncall</li><li><a href="https://home.robusta.dev/">https://home.robusta.dev/</a>: RCA, filter alerts</li><li><a href="http://traversal.com">Traversal.com</a>: RCA, remediation</li><li><a href="https://engage.groundcover.com/agent-mode">https://engage.groundcover.com/agent-mode</a></li><li><a href="https://betterstack.com/ai-sre">https://betterstack.com/ai-sre</a></li><li><a href="https://opsverse.io/aiden/">https://opsverse.io/aiden/</a></li><li><a href="https://syncable.dev/ai-sre">https://syncable.dev/ai-sre</a></li><li><a href="https://aihelpdesk.biz/">aiHelpDesk</a>: diagnose and remediate database issues</li><li><a href="https://www.prodrescueai.com/">ProdRescue AI</a>: incident reports, RCA</li></ul><p>I’d imagine that this could drive significant changes to observability tools and other tools used during incident management (e.g., Slack, Jira).</p><p><strong>Alternatives to Infrastructure/configuration as code</strong>. <a href="https://github.com/platform-engineering-labs/formae">Formae</a> is an IaC tool that can generate Pkl code from live infrastructure. <a href="https://github.com/systeminit/si">System Initiative</a> launched and was archived, in favor of SWAMP, which was mentioned earlier. <a href="https://github.com/spacelift-io/spacelift-intent">Spacelift Intent</a> directly calls provider APIs without using OpenTofu or other IaC tool. I’ve seen at least <a href="https://dev.to/mikesol/using-claude-code-to-pilot-kubernetes-on-autodock-3k04">one example of someone replacing Helm with markdown files</a> also. With ConfigHub, we’re using the approach of using AI to <a href="https://itnext.io/there-should-be-an-app-for-that-1f4d23286a9e">generate tools</a> to manipulate <a href="https://itnext.io/what-is-configuration-as-data-210b0c4be324">configuration as data</a>.</p><p>Infrastructure/configuration as code tools and formats were developed by software engineers for (human) software engineers. <a href="https://itnext.io/how-software-engineering-instincts-clash-with-infrastructure-as-code-6b18a9cd9cef">Human software-engineering instincts are already strained</a> by IaC/CaC. AI agents don’t have the same constraints and preferences or strengths and weaknesses, and design choices made for existing tools may not be the best choices for agents — DX vs. AX.</p><p>I think the question is not whether “X as code”, at least as we know it, will be superseded, but what will it be replaced by.</p><p>What operational tools do you remember using earlier in your career? How do you think AI will shape future tools? Are you already using AI in operations? In production? To make changes? What safety mechanisms do you use? Have you used AI to build any new operational tools, even if just for yourself?</p><p>Feel free to reply here, or send me a message on <a href="https://www.linkedin.com/in/bgrant0607/">LinkedIn</a>, <a href="https://x.com/bgrant0607">X/Twitter</a>, or <a href="https://bsky.app/profile/bgrant0607.bsky.social">Bluesky</a>, where I plan to crosspost this.</p><p>You could also <a href="https://auth.confighub.com/sign-up">try out ConfigHub</a>, which is now in preview.</p><p>If you found this interesting, you may be interested in other posts in my <a href="https://medium.com/@bgrant0607/list/infrastructure-as-code-and-declarative-configuration-8c441ae74836">Infrastructure as Code and Declarative Configuration series</a>.</p><h3>References</h3><p><strong>Version control</strong></p><ul><li>Tichy, “RCS — A System for Version Control,” <em>Software: Practice and Experience</em> 15(7), 1985 — <a href="https://www.gnu.org/software/rcs/tichy-paper.pdf">https://www.gnu.org/software/rcs/tichy-paper.pdf</a></li><li>Dick Grune’s own CVS history page — <a href="https://dickgrune.com/Programs/CVS.orig/">https://dickgrune.com/Programs/CVS.orig/</a></li><li>Wikipedia: Concurrent Versions System — <a href="https://en.wikipedia.org/wiki/Concurrent_Versions_System">https://en.wikipedia.org/wiki/Concurrent_Versions_System</a></li><li>Wikipedia: Apache Subversion — <a href="https://en.wikipedia.org/wiki/Apache_Subversion">https://en.wikipedia.org/wiki/Apache_Subversion</a></li><li>“The World Before Git” — <a href="https://osshistory.org/p/the-world-before-git">https://osshistory.org/p/the-world-before-git</a></li><li>“A Git Origin Story,” <em>Linux Journal</em> — <a href="https://www.linuxjournal.com/content/git-origin-story">https://www.linuxjournal.com/content/git-origin-story</a></li></ul><p><strong>Configuration management</strong></p><ul><li>Burgess, “Cfengine: a site configuration engine,” USENIX Computing Systems, 1995 (and “Computer Immunology,” LISA ‘98)</li><li>Mark Burgess, “CFEngine’s Star Trek and AI origins” — <a href="https://mark-burgess-oslo-mb.medium.com/cfengines-star-trek-and-ai-origins-e99096fe845b">https://mark-burgess-oslo-mb.medium.com/cfengines-star-trek-and-ai-origins-e99096fe845b</a></li><li>Wikipedia: CFEngine — <a href="https://en.wikipedia.org/wiki/CFEngine">https://en.wikipedia.org/wiki/CFEngine</a></li><li>Wikipedia: Puppet (software) — <a href="https://en.wikipedia.org/wiki/Puppet_(software)">https://en.wikipedia.org/wiki/Puppet_(software)</a></li><li>“Engines, Puppets, Promises: The Figurations of Configuration Management” (Springer) — <a href="https://link.springer.com/chapter/10.1007/978-981-19-2476-7_6">https://link.springer.com/chapter/10.1007/978-981-19-2476-7_6</a></li></ul><p><strong>Clustering / HPC</strong></p><ul><li>Sterling, Becker, et al., “BEOWULF: A parallel workstation for scientific computation,” ICPP 1995</li><li>Beowulf.org history — <a href="https://www.beowulf.org/overview/history.html">https://www.beowulf.org/overview/history.html</a></li><li>“The Roots of Beowulf” (NASA NTRS) — <a href="https://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/20150001285.pdf">https://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/20150001285.pdf</a></li></ul><p><strong>Batch schedulers / queuing</strong></p><ul><li>Zhou et al., “Utopia: A Load Sharing Facility for Large, Heterogeneous Distributed Computer Systems,” 1993</li><li>Wikipedia: IBM Spectrum LSF — <a href="https://en.wikipedia.org/wiki/IBM_Spectrum_LSF">https://en.wikipedia.org/wiki/IBM_Spectrum_LSF</a> and Platform Computing — <a href="https://en.wikipedia.org/wiki/Platform_Computing">https://en.wikipedia.org/wiki/Platform_Computing</a></li><li>NASA TM 109025, “A Comparison of Queueing, Cluster and Distributed Computing Systems” — <a href="https://www.cs.odu.edu/~mln/ltrs-pdfs/tm109025.pdf">https://www.cs.odu.edu/~mln/ltrs-pdfs/tm109025.pdf</a> (excellent primary comparison of NQS, LSF, Condor, DQS, PBS)</li><li>“Scalable System Scheduling for HPC and Big Data” (arXiv 1705.03102) — <a href="https://arxiv.org/pdf/1705.03102">https://arxiv.org/pdf/1705.03102</a></li></ul><p><strong>Monitoring / observability</strong></p><ul><li>Nagios official history — <a href="https://www.nagios.org/about/history/">https://www.nagios.org/about/history/</a></li><li>Wikipedia: Nagios — <a href="https://en.wikipedia.org/wiki/Nagios">https://en.wikipedia.org/wiki/Nagios</a></li><li>Wikipedia: RRDtool — <a href="https://en.wikipedia.org/wiki/RRDtool">https://en.wikipedia.org/wiki/RRDtool</a> and MRTG — <a href="https://en.wikipedia.org/wiki/Multi_Router_Traffic_Grapher">https://en.wikipedia.org/wiki/Multi_Router_Traffic_Grapher</a></li><li>Wikipedia: Cacti — <a href="https://en.wikipedia.org/wiki/Cacti_(software)">https://en.wikipedia.org/wiki/Cacti_(software)</a></li><li>Sysbee, “Monitoring origins: Where it all began” — <a href="https://www.sysbee.net/blog/monitoring-origins-where-it-all-began/">https://www.sysbee.net/blog/monitoring-origins-where-it-all-began/</a></li></ul><p><strong>CI</strong></p><ul><li>Wikipedia: Jenkins — <a href="https://en.wikipedia.org/wiki/Jenkins_(software)">https://en.wikipedia.org/wiki/Jenkins_(software)</a> and Hudson — <a href="https://en.wikipedia.org/wiki/Hudson_(software)">https://en.wikipedia.org/wiki/Hudson_(software)</a></li><li>“Hudson is Retiring. The End of a Jenkins’ Story” (DamageControl/CruiseControl lineage) — <a href="https://medium.com/swlh/hudson-is-retiring-the-end-of-a-jenkins-story-9cee866135ab">https://medium.com/swlh/hudson-is-retiring-the-end-of-a-jenkins-story-9cee866135ab</a></li></ul><p><strong>Package managers</strong></p><ul><li>Andrew Nesbitt, “Package Manager Timeline” — <a href="https://nesbitt.io/2025/11/15/package-manager-timeline.html">https://nesbitt.io/2025/11/15/package-manager-timeline.html</a></li><li>Wikipedia: RPM — <a href="https://en.wikipedia.org/wiki/RPM_Package_Manager">https://en.wikipedia.org/wiki/RPM_Package_Manager</a> and dpkg — <a href="https://en.wikipedia.org/wiki/Dpkg">https://en.wikipedia.org/wiki/Dpkg</a></li></ul><p><strong>Process / service supervision</strong></p><ul><li>Wikipedia: Process supervision — <a href="https://en.wikipedia.org/wiki/Process_supervision">https://en.wikipedia.org/wiki/Process_supervision</a></li><li>D.J. Bernstein’s daemontools — <a href="https://cr.yp.to/daemontools.html">https://cr.yp.to/daemontools.html</a></li><li>“A Survey of Unix Init Schemes” (arXiv 0706.2748) — <a href="https://arxiv.org/pdf/0706.2748">https://arxiv.org/pdf/0706.2748</a></li></ul><p><strong>ITSM / ITIL / CMDB</strong></p><ul><li>IBM, “What Is ITIL?” — <a href="https://www.ibm.com/think/topics/it-infrastructure-library">https://www.ibm.com/think/topics/it-infrastructure-library</a></li><li>“A Brief History of ITIL” (year-by-year timeline) — <a href="http://itservicemngmt.blogspot.com/2007/09/brief-history-of-itil.html">http://itservicemngmt.blogspot.com/2007/09/brief-history-of-itil.html</a></li><li>TechTarget, “What Is a CMDB?” — <a href="https://www.techtarget.com/searchdatacenter/definition/configuration-management-database">https://www.techtarget.com/searchdatacenter/definition/configuration-management-database</a></li><li><a href="https://www.itgovernance.eu/blog/en/the-evolution-of-itil-how-the-framework-has-reshaped-it-service-management">https://www.itgovernance.eu/blog/en/the-evolution-of-itil-how-the-framework-has-reshaped-it-service-management</a></li></ul><p><strong>DevOps</strong></p><ul><li><a href="https://www.reddit.com/r/devops/comments/117q820/what_were_the_ancient_tools_used_in_late_90s/">https://www.reddit.com/r/devops/comments/117q820/what_were_the_ancient_tools_used_in_late_90s/</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b764b7785dc2" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/putting-agentic-ops-into-context-the-evolution-of-operational-tools-over-the-past-40-years-b764b7785dc2">Putting Agentic Ops into context: the Evolution of Operational Tools over the past 40 years</a> was originally published in <a href="https://itnext.io">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building tools to manage cross-cutting concerns in Kubernetes]]></title>
            <link>https://itnext.io/building-tools-to-manage-cross-cutting-concerns-in-kubernetes-by-using-confighub-6bd64afcdd69?source=rss-ccdbe9a99319------2</link>
            <guid isPermaLink="false">https://medium.com/p/6bd64afcdd69</guid>
            <category><![CDATA[config-as-code]]></category>
            <category><![CDATA[config-as-data]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Brian Grant]]></dc:creator>
            <pubDate>Mon, 13 Jul 2026 15:01:45 GMT</pubDate>
            <atom:updated>2026-07-14T21:18:45.645Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W5wsjHgClkt-m3b5jIAU0Q.png" /><figcaption>Robots operating on many similar machines</figcaption></figure><p>I previously wrote about <a href="https://itnext.io/configuration-as-code-is-a-liability-for-security-d0e53727cd17">some Kubernetes namespace policy management tools</a> I built on ConfigHub: <a href="https://github.com/confighub/examples/tree/main/namespace-manager">namespace-manager</a>, <a href="https://github.com/confighub/examples/tree/main/rbac-manager-for-agents">rbac-manager</a>, <a href="https://github.com/confighub/examples/tree/main/network-policy-manager">network-policy-manager</a>. Next, I built some workload-oriented management tools.</p><p>Workloads have some interfaces that are specific to them, such as command-line flags, environment variables, <a href="https://itnext.io/the-challenge-of-configmap-rollouts-in-kubernetes-1431398c0866">application configuration</a>, probe endpoints, and so on. The configuration of those properties is best performed with the schemas and constraints for those interfaces in hand. I’ll write another post about that.</p><p>Workloads also have security and operational <a href="https://itnext.io/the-tension-between-flexibility-and-simplicity-in-infrastructure-as-code-6cec841e3d16">aspects that are cross-cutting</a> and at least partly determined by the deployment environment, such as operating-system capabilities, computational resources, scaling automation, network access, and observability integrations. The workload may have some requirements of these aspects, but their configuration is not unique to the workload and is often at least partly determined by factors outside the workload implementation, such as the choice of autoscaling tool or observability tool, as well as the <a href="https://itnext.io/kubernetes-and-application-platforms-7a36aa5c3c08">application or infrastructure platform</a>, which is Kubernetes for this discussion. Typically the security posture is determined at the organization level, as well, or at least at a larger scope than just one workload.</p><p>With <a href="https://itnext.io/what-is-configuration-as-data-210b0c4be324">configuration as data</a>, not all changes to configuration need to be implemented in a single <a href="https://itnext.io/fundamental-challenges-with-infrastructure-as-code-imply-the-language-doesnt-matter-41030475c296">monolithic generator</a>. They can be decoupled and performed by interoperable tools, just as when building tools <a href="https://itnext.io/configuration-needs-an-api-b36f08b92551">against APIs</a>.</p><p>To demonstrate this, I built a suite of tools, in one day, each with its own agent skills:</p><ul><li><a href="https://github.com/confighub/examples/tree/main/workload-manager"><strong>workload-manager</strong></a>: ensures workloads follow best practices, such as resource specifications, probes, pod disruption budgets, and security context</li><li><a href="https://github.com/confighub/examples/tree/main/scheduling-manager"><strong>scheduling-manager</strong></a>: checks and sets node selectors, node affinity, and tolerations according to some pre-defined profiles</li><li><a href="https://github.com/confighub/examples/tree/main/observability-manager"><strong>observability-manager</strong></a>: manages <a href="https://prometheus-operator.dev/docs/developer/getting-started/#using-servicemonitors">Prometheus ServiceMonitor</a> resources and <a href="https://opentelemetry.io/docs/collector/">Open Telemetry collector</a> sidecar containers</li><li><a href="https://github.com/confighub/examples/tree/main/autoscale-manager"><strong>autoscale-manager</strong></a>: checks and edits HorizontalPodAutoscaler and/or <a href="https://keda.sh/">KEDA</a> resources, and can convert HPA to KEDA</li></ul><p>All of these tools can enforce desired postures, as well, by installing <a href="https://docs.confighub.com/background/entities/trigger/">ConfigHub Triggers</a> via guardrails commands.</p><p>As with the policy tools, these tools can provide lists of the relevant resources across clusters, snapshot summaries, and detailed lists of findings.</p><p>For example, from the workload manager:</p><pre>CLUSTER                                 WORKLOADS  PDBS  UNITS  GATED  UNAPPLIED<br>dev-cluster                             19         0     33     0      21<br>prod-cluster                            19         0     38     0      24<br><br>2 clusters, 38 workloads, 0 pdbs, 71 units (0 gated, 45 unapplied)<br><br>SEVERITY  ANALYZER   CLUSTER          NAMESPACE  KIND         NAME     MESSAGE<br>...<br>HIGH      security   prod-cluster     appvote    Deployment   vote     container &quot;vote&quot; does not set runAsNonRoot: true<br>HIGH      security   prod-cluster     appvote    Deployment   worker   container &quot;worker&quot; allowPrivilegeEscalation is not false<br>HIGH      security   prod-cluster     appvote    Deployment   worker   container &quot;worker&quot; does not set runAsNonRoot: true<br>...<br>MEDIUM    probes     dev-cluster      appchat    Deployment   backend  container &quot;backend&quot; no readiness probe<br>MEDIUM    resources  dev-cluster      appchat    Deployment   backend  container &quot;backend&quot; no cpu limit<br>MEDIUM    resources  dev-cluster      appchat    Deployment   backend  container &quot;backend&quot; no cpu request<br>MEDIUM    security   dev-cluster      appchat    Deployment   backend  container &quot;backend&quot; readOnlyRootFilesystem is not true<br>...<br>LOW       hygiene    dev-cluster      appvote    Deployment   redis    container &quot;redis&quot; terminationMessagePolicy is not FallbackToLogsOnError<br>...<br>39 workloads (0 pass, 21 warn, 18 fail)<br>...<br>  dev-cluster/appchat Deployment/backend:<br>    [hygiene] container &quot;backend&quot; terminationMessagePolicy is not FallbackToLogsOnError<br>    [probes] container &quot;backend&quot; no liveness probe<br>    [probes] container &quot;backend&quot; no readiness probe<br>    [resources] container &quot;backend&quot; no cpu limit<br>    [resources] container &quot;backend&quot; no cpu request<br>    [security] automountServiceAccountToken is not false<br>    [security] container &quot;backend&quot; allowPrivilegeEscalation is not false<br>    [security] container &quot;backend&quot; does not drop ALL capabilities<br>    [security] container &quot;backend&quot; does not set runAsNonRoot: true<br>    [security] container &quot;backend&quot; readOnlyRootFilesystem is not true<br>    [security] container &quot;backend&quot; seccompProfile is not RuntimeDefault<br>...</pre><p>(terminationMessagePolicy is a <a href="https://github.com/kubernetes/kubernetes/issues/139">pet feature of mine</a>. I find it useful.)</p><p>If this were a typical policy tool and you were using helm, cdk8s, tanka, or other configuration as code tool, then you’d have to track down the right repo and configuration files, render the configuration and feed it to the policy tool, ask your AI agent to implement fixes for the correct environment(s), review the changes, and render the configuration again, for all potentially affected environments, and feed it to a validation tool.</p><p>But this is configuration as data, so those checks ran against fully materialized configuration and you can just change the configuration directly with the tools.</p><p>Want a PodDisruptionBudget?</p><pre>% ./bin/cub-workload ensure-pdb appchat-prod/frontend --max-unavailable &quot;10%&quot; -o table<br>Dry run — would create PDB Unit appchat-prod/frontend-pdb:<br><br>apiVersion: policy/v1<br>kind: PodDisruptionBudget<br>metadata:<br>  name: frontend-pdb<br>  namespace: appchat<br>spec:<br>  maxUnavailable: 10%<br>  selector:<br>    matchLabels:<br>      app: frontend<br><br>Re-run with --commit --change-desc &quot;…&quot; to create the Unit. It is not applied until you apply it.</pre><p>That automatically matched the namespace and labels of the specified workload, similar to kubectl expose, which makes it even simpler than kubectl create pdb. For a single workload, an AI agent could also author the resource from whole cloth, especially if it had a sample to start from, as in <a href="https://github.com/confighub/confighub-skills/blob/main/references/yaml-patterns.md#poddisruptionbudget">our agent skills</a> or in our <a href="https://github.com/confighub/installer/blob/main/packages/kubernetes-resources/bases/default/hello-pdb.yaml">package of resource samples</a>.</p><p>Want spreading via pod anti-affinity? ./bin/cub-workload ensure-spread appchat-prod/frontend. CPU and memory resources? set-resources. Probes? set-probes . Security context and disabled service account token? harden . There are lots more features I don’t have space to show in this post. There’s no need to write YAML by hand when tools are perfectly capable of doing it. Remember, these tools are operating on automatically versioned storage, not operating on the clusters directly.</p><p>To be agent-friendly, the commands execute in dry-run mode and output JSON by default. Agents can execute the commands and generate summary tables regarding the current state and changes they made. With approval, they can “commit” changes by performing the same operations without dry run. The changed configuration will be validated automatically, and humans can still review and approve the changes before releasing them to the clusters, and can revert or adjust them as needed.</p><p>I’m also experimenting with writing higher-level intent to the resources using annotations. I’ll write about that in a future post once I have more experience with it.</p><p>Under the hood, the tools work through a combination of ConfigHub <a href="https://docs.confighub.com/background/concepts/filters/">filters</a>, <a href="https://docs.confighub.com/guide/functions/">functions</a>, such as set-container-resources and set-pod-container-security-context-defaults, client-side analysis, client-side mutations using the <a href="https://github.com/confighub/sdk/tree/main/function-impl#readme">function SDK</a>, saved function <a href="https://docs.confighub.com/background/entities/invocation/">invocations</a>, triggers, and other ConfigHub features.</p><p>Because the configuration data is authoritative, these tools can interoperate with other tools that read and write the configuration data, including other ConfigHub functions, such as set-container-image-reference and set-replicas, and AI agents editing the configuration. Tools can be single-purpose or multi-purpose, as desired, but they don’t need to be monolithic, covering every field of Kubernetes Deployment, StatefulSet, and DaemonSet resources. The tools can also be versatile. They can analyze, summarize, generate, backfill, promote, validate, and enforce.</p><p>And because we <a href="https://itnext.io/configuration-belongs-in-a-database-be6d31f46d3d">store configuration in a database</a> instead it being sprawled across many git repositories owned by different teams, the tools don’t need to clone lots of repositories, make ad hoc edits, commit and push changes, and create lots of PRs. Instead, they can call APIs. I’ll write another post about the “initiative” system that we’re working on for changes that span multiple teams.</p><p>I’m sure the tools still have rough edges and need more work to make them usable — I’ve barely had time to try them yet, but the point is that we CAN build and use tools if we’re not using configuration as code. The tools can operate on variants of the same application across clusters, and across multiple different applications, not just a single application, using standard APIs. Composable, interoperable tools are more reusable than all-encompassing templates.</p><p>That last point is pretty important. Yes, for one application, you could use a heavily parameterized <a href="https://github.com/companyinfo/helm-charts/blob/main/charts/helmet/templates/_hpa.yaml">template library</a> to create, say, a HorizontalPodAutoscaler, and add <a href="https://github.com/companyinfo/helm-charts/blob/main/charts/helmet/values.yaml#L441">some values</a> to drive it. But every Helm chart exposes different input values, so there’s no standard API that one could build tooling around.</p><p>The common alternative automated mutation methods are less transparent and more complicated operationally:</p><ul><li>kustomize overlays</li><li>mutating dynamic admission control</li><li>Operators (e.g., to inject sidecar containers or <a href="https://www.reddit.com/r/kubernetes/comments/1ultrqo/wow_so_poddisruptionbudget_pdb_is_exactly_what/">create PDBs</a>)</li></ul><p>ConfigHub makes it possible to directly modify configuration while maintaining multiple similar-but-different variants of the same configuration through <a href="https://itnext.io/how-are-variants-managed-in-confighub-061bc2f89cff">mechanisms that help keep them in sync</a>. That’s one of the key challenges we’re solving with ConfigHub.</p><p>I think the approach of building tools on top of configuration as data has a lot of promise. Try the tools and let me know what you think. Or build your own and let me know how they turn out.</p><p>What cross-cutting customizations do you frequently need to make to Kubernetes workload configurations? What are common changes that aren’t supported by helm charts you use? How do you override them? What common operational changes do you need to make beyond just replicas and resources? How often do people in your organization “break glass” and change clusters directly? What are the reasons? Have you migrated from HPA to KEDA, or Ingress to Gateway, or Nginx Ingress to Traefik? Do you have operations, platform, or security teams that need to modify configurations owned by application teams? What kubectl commands or other tools should we adapt to ConfigHub?</p><p>Reply here, or send me a message on <a href="https://www.linkedin.com/in/bgrant0607/">LinkedIn</a>, <a href="https://x.com/bgrant0607">X/Twitter</a>, or <a href="https://bsky.app/profile/bgrant0607.bsky.social">Bluesky</a>, where I plan to crosspost this.</p><p>You could also <a href="https://auth.confighub.com/sign-up">try out ConfigHub</a>, which is now in preview.</p><p>If you found this interesting, you may be interested in other posts in my <a href="https://medium.com/@bgrant0607/list/kubernetes-8b0b8930195b">Kubernetes series</a> or my <a href="https://medium.com/@bgrant0607/list/infrastructure-as-code-and-declarative-configuration-8c441ae74836">IaC / declarative configuration series</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6bd64afcdd69" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/building-tools-to-manage-cross-cutting-concerns-in-kubernetes-by-using-confighub-6bd64afcdd69">Building tools to manage cross-cutting concerns in Kubernetes</a> was originally published in <a href="https://itnext.io">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>