<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[DevOps.dev - Medium]]></title>
        <description><![CDATA[Devops.dev is a community of DevOps enthusiasts sharing insight, stories, and the latest development in the field. - Medium]]></description>
        <link>https://blog.devops.dev?source=rss----33f8b2d9a328---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>DevOps.dev - Medium</title>
            <link>https://blog.devops.dev?source=rss----33f8b2d9a328---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 01:41:35 GMT</lastBuildDate>
        <atom:link href="https://blog.devops.dev/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com" rel="hub"/>
        <item>
            <title><![CDATA[A Bug Called ‘SSHole’ Lets You Log Into Alpine Linux With Any Password Over 256 Characters]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://blog.devops.dev/a-bug-called-sshole-lets-you-log-into-alpine-linux-with-any-password-over-256-characters-c47cc6f947a3?source=rss----33f8b2d9a328---4"><img src="https://cdn-images-1.medium.com/max/1200/1*C0MKzmk6okdh_AI-M_b7Dw.png" width="1200"></a></p><p class="medium-feed-snippet">CVE-2026&#x2013;123158 combines a 2012 musl libc decision with a 2016 OpenSSH fix to let an unauthenticated attacker log into key-only accounts &#x2014;&#x2026;</p><p class="medium-feed-link"><a href="https://blog.devops.dev/a-bug-called-sshole-lets-you-log-into-alpine-linux-with-any-password-over-256-characters-c47cc6f947a3?source=rss----33f8b2d9a328---4">Continue reading on DevOps.dev »</a></p></div>]]></description>
            <link>https://blog.devops.dev/a-bug-called-sshole-lets-you-log-into-alpine-linux-with-any-password-over-256-characters-c47cc6f947a3?source=rss----33f8b2d9a328---4</link>
            <guid isPermaLink="false">https://medium.com/p/c47cc6f947a3</guid>
            <category><![CDATA[container-security]]></category>
            <category><![CDATA[cve]]></category>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[ssh]]></category>
            <category><![CDATA[alphalinux]]></category>
            <dc:creator><![CDATA[Vortex 404]]></dc:creator>
            <pubDate>Mon, 16 Nov 2026 13:26:15 GMT</pubDate>
            <atom:updated>2026-11-16T13:26:14.150Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Your Team Agrees on the Fix in Teams, Then Retypes It Into GitHub]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://blog.devops.dev/github-app-microsoft-teams-copilot-thread-to-pull-request-606a8ca2150a?source=rss----33f8b2d9a328---4"><img src="https://cdn-images-1.medium.com/max/1672/1*Vfc0BnOJnZ44dDdvwOcAlQ.png" width="1672"></a></p><p class="medium-feed-snippet">The GitHub app in Microsoft Teams is now GA. How @GitHub turns a thread into a Copilot pull request, who it acts as, and what to set up&#x2026;</p><p class="medium-feed-link"><a href="https://blog.devops.dev/github-app-microsoft-teams-copilot-thread-to-pull-request-606a8ca2150a?source=rss----33f8b2d9a328---4">Continue reading on DevOps.dev »</a></p></div>]]></description>
            <link>https://blog.devops.dev/github-app-microsoft-teams-copilot-thread-to-pull-request-606a8ca2150a?source=rss----33f8b2d9a328---4</link>
            <guid isPermaLink="false">https://medium.com/p/606a8ca2150a</guid>
            <category><![CDATA[github]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[github-copilot]]></category>
            <dc:creator><![CDATA[Dave R | Microsoft Azure & AI MVP ☁️]]></dc:creator>
            <pubDate>Mon, 16 Nov 2026 13:26:12 GMT</pubDate>
            <atom:updated>2026-11-16T13:26:11.571Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Kubernetes Is Retiring kube-proxy’s IPVS Mode. Canary nftables First]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://blog.devops.dev/kubernetes-is-retiring-kube-proxys-ipvs-mode-canary-nftables-first-5ec3e12c91fb?source=rss----33f8b2d9a328---4"><img src="https://cdn-images-1.medium.com/max/1600/1*FPEX6Pf8TdWN38iazCg2_Q.png" width="1600"></a></p><p class="medium-feed-snippet">Trace one packet through iptables, IPVS, and nftables mode, then canary nftables on one node pool before the upgrade decides for you.</p><p class="medium-feed-link"><a href="https://blog.devops.dev/kubernetes-is-retiring-kube-proxys-ipvs-mode-canary-nftables-first-5ec3e12c91fb?source=rss----33f8b2d9a328---4">Continue reading on DevOps.dev »</a></p></div>]]></description>
            <link>https://blog.devops.dev/kubernetes-is-retiring-kube-proxys-ipvs-mode-canary-nftables-first-5ec3e12c91fb?source=rss----33f8b2d9a328---4</link>
            <guid isPermaLink="false">https://medium.com/p/5ec3e12c91fb</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[technology]]></category>
            <dc:creator><![CDATA[Daniel Valev]]></dc:creator>
            <pubDate>Mon, 16 Nov 2026 13:26:06 GMT</pubDate>
            <atom:updated>2026-11-16T13:26:05.019Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[40 Kubernetes Interview Questions That Expose Weak Senior Engineers]]></title>
            <link>https://blog.devops.dev/40-kubernetes-interview-questions-that-expose-weak-senior-engineers-f183baad39fb?source=rss----33f8b2d9a328---4</link>
            <guid isPermaLink="false">https://medium.com/p/f183baad39fb</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[kubernetes-cluster]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Devrim Ozcay  | Founder, ProdRescue]]></dc:creator>
            <pubDate>Mon, 16 Nov 2026 13:26:05 GMT</pubDate>
            <atom:updated>2026-11-16T13:26:03.822Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BCyJZh4vbL_Kt5cPy9MTRw.png" /></figure><h4><em>Advanced Kubernetes troubleshooting, distributed systems, networking, autoscaling, and production reliability for Senior and Staff Engineers.</em></h4><p>A Kubernetes cluster with 180 nodes had plenty of available CPU. Every deployment reported success, and the control plane appeared healthy. Yet customers were experiencing intermittent 503 errors, and the API’s p99 latency had climbed from 180 milliseconds to almost nine seconds.</p><p>The infrastructure team increased the number of replicas. The application team increased connection pool sizes. Someone suggested moving the workloads to larger nodes because the existing cluster was obviously struggling.</p><p>The situation became worse.</p><p>The problem was that newly created Pods were passing readiness checks before their applications could serve real traffic. Every additional replica introduced another temporarily unhealthy endpoint into the load-balancing path. Autoscaling was increasing the number of requests routed toward instances that were technically running but operationally useless.</p><p>This is why Kubernetes interviews become considerably harder at the Staff level. Knowing which command describes a Pod is useful, but it does not demonstrate an understanding of how scheduling, network routing, application lifecycle, resource allocation, and distributed dependencies interact during failure.</p><p>The following 40 questions focus on those interactions. The scenarios are hypothetical, but the failure mechanisms and diagnostic approaches are grounded in Kubernetes and production engineering behavior.</p><h3>Part I — Kubernetes Scheduling and Resource Management</h3><h3>1. A Pod is Pending even though the cluster has available CPU and memory. Why?</h3><p>A candidate who immediately recommends adding nodes is missing several possibilities. Kubernetes scheduling evaluates requested resources alongside node selectors, affinity rules, taints, tolerations, topology constraints, persistent-volume placement, and other scheduling restrictions. Aggregate cluster capacity is irrelevant when no eligible node satisfies the Pod’s complete set of constraints.</p><p>Start with kubectl describe pod and examine scheduling events. Then compare node allocatable resources against requests rather than actual utilization, and inspect placement rules. A node showing 30% CPU usage can still reject a Pod because its remaining schedulable CPU requests are insufficient.</p><p><strong>Staff-level follow-up:</strong> How would you design scheduling policies that preserve high availability without creating unnecessary fragmentation across availability zones?</p><p><strong>Resource:</strong> <a href="https://devrimozcay.gumroad.com/l/ghfzo">Kubernetes Production Debugging OS</a>.</p><h3>2. Why can increasing CPU limits make latency worse?</h3><p>CPU limits are enforced through Linux cgroup mechanisms. A container can exhaust its allocated CPU quota during a scheduling period and experience throttling even when the physical node has spare CPU capacity. This can increase request latency, particularly for workloads with bursty CPU demand.</p><p>Investigate container throttling metrics alongside application latency and CPU utilization. Removing CPU limits may help some latency-sensitive workloads, but doing so without considering node contention can create noisy-neighbor problems. The correct decision depends on workload isolation, resource requests, and available node capacity.</p><p><strong>Staff-level follow-up:</strong> Explain how you would evaluate CPU limits for latency-sensitive Java services sharing a Kubernetes node with batch processing workloads.</p><p><strong>Resource:</strong> <a href="https://devrimozcay.gumroad.com/l/rkdgug">Backend Performance Engineering OS</a>.</p><h3>3. A Java application is OOMKilled while heap usage remains below Xmx. Explain.</h3><p>The JVM heap is only part of a Java process’s memory footprint. Metaspace, thread stacks, direct byte buffers, JIT code cache, native allocations, and other memory consumers contribute to the container’s total memory usage. Kubernetes memory limits apply to the container’s cgroup-accounted memory rather than exclusively to Java heap allocations.</p><p>Inspect container memory metrics, JVM native memory information where available, thread counts, and off-heap allocations. Increasing -Xmx may make the situation worse because it leaves less headroom for non-heap memory.</p><p><strong>Staff-level follow-up:</strong> Design a memory budget for a Spring Boot application deployed with a 2 GiB container memory limit.</p><p><strong>Resource:</strong> <a href="https://devrimozcay.gumroad.com/l/rlmxr">Java &amp; Spring Boot Production Debugging OS</a>.</p><h3>4. Why can Kubernetes scheduling create uneven availability-zone distribution?</h3><p>The scheduler attempts to satisfy placement constraints and optimize placement according to its configuration. Resource availability, existing Pods, topology spread constraints, affinity rules, and node-group composition can all influence placement.</p><p>A Deployment with six replicas does not automatically guarantee two replicas in each of three zones. Explicit topology spread constraints and appropriately configured node capacity are needed when zone distribution is an availability requirement.</p><p><strong>Staff-level follow-up:</strong> Design a workload that remains available after losing an entire availability zone while avoiding unnecessary scheduling failures.</p><p><strong>Resource:</strong> <a href="https://devrimozcay.gumroad.com/l/jwtmh">System Design Architecture Tradeoffs</a>.</p><h3>5. Why can a PodDisruptionBudget fail to protect an application during an outage?</h3><p>PodDisruptionBudgets primarily constrain voluntary disruptions that use the eviction API. They cannot prevent node failures, kernel crashes, or every form of Pod termination.</p><p>A PDB also cannot create capacity that does not exist. If a workload has insufficient healthy replicas or replacement Pods cannot schedule, maintenance operations may become blocked without improving the application’s actual resilience.</p><p><strong>Staff-level follow-up:</strong> Explain how PDBs, topology spread constraints, and node-drain procedures should work together.</p><p><strong>Resource:</strong> <a href="https://devrimozcay.gumroad.com/l/kfccl">Production Deployment Checklist</a>.</p><h3>Part II — Networking, DNS, and Service Discovery</h3><h3>6. DNS resolution becomes intermittent only during peak traffic. Where do you investigate?</h3><p>DNS failures can originate from CoreDNS saturation, upstream resolver behavior, network packet loss, conntrack pressure, application-side caching, or excessive query volume. The first step is determining whether failures affect all Pods, particular nodes, or specific applications.</p><p>Inspect CoreDNS metrics and logs, DNS request latency, node-level networking, and the application’s resolver behavior. Scaling CoreDNS may help when it is genuinely saturated, but it will not fix a node-level packet-drop problem.</p><p><strong>Staff-level follow-up:</strong> Design a DNS reliability strategy for a cluster with thousands of short-lived workloads and frequent service discovery.</p><p><strong>Resource:</strong> <a href="https://devrimozcay.gumroad.com/l/tlrgao">Linux Production Troubleshooting OS</a>.</p><h3>7. A Service has healthy Pods but intermittently returns connection refused. Why?</h3><p>A Service routes traffic through its endpoint selection and the cluster’s networking implementation. Intermittent failures may involve endpoint readiness transitions, application listener configuration, terminating Pods, connection reuse, or networking faults.</p><p>Check EndpointSlices, Pod readiness, target ports, application logs, and connection behavior during rollout. Distinguish a failure to establish a TCP connection from an application-generated HTTP error because those symptoms indicate different failure layers.</p><p><strong>Staff-level follow-up:</strong> Explain how EndpointSlice updates and terminating connections affect zero-downtime deployments.</p><p><strong>Resource:</strong> <a href="https://devrimozcay.gumroad.com/l/ccfeud">Kubernetes &amp; Docker Troubleshooting Playbook</a>.</p><h3>8. What happens when conntrack tables become saturated?</h3><p>Linux connection tracking maintains state for network flows. Under high connection churn, the table can approach capacity, causing new flows to fail or packets to be dropped depending on the networking path and configuration.</p><p>Investigate conntrack utilization, insertion failures, packet drops, and connection creation rates. Increasing table capacity without addressing excessive connection churn can postpone the failure while increasing memory consumption.</p><p><strong>Staff-level follow-up:</strong> Design a mitigation for a high-QPS service generating excessive short-lived TCP connections.</p><p><strong>Resource:</strong> <a href="https://devrimozcay.gumroad.com/l/xbihfx">Production Incident Debugging OS</a>.</p><h3>9. Why can NetworkPolicies appear correct while traffic is still blocked?</h3><p>NetworkPolicy enforcement depends on the cluster’s networking implementation. Policies are additive for allowed traffic, and both ingress and egress restrictions can affect a connection.</p><p>A workload may be permitted to receive traffic but unable to reach its dependency because egress rules are missing. DNS access is another common dependency that engineers overlook when introducing restrictive policies.</p><p><strong>Staff-level follow-up:</strong> Design a default-deny networking strategy for a payment platform without breaking service discovery or operational access.</p><p><strong>Resource:</strong> <a href="https://devrimozcay.gumroad.com/l/qjlaki">Senior System Design Interview OS</a>.</p><h3>10. How do you investigate cross-zone network latency?</h3><p>Start by establishing whether latency is caused by network transit, service routing, dependency placement, or application processing. Distributed traces can reveal whether a request crosses availability zones repeatedly before reaching its final dependency.</p><p>Topology-aware routing may reduce unnecessary cross-zone traffic in suitable environments, but it introduces tradeoffs around load distribution and availability. Locality should not be optimized at the expense of correctness or resilience.</p><p><strong>Staff-level follow-up:</strong> Design a routing strategy that balances cross-zone cost, latency, and fault tolerance.</p><p><strong>Resource:</strong> <a href="https://devrimozcay.gumroad.com/l/wtbklt">Backend Latency Debugging OS</a>.</p><h3>Part III — Deployments, Readiness, and Availability</h3><h3>11. Why can a Deployment report success while customers receive 503 errors?</h3><p>Deployment availability is based on Kubernetes-defined readiness and availability conditions. Those conditions only represent real application health when readiness probes accurately reflect whether the application can serve traffic.</p><p>A probe that returns success before critical initialization finishes can route requests to an unprepared instance. Conversely, a readiness probe that checks every downstream dependency can remove all replicas from service during a shared dependency outage.</p><p><strong>Staff-level follow-up:</strong> Design readiness behavior for a payment API that depends on PostgreSQL, Redis, and an external payment provider.</p><p><strong>Resource:</strong> <a href="https://devrimozcay.gumroad.com/l/fmcerf">Spring Boot Production Readiness OS</a>.</p><h3>12. What is the difference between startup, readiness, and liveness probes?</h3><p>Startup probes allow applications with long initialization times to avoid premature liveness and readiness evaluation. Readiness probes determine whether a Pod should receive traffic through normal Service endpoint selection. Liveness probes indicate whether the container should be restarted.</p><p>A liveness probe that fails whenever PostgreSQL becomes unavailable can restart every application replica during a database incident. That restart storm may destroy warm caches and intensify recovery pressure.</p><p><strong>Staff-level follow-up:</strong> Explain which dependencies should influence readiness and which should influence liveness.</p><p><strong>Resource:</strong> <a href="https://devrimozcay.gumroad.com/l/spring-boot-troubleshooting">Spring Boot Production Troubleshooting OS</a>.</p><h3>13. How can rolling deployments cause database connection storms?</h3><p>A rolling deployment temporarily increases the number of application instances. If each Spring Boot instance initializes a HikariCP pool with dozens of connections, the combined connection demand can exceed PostgreSQL capacity.</p><p>For example, 40 replicas with a maximum pool size of 30 represent a potential 1,200 application connections. Temporary rollout surge, other services, and background workers can push demand higher.</p><p><strong>Staff-level follow-up:</strong> Design a connection budget that accounts for maximum rollout surge and multiple application deployments.</p><p><strong>Resource:</strong> <a href="https://devrimozcay.gumroad.com/l/pwjrlt">PostgreSQL Production Debugging OS</a>.</p><h3>14. Why does graceful shutdown sometimes fail?</h3><p>Kubernetes begins Pod termination and provides a termination grace period. The application must stop accepting new work and coordinate the completion or recovery of in-flight operations.</p><p>Connection draining depends on several layers, including the application, networking path, load balancer, and timing of endpoint updates. A preStop hook alone does not guarantee that every existing connection will finish successfully.</p><p><strong>Staff-level follow-up:</strong> Design graceful shutdown for a Spring Boot API with 30-second payment provider calls.</p><p><strong>Resource:</strong> <a href="https://devrimozcay.gumroad.com/l/kfccl">Production Deployment Checklist</a>.</p><h3>15. What makes a Kubernetes deployment genuinely rollback-safe?</h3><p>Application rollback is straightforward only when the previous application version remains compatible with the database schema, message formats, and external dependencies. Destructive migrations can make a binary rollback insufficient.</p><p>Use expand-and-contract schema changes, backward-compatible message evolution, and staged deployment strategies. Define rollback triggers using customer-facing metrics rather than relying exclusively on Pod health.</p><p><strong>Staff-level follow-up:</strong> Design a zero-downtime migration that renames a heavily used database column across multiple service versions.</p><p><strong>Resource:</strong> <a href="https://devrimozcay.gumroad.com/l/decision-map">Backend Architecture Decisions OS</a>.</p><h3>Part IV — Autoscaling and Overload</h3><h3>16. Why can HPA scaling make an incident worse?</h3><p>Horizontal Pod Autoscaler increases replicas based on configured metrics and scaling behavior. If a shared dependency is saturated, adding application replicas can increase requests against that dependency.</p><p>A database already struggling with connection contention may experience even greater pressure after application scaling. The scaling policy needs to account for downstream capacity and workload characteristics.</p><p><strong>Staff-level follow-up:</strong> Design autoscaling for an API constrained by a PostgreSQL database with a fixed connection budget.</p><p><strong>Resource:</strong> <a href="https://devrimozcay.gumroad.com/l/rkdgug">Backend Performance Engineering OS</a>.</p><h3>17. Why does CPU-based HPA fail for I/O-bound workloads?</h3><p>An application can spend most of its time waiting on databases or external APIs while using relatively little CPU. CPU utilization therefore may not represent request pressure or customer latency.</p><p>Consider metrics such as request concurrency, queue depth, or workload-specific saturation. Each metric requires careful interpretation because scaling on an unbounded queue can create unstable feedback loops.</p><p><strong>Staff-level follow-up:</strong> Design an autoscaling signal for a Kafka consumer whose processing rate is limited by a downstream API.</p><p><strong>Resource:</strong> <a href="https://devrimozcay.gumroad.com/l/menhx">Backend Production Incident OS</a>.</p><h3>18. What causes autoscaling oscillation?</h3><p>Delayed metrics, rapid scaling decisions, changing workloads, and insufficient stabilization can cause replicas to repeatedly increase and decrease.</p><p>HPA behavior configuration can help stabilize decisions, but it cannot compensate for a fundamentally incorrect metric. Evaluate how quickly new replicas become useful and how long the workload takes to respond to scaling.</p><p><strong>Staff-level follow-up:</strong> Explain how you would tune scaling behavior for a service with a two-minute startup time.</p><p><strong>Resource:</strong> <a href="https://devrimozcay.gumroad.com/l/artmf">Production Failure Field Guide</a>.</p><h3>19. How do you design backpressure in Kubernetes-based systems?</h3><p>Kubernetes can add compute capacity, but it does not automatically provide application-level admission control. Services need bounded concurrency, queue limits, and predictable overload responses.</p><p>When a downstream dependency becomes saturated, rejecting some work may protect the overall system. For asynchronous workloads, durable queues and controlled consumption can separate incoming demand from processing capacity.</p><p><strong>Staff-level follow-up:</strong> Design a system that survives a tenfold traffic spike without overwhelming PostgreSQL.</p><p><strong>Resource:</strong> <a href="https://devrimozcay.gumroad.com/l/jwtmh">System Design Architecture Tradeoffs</a>.</p><h3>20. Why can Cluster Autoscaler fail to add useful capacity?</h3><p>Node autoscaling depends on the available node groups, infrastructure quotas, instance types, scheduling constraints, and cloud-provider behavior. Adding nodes of the wrong type will not help Pods requiring specific resources.</p><p>Inspect unschedulable Pod reasons and autoscaler events. A workload requiring a GPU or particular topology cannot be scheduled merely by adding generic CPU nodes.</p><p><strong>Staff-level follow-up:</strong> Design node groups that balance workload specialization, cost, and recovery capacity.</p><p><strong>Resource:</strong> <a href="https://devrimozcay.gumroad.com/l/ghfzo">Kubernetes Production Debugging OS</a>.</p><h3>Part V — Stateful Workloads and Distributed Consistency</h3><h3>21. Why is running PostgreSQL on Kubernetes operationally difficult?</h3><p>PostgreSQL requires durable storage, controlled failover, replication management, backup recovery, and protection against split-brain behavior. Kubernetes can manage Pods and storage attachments, but it does not automatically implement database consistency guarantees.</p><p>Operators can automate database lifecycle operations, although their correctness and failure behavior must still be evaluated. Persistent volumes preserve data according to their storage guarantees; they do not make database recovery automatic.</p><p><strong>Staff-level follow-up:</strong> Compare managed PostgreSQL with an operator-managed PostgreSQL cluster on Kubernetes.</p><p><strong>Resource:</strong> <a href="https://devrimozcay.gumroad.com/l/jgogeh">Database Outage Runbook</a>.</p><h3>22. What happens when a StatefulSet Pod moves to another node?</h3><p>StatefulSets provide stable Pod identity and associated persistent storage semantics. Whether the volume can attach to the new node depends on its access mode, topology, storage driver, and underlying infrastructure.</p><p>A volume restricted to one availability zone may prevent the replacement Pod from scheduling in another zone. Storage architecture therefore becomes part of the application’s availability design.</p><p><strong>Staff-level follow-up:</strong> Design recovery for a stateful service after complete availability-zone failure.</p><p><strong>Resource:</strong> <a href="https://devrimozcay.gumroad.com/l/qjlaki">Senior System Design Interview OS</a>.</p><h3>23. How can Kubernetes failover create split-brain risks?</h3><p>A distributed stateful system can face situations where an old leader is isolated but remains operational while a replacement leader is promoted. Without appropriate coordination and fencing, both may accept conflicting writes.</p><p>Kubernetes leader election mechanisms do not automatically provide fencing for every external resource. Stateful applications must define how stale leaders lose authority.</p><p><strong>Staff-level follow-up:</strong> Design a fencing strategy for a database failover system.</p><p><strong>Resource:</strong> <a href="https://devrimozcay.gumroad.com/l/qktrrh">Staff Engineer Architecture Decision OS</a>.</p><h3>24. Why can Kafka consumers become unstable after scaling?</h3><p>Increasing consumer replicas can trigger group rebalancing. If partitions are limited, adding more consumers than useful partition assignments provides little additional parallelism.</p><p>Slow processing, poorly configured timeouts, and non-idempotent operations can make rebalances particularly disruptive. Scaling must account for partition count and downstream capacity.</p><p><strong>Staff-level follow-up:</strong> Design consumer scaling that preserves ordering and limits duplicate business effects.</p><p><strong>Resource:</strong> <a href="https://devrimozcay.gumroad.com/l/menhx">Backend Production Incident OS</a>.</p><h3>25. Why can Redis failover cause inconsistent application behavior?</h3><p>A Redis failover can change the primary node while clients hold stale connections or retry operations. Depending on replication and persistence configuration, recently acknowledged writes may not survive certain failures.</p><p>Applications must define which Redis data can be reconstructed and which operations require stronger durability guarantees.</p><p><strong>Staff-level follow-up:</strong> Explain why Redis-based distributed locks may require fencing tokens for critical operations.</p><p><strong>Resource:</strong> <a href="https://devrimozcay.gumroad.com/l/hrqvq">Redis Production Debugging OS</a>.</p><h3>Part VI — Security, Isolation, and Supply Chain</h3><h3>26. Why is running containers as root dangerous?</h3><p>Container isolation does not make elevated privileges harmless. Vulnerabilities, unsafe mounts, or excessive capabilities can increase the impact of a compromised workload.</p><p>Use appropriate security contexts, restricted capabilities, filesystem permissions, and admission policies. Evaluate whether workloads genuinely require privileged operations.</p><p><strong>Staff-level follow-up:</strong> Design a security baseline for a multi-tenant Kubernetes platform.</p><p><strong>Resource:</strong> <a href="https://devrimozcay.gumroad.com/l/tljcyg">Docker Production Debugging OS</a>.</p><h3>27. How do you prevent Kubernetes secret exposure?</h3><p>Kubernetes Secrets are API objects intended for sensitive data, but their protection depends on access controls, encryption configuration, operational practices, and workload exposure.</p><p>Restrict RBAC permissions, avoid unnecessary secret distribution, and establish rotation procedures. Base64 encoding is not encryption.</p><p><strong>Staff-level follow-up:</strong> Design secret rotation without restarting every service simultaneously.</p><p><strong>Resource:</strong> <a href="https://devrimozcay.gumroad.com/l/fmcerf">Spring Boot Production Readiness OS</a>.</p><h3>28. How can a compromised service account affect an entire cluster?</h3><p>A workload using a service account with excessive permissions may allow an attacker to access Kubernetes API resources beyond the workload’s intended scope.</p><p>Use least-privilege RBAC, restrict unnecessary token mounting, and monitor sensitive API operations.</p><p><strong>Staff-level follow-up:</strong> Design namespace and service-account boundaries for an internal developer platform.</p><p><strong>Resource:</strong> <a href="https://devrimozcay.gumroad.com/l/decision-map">Backend Architecture Decisions OS</a>.</p><h3>29. How do you secure container image deployment?</h3><p>A secure image pipeline should address provenance, vulnerability management, immutable image references, and deployment policy enforcement.</p><p>Signing and verification can improve supply-chain integrity, but they do not guarantee that an image is free from vulnerabilities.</p><p><strong>Staff-level follow-up:</strong> Design an admission policy that rejects unapproved images while supporting emergency recovery procedures.</p><p><strong>Resource:</strong> <a href="https://devrimozcay.gumroad.com/l/kfccl">Production Deployment Checklist</a>.</p><h3>30. What are the risks of multi-tenant Kubernetes clusters?</h3><p>Different tenants may compete for CPU, memory, network capacity, storage, and shared control-plane resources. Security boundaries and operational blast radius must be considered alongside infrastructure cost.</p><p>Namespace separation alone does not provide complete isolation. Depending on trust requirements, dedicated node pools or separate clusters may be necessary.</p><p><strong>Staff-level follow-up:</strong> Define when a company should separate production workloads into multiple Kubernetes clusters.</p><p><strong>Resource:</strong> <a href="https://devrimozcay.gumroad.com/l/jwtmh">System Design Architecture Tradeoffs</a>.</p><h3>Part VII — Observability and Advanced Incident Response</h3><h3>31. A cluster looks healthy, but p99 latency is terrible. What do you inspect?</h3><p>Aggregate CPU and memory utilization cannot explain every latency problem. Examine request traces, dependency latency, connection-pool waiting, CPU throttling, network retransmissions, and garbage collection behavior.</p><p>A small subset of requests can dominate p99 latency while average response time remains acceptable. The investigation should identify the specific request paths and conditions associated with the tail.</p><p><strong>Staff-level follow-up:</strong> Design observability that distinguishes application latency from Kubernetes infrastructure latency.</p><p><strong>Resource:</strong> <a href="https://devrimozcay.gumroad.com/l/wtbklt">Backend Latency Debugging OS</a>.</p><h3>32. How do you investigate intermittent failures affecting only one node?</h3><p>Compare failing Pods by node placement and inspect node-specific metrics, kubelet logs, networking, storage behavior, and kernel events.</p><p>If failures follow a node rather than an application version, the investigation should shift toward node-level causes. Draining the node may be a useful mitigation, but the underlying cause still needs diagnosis.</p><p><strong>Staff-level follow-up:</strong> Design automated node quarantine without causing unnecessary capacity loss.</p><p><strong>Resource:</strong> <a href="https://devrimozcay.gumroad.com/l/tlrgao">Linux Production Troubleshooting OS</a>.</p><h3>33. How do you detect cascading failures across microservices?</h3><p>A slow dependency causes upstream requests to accumulate. Those requests consume threads, connections, and memory, eventually affecting otherwise healthy services.</p><p>Distributed tracing, dependency metrics, and saturation indicators can reveal the propagation path. Mitigation may require load shedding and concurrency limits rather than additional replicas.</p><p><strong>Staff-level follow-up:</strong> Design bulkheads that prevent one dependency from exhausting an entire API service.</p><p><strong>Resource:</strong> <a href="https://devrimozcay.gumroad.com/l/sycbns">Production Incident Response Playbook</a>.</p><h3>34. How do you reconstruct a Kubernetes incident timeline?</h3><p>Collect deployment events, Pod lifecycle changes, autoscaling decisions, node events, application metrics, and distributed traces. Normalize timestamps and identify which changes preceded customer impact.</p><p>A rollout occurring near an incident is evidence worth investigating, but temporal proximity alone does not establish causation.</p><p><strong>Staff-level follow-up:</strong> Design an incident timeline for a cascading outage involving HPA, PostgreSQL, and API retries.</p><p><strong>Resource:</strong> <a href="https://devrimozcay.gumroad.com/l/atbihi">Production Incident Investigation OS</a>.</p><h3>35. What makes a useful Kubernetes postmortem?</h3><p>A useful postmortem explains how the system entered the failure state, why existing protections did not prevent it, and which changes will reduce recurrence.</p><p>Avoid limiting corrective actions to adding alerts. Consider whether architectural changes, safer deployment defaults, or improved capacity controls would prevent the same failure class.</p><p><strong>Staff-level follow-up:</strong> Write a postmortem for an HPA-driven database connection storm.</p><p><strong>Resource:</strong> <a href="https://devrimozcay.gumroad.com/l/dtefzc">Production Incident Postmortem OS</a>.</p><h3>Part VIII — Staff and Principal Engineering Architecture Decisions</h3><h3>36. When should a company avoid Kubernetes?</h3><p>Kubernetes introduces control-plane operations, networking complexity, security responsibilities, and workload-management overhead. Those costs may be unjustified for small teams running a limited number of straightforward services.</p><p>The decision should reflect operational capability, deployment requirements, scale, and existing infrastructure. Choosing Kubernetes because competitors use it is not an architectural justification.</p><p><strong>Staff-level follow-up:</strong> Prepare an architecture decision record comparing Kubernetes, managed container services, and virtual machines.</p><p><strong>Resource:</strong> <a href="https://devrimozcay.gumroad.com/l/qktrrh">Architecture Decision OS</a>.</p><h3>37. How do you design a multi-region Kubernetes architecture?</h3><p>Multiple clusters across regions can improve geographic resilience, but application state, traffic routing, and dependency replication determine actual recovery behavior.</p><p>Cross-region database consistency and failover are often harder than deploying additional application replicas. Define recovery objectives and consistency guarantees before choosing routing mechanisms.</p><p><strong>Staff-level follow-up:</strong> Design a multi-region payment platform with explicit recovery-time and recovery-point objectives.</p><p><strong>Resource:</strong> <a href="https://devrimozcay.gumroad.com/l/orgnsy">Senior &amp; Staff AI System Design Interview OS</a>.</p><h3>38. How do you reduce Kubernetes infrastructure costs without reducing reliability?</h3><p>Analyze workload requests, actual utilization, scaling patterns, node fragmentation, and availability requirements.</p><p>Aggressive consolidation may reduce cost while increasing blast radius. Savings should be evaluated against capacity headroom, failure recovery, and operational risk.</p><p><strong>Staff-level follow-up:</strong> Design a cost-reduction initiative that preserves availability during a complete node-pool failure.</p><p><strong>Resource:</strong> <a href="https://devrimozcay.gumroad.com/l/jllgd">Senior to Staff Engineer Roadmap</a>.</p><h3>39. How do you introduce Kubernetes standards across 50 engineering teams?</h3><p>A platform team cannot reliably improve production engineering by publishing documentation alone. Standards need usable deployment templates, automated policy enforcement, observability, and clear ownership.</p><p>Rollout should account for existing workloads and migration risk. Measure adoption and incident outcomes rather than treating policy publication as success.</p><p><strong>Staff-level follow-up:</strong> Design a platform engineering initiative that reduces deployment-related incidents across an organization.</p><p><strong>Resource:</strong> <a href="https://devrimozcay.gumroad.com/l/gviqm">Senior &amp; Staff Engineer Promotion OS</a>.</p><h3>40. How do you demonstrate Staff-level Kubernetes expertise in an interview?</h3><p>Strong answers connect Kubernetes mechanisms to application correctness, resource economics, and failure recovery. An engineer should explain what the scheduler, networking layer, runtime, and application each guarantee, and where those guarantees stop.</p><p>A Staff-level candidate should also be able to defend tradeoffs. A system can be highly available but unnecessarily expensive, or resource-efficient but vulnerable to cascading failures. Architecture decisions become credible when their failure modes and operational costs are explicit.</p><p><strong>Staff-level follow-up:</strong> Design a Kubernetes platform supporting 200 microservices across three availability zones. Define workload isolation, database connection budgets, rollout safety, autoscaling limits, and disaster recovery before discussing implementation details.</p><p><strong>Resource:</strong> <a href="https://devrimozcay.gumroad.com/l/ifopc">Senior to Staff &amp; Principal Engineer OS</a>.</p><h3>Frequently Asked Questions</h3><h3>What are the hardest Kubernetes interview questions for Senior Engineers?</h3><p>The most difficult questions typically involve scheduling constraints, networking failures, CPU throttling, memory exhaustion, autoscaling instability, database connection storms, and safe deployments. They require understanding interactions between Kubernetes and the applications running on it.</p><h3>What should a Staff Engineer know about Kubernetes?</h3><p>A Staff Engineer should understand cluster architecture, workload isolation, production reliability, scaling constraints, security boundaries, and multi-team operational standards. The ability to reason about failure propagation and architectural tradeoffs is particularly important.</p><h3>How do you debug CrashLoopBackOff in Kubernetes?</h3><p>Inspect Pod events, current and previous container logs, exit codes, resource limits, startup behavior, and probe configuration. CrashLoopBackOff describes repeated container failures and restart backoff rather than identifying a single root cause.</p><h3>What is the difference between OOMKilled and CPU throttling?</h3><p>OOMKilled indicates that a process was terminated because of an out-of-memory condition associated with its execution environment. CPU throttling limits execution time when a container exhausts its configured CPU quota.</p><h3>How do you prepare for advanced Kubernetes interviews?</h3><p>Practice investigating realistic production incidents using resource metrics, Pod events, thread dumps, network diagnostics, and distributed traces. Be prepared to defend mitigation choices and explain how those choices affect downstream dependencies.</p><h3>Continue Learning</h3><p>I publish advanced production engineering and system design breakdowns for Senior, Staff, and Principal Engineers. You can <a href="https://substack.com/@devrimozcay1">subscribe to my Substack</a> for deeper discussions about Kubernetes, distributed systems, Java, databases, and reliability engineering.</p><p>If you want a structured starting point, <a href="https://devrimozcay.gumroad.com/l/fdiown">get the free Senior Engineer Interview &amp; Production Playbook</a>. It covers practical frameworks for production debugging and technical interview preparation.</p><p>I’m also building an AI-powered system design interview platform where engineers can practice architecture decisions and receive structured feedback. You can <a href="https://docs.google.com/forms/d/e/1FAIpQLScLD53xFU73sirxXmKFfL9-M7FGL_5mM6a8_B0qQdaYce2atA/viewform?usp=header">share your feedback and request early access to the private beta</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f183baad39fb" width="1" height="1" alt=""><hr><p><a href="https://blog.devops.dev/40-kubernetes-interview-questions-that-expose-weak-senior-engineers-f183baad39fb">40 Kubernetes Interview Questions That Expose Weak Senior Engineers</a> was originally published in <a href="https://blog.devops.dev">DevOps.dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Managing Secrets in Dockerized Applications: From GitHub Actions to Production]]></title>
            <link>https://blog.devops.dev/managing-secrets-with-docker-from-github-actions-to-the-production-server-99a5bc0cd53c?source=rss----33f8b2d9a328---4</link>
            <guid isPermaLink="false">https://medium.com/p/99a5bc0cd53c</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[github-actions]]></category>
            <category><![CDATA[aspdotnetcore]]></category>
            <category><![CDATA[management-secrets]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Vahid Faraji]]></dc:creator>
            <pubDate>Mon, 16 Nov 2026 13:26:01 GMT</pubDate>
            <atom:updated>2026-11-16T13:25:59.829Z</atom:updated>
            <content:encoded><![CDATA[<p>In one of my projects developed with ASP.NET Core, I was using Docker to run the services and GitHub Actions to manage the CI/CD process.</p><p>The deployment process was automated: with every push to the main branch, tests were executed first, then a new Docker image was built and transferred to the VPS, and finally the previous container was replaced with the new version.</p><p>While reviewing this process, I decided to make the management of sensitive production information independent of the build and deployment lifecycle as well.</p><p>It wasn’t just about securing secrets; <strong>I wanted each part of the infrastructure to have a clearly defined responsibility.</strong> Source code would be stored in Git, the Docker image would be responsible for running the application, GitHub Actions would manage the deployment process, and sensitive information such as API keys, database connection strings, and client secrets would remain in the production environment.</p><p>My goal was to ensure that each deployment changed only the application version, not the way its sensitive information was managed.</p><p>I had several options to achieve this, from environment variables and GitHub Actions Secrets to dedicated secret management services such as <strong>Azure Key Vault, HashiCorp Vault, and AWS Secrets Manager</strong>. However, considering the project’s existing architecture, which ran on a VPS with multiple Docker containers, I preferred a simpler and more manageable approach.</p><p>In the end, I chose <strong>protected files on the VPS and read-only Docker bind mounts</strong> and implemented this structure with a few limited changes to Program.cs and the GitHub Actions workflow.</p><p>In the following sections, I’ll share my experience with this change, the technical decisions I made, and the lessons I learned during implementation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wpNEydMcK_X2XUCnvnIetA.png" /></figure><h3>Step 1: Moving Secrets to the VPS</h3><p>The first thing I did was decide where to store sensitive information on the server.</p><p>I didn’t want to create separate, scattered paths for every secret. I preferred a clear structure that could also be reused for other projects in the future.</p><p>I eventually chose the following structure:</p><pre>/etc/app-secrets/<br>└── my-app/<br>    ├── external-api-key<br>    ├── database-password<br>    └── ...</pre><p>Each file contains only one sensitive value. This allows each secret to be managed and replaced independently of the others.</p><p>To create the directory and restrict access to it, I used the following commands:</p><pre>mkdir -p /etc/app-secrets/my-app<br>chmod 700 /etc/app-secrets/my-app</pre><p>I then created the required files and set their permissions to 600.</p><p>For example:</p><pre>chmod 600 /etc/app-secrets/my-app/external-api-key</pre><p>These settings restrict access to the directory and sensitive files.</p><p>At this point, the secrets were stored on the VPS, but the application still couldn’t access them directly because the services were running inside Docker containers.</p><h3>Step 2: Mounting Secrets into Docker Containers</h3><p>To make the files on the VPS accessible inside the containers, I used Docker bind mounts.</p><p>This feature allows a file or directory from the host to be made available at a specific path inside a container, without having to include it in the Docker image.</p><p>For example, the API key file was located on the VPS at:</p><pre>/etc/app-secrets/my-app/external-api-key</pre><p>I mounted it at the following path inside the container:</p><pre>/run/secrets/external-api-key</pre><p>The corresponding Docker option looked like this:</p><pre>--mount type=bind,source=/etc/app-secrets/my-app/external-api-key,target=/run/secrets/external-api-key,readonly</pre><p>The readonly option was important to me because the application only needed to read the secret, not modify it.</p><p>One of the key aspects of this design is that the files are independent of the container lifecycle.</p><p>Even if a container is removed and recreated, the original file remains on the VPS. Of course, the new container must be started with the same bind mount configuration.</p><p>My project had two independent containers: one for the UI and another for the API.</p><p>A shared API key was mounted into both containers, while information such as database connection details and credentials for external services was made available only to the API.</p><p>This separation ensured that each service had access only to the secrets it actually needed.</p><h3>Step 3: Integrating Docker Secrets into GitHub Actions</h3><p>By this point, the secret files were ready on the VPS, and I knew how to mount them into Docker. However, one important part was still missing: <strong>the automated deployment process</strong>.</p><p>In my project, GitHub Actions built a new Docker image after the tests passed and transferred it to the VPS.</p><p>The previous container was then stopped and removed, and a new container was started using docker run.</p><p>This meant that if I configured the bind mounts only manually on the server, those settings wouldn’t automatically be included when the next deployment created a new container.</p><p>To solve this, I updated the GitHub Actions workflow and added the new mounts directly to the docker run command.</p><p>A simplified example of the API configuration looks like this:</p><pre>docker run -d \<br>  --name app-api \<br>  --mount type=bind,source=/etc/app-secrets/my-app/external-api-key,target=/run/secrets/external-api-key,readonly \<br>  --mount type=bind,source=/etc/app-secrets/my-app/database-password,target=/run/secrets/database-password,readonly \<br>  my-app-api:latest</pre><p>This code snippet shows only the part related to secrets; the other mounts and service settings were preserved in the original workflow.</p><p>What I found interesting was that <strong>I didn’t need to change the CI/CD architecture; I only needed to extend how the new container was started.</strong></p><p>From that point on, whenever GitHub Actions deployed a new version of the application, the new container would start with access to the same secret files already stored on the VPS.</p><p>This allowed the deployment process to remain fully automated without placing sensitive values in the workflow or Docker image.</p><h3>Step 4: Reading Secrets in ASP.NET Core</h3><p>Now it was time to update the application.</p><p>I wanted to keep this change as small as possible and avoid rewriting the services that already relied on configuration values.</p><p>Fortunately, ASP.NET Core provides this flexibility through IConfiguration.</p><p>In Program.cs, for the Production environment, I read the secret values from the files inside the container and added them to the configuration.</p><p>For example, the following code is a simplified version of this approach:</p><pre>if (builder.Environment.IsProduction())<br>{<br>    builder.Configuration[&quot;ExternalApi:Key&quot;] =<br>        File.ReadAllText(&quot;/run/secrets/external-api-key&quot;)<br>            .TrimEnd(&#39;\r&#39;, &#39;\n&#39;);<br>    // ... other secrets<br>}</pre><p>With this approach, the rest of the application continued using the same configuration keys as before.</p><p>For example, a service using ExternalApi:Key didn&#39;t need to know whether the API key came from a JSON file or a file mounted inside Docker.</p><p>In the Development environment, I also used the built-in .NET User Secrets feature so that production-specific files were only required on the server.</p><p><strong>The result was that the configuration source changed, but the contract for accessing configuration values remained the same.</strong></p><p>This was exactly what I wanted from the beginning: a change in how the infrastructure was managed, with minimal impact on the application logic.</p><h3>Step 5: Testing and Validating the Changes</h3><p>After making the changes, it was time to verify that everything worked as expected.</p><p>I didn’t want to rely solely on the configuration looking correct. It was important to me to make sure the containers could actually access the secret files and that the application could run without issues.</p><p>First, I checked that the files existed on the VPS and verified their permissions. Then, using Docker, I confirmed that the mounted files were readable inside the container without having to display their sensitive values.</p><p>In the Development environment, I also moved the settings to .NET User Secrets and adjusted the integration tests using test values and replacement dependencies so that the tests wouldn&#39;t rely on real production credentials.</p><p>One important distinction I paid attention to during this stage was the difference between a successful pipeline and a healthy running application.</p><p><strong>A green GitHub Actions workflow doesn’t necessarily mean the application is running correctly in production.</strong></p><p>A Docker image may build successfully, and even the container creation command may execute without errors, while the application still fails during startup because a secret is missing, a mount path is incorrect, or the file cannot be accessed.</p><p>That’s why I considered checking the container status, application logs, and actual service behavior after deployment to be an important part of the validation process.</p><h3>Step 6: What I Learned from This Experience</h3><p>This change didn’t involve a large amount of code, but it made me pay closer attention to several aspects of the deployment architecture.</p><p><strong>First, keeping secrets independent of Docker images.</strong></p><p>A Docker image should be reusable across different environments without carrying sensitive information specific to any one environment. With this structure, the image stays the same, and each environment provides its own secrets.</p><p><strong>Second, restricting access.</strong></p><p>Using a bind mount alone isn’t enough. File permissions on the VPS, system user access, and which secrets each container can access are all important. The readonly option also helps prevent the application from modifying the mounted file.</p><p><strong>Third, the importance of backup and recovery.</strong></p><p>When secrets are stored outside the Docker image, there must also be a clear plan for preserving and recovering them.</p><p>If the server is lost, having the source code and Docker image isn’t enough to bring the system back online. The secret files, or a secure way to recreate the credentials, are also required.</p><p><strong>Fourth, managing secret changes.</strong></p><p>In my implementation, secret values are read during startup and added to IConfiguration. Therefore, if a file&#39;s value changes, the corresponding container must be restarted for the new value to take effect.</p><p>This may seem like a small detail, but it matters when designing a credential rotation process.</p><h3>Conclusion</h3><p>My goal with this change wasn’t to build a complex secret management system. I wanted a solution that was simple, manageable, and maintainable within the project’s existing architecture.</p><p>By using protected files on the VPS, read-only Docker bind mounts, and a few limited changes to ASP.NET Core and GitHub Actions, I was able to separate the management of sensitive production information from the Docker image and build lifecycle.</p><p>Of course, this approach doesn’t replace all the capabilities of a dedicated secret manager. Areas such as secure backups, access control, auditing, and credential rotation still require attention and ongoing management.</p><p>But the most important outcome of this experience for me was <strong>a clearer separation of responsibilities across the infrastructure</strong>.</p><p>Source code is managed in Git, build and deployment processes in GitHub Actions, application execution in Docker, and sensitive production information in the server environment.</p><p>Sometimes, improving an architecture doesn’t require adding new tools; it simply requires separating responsibilities more clearly.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=99a5bc0cd53c" width="1" height="1" alt=""><hr><p><a href="https://blog.devops.dev/managing-secrets-with-docker-from-github-actions-to-the-production-server-99a5bc0cd53c">Managing Secrets in Dockerized Applications: From GitHub Actions to Production</a> was originally published in <a href="https://blog.devops.dev">DevOps.dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Python 3.15 Lazy Imports Move the Crash From Deploy to First Request]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://blog.devops.dev/python-3-15-lazy-imports-move-the-crash-from-deploy-to-first-request-5489e98b992c?source=rss----33f8b2d9a328---4"><img src="https://cdn-images-1.medium.com/max/1600/1*NyNlzPp8a4bSBT-adD0WTw.png" width="1600"></a></p><p class="medium-feed-snippet">What the interpreter binds, when it reifies, and the boot check that puts the ImportError back where you can see it.</p><p class="medium-feed-link"><a href="https://blog.devops.dev/python-3-15-lazy-imports-move-the-crash-from-deploy-to-first-request-5489e98b992c?source=rss----33f8b2d9a328---4">Continue reading on DevOps.dev »</a></p></div>]]></description>
            <link>https://blog.devops.dev/python-3-15-lazy-imports-move-the-crash-from-deploy-to-first-request-5489e98b992c?source=rss----33f8b2d9a328---4</link>
            <guid isPermaLink="false">https://medium.com/p/5489e98b992c</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Daniel Valev]]></dc:creator>
            <pubDate>Mon, 16 Nov 2026 13:25:55 GMT</pubDate>
            <atom:updated>2026-11-16T13:25:54.107Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Every AKS Node You Can Patch in Place Is a Node That Can Drift]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://blog.devops.dev/aks-node-drift-azure-container-linux-immutable-os-90df033e23ac?source=rss----33f8b2d9a328---4"><img src="https://cdn-images-1.medium.com/max/1536/1*lr5C05bRnhbJfF26M-YYbg.png" width="1536"></a></p><p class="medium-feed-snippet">How Azure Container Linux locks down the node OS layer by layer, and how to move off OS Guard before its patches end December 10</p><p class="medium-feed-link"><a href="https://blog.devops.dev/aks-node-drift-azure-container-linux-immutable-os-90df033e23ac?source=rss----33f8b2d9a328---4">Continue reading on DevOps.dev »</a></p></div>]]></description>
            <link>https://blog.devops.dev/aks-node-drift-azure-container-linux-immutable-os-90df033e23ac?source=rss----33f8b2d9a328---4</link>
            <guid isPermaLink="false">https://medium.com/p/90df033e23ac</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[cybersecurity]]></category>
            <dc:creator><![CDATA[Dave R | Microsoft Azure & AI MVP ☁️]]></dc:creator>
            <pubDate>Mon, 16 Nov 2026 13:25:26 GMT</pubDate>
            <atom:updated>2026-11-16T13:25:25.136Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[ Your Kubernetes Knowledge Is Becoming Cheaper. Your Production Judgment Isn’t.]]></title>
            <link>https://blog.devops.dev/your-kubernetes-knowledge-is-becoming-cheaper-your-production-judgment-isnt-cabf2adfc61a?source=rss----33f8b2d9a328---4</link>
            <guid isPermaLink="false">https://medium.com/p/cabf2adfc61a</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Arvind Verma]]></dc:creator>
            <pubDate>Mon, 16 Nov 2026 13:25:19 GMT</pubDate>
            <atom:updated>2026-11-16T13:25:18.053Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>There was a time when knowing how to write Kubernetes YAML felt like a superpower.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ThO1sqQSmNMmlmhBHe6M-Q.png" /></figure><p>You knew Deployments.</p><p>You knew Services.</p><p>You knew ConfigMaps.</p><p>You knew Ingress.</p><p>You knew how to write a Helm chart without accidentally deleting half the cluster.</p><p>You learned Terraform.</p><p>You learned Jenkins.</p><p>You learned Bash.</p><p>You learned Prometheus queries.</p><p>You spent years collecting tools, commands, syntax, and implementation patterns.</p><p>And then AI showed up.</p><p>Now you can describe what you want in plain English and get a Terraform module.</p><p>Ask for a Kubernetes Deployment and you get YAML.</p><p>Ask for a CI/CD pipeline and you get a pipeline.</p><p>Ask for a Bash script and you get one.</p><p>Ask for a PromQL query and, assuming the machines haven’t collectively decided to punish humanity that day, you get something reasonably useful. 😄</p><p>So here’s the uncomfortable question:</p><blockquote><strong><em>If AI can increasingly handle implementation, what exactly becomes more valuable for a DevOps engineer?</em></strong></blockquote><p>I think the answer is simple.</p><p><strong>Judgment.</strong></p><p>Not knowing more commands.</p><p>Not memorizing more YAML.</p><p>Not collecting another 47 certifications.</p><p><strong>Knowing what to do when the obvious answer doesn’t work.</strong></p><p>And production has a beautiful way of ensuring that the obvious answer almost never works.</p><h3>1️⃣ The Shift: From Tool Operator to Decision Maker</h3><p>For years, DevOps careers were heavily associated with tools.</p><p>Learn AWS.</p><p>Learn Docker.</p><p>Learn Kubernetes.</p><p>Learn Terraform.</p><p>Learn Jenkins.</p><p>Learn Ansible.</p><p>Learn Prometheus.</p><p>Learn Grafana.</p><p>Learn Git.</p><p>Learn Linux.</p><p>And yes, these skills still matter.</p><p>The tools aren’t suddenly useless because AI can generate configuration for them.</p><p>But something important is changing.</p><p>The cost of producing configuration is falling.</p><p>Consider a simple Kubernetes deployment.</p><p>Previously, an engineer might spend time writing:</p><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: payment-api<br>spec:<br>  replicas: 3<br>  selector:<br>    matchLabels:<br>      app: payment-api<br>  template:<br>    metadata:<br>      labels:<br>        app: payment-api<br>    spec:<br>      containers:<br>      - name: payment-api<br>        image: payment-api:v1.4.2<br>        ports:<br>        - containerPort: 8080</pre><p>Today, AI can generate something similar almost instantly.</p><p>The YAML isn’t the difficult part anymore.</p><p>The difficult questions are:</p><p><strong>Should this application have three replicas?</strong></p><p><strong>What happens when traffic increases 10x?</strong></p><p><strong>Should the Pods be distributed across availability zones?</strong></p><p><strong>What are the CPU and memory requests?</strong></p><p><strong>What happens if the database becomes unavailable?</strong></p><p><strong>Should the application retry?</strong></p><p><strong>Could those retries create a retry storm?</strong></p><p><strong>What happens during a rolling deployment?</strong></p><p><strong>What’s the blast radius if this configuration is wrong?</strong></p><p>That’s engineering judgment.</p><p>AI can help generate the implementation.</p><p>You still need to understand whether the implementation makes sense.</p><p>And that distinction is going to become increasingly important.</p><h3>🧠 2️⃣ Production Doesn’t Give You Clean Problems</h3><p>Training environments are wonderfully polite.</p><p>They give you an error.</p><p>You read the error.</p><p>You fix the error.</p><p>Everything works.</p><p>Production doesn’t behave like that.</p><p>Production gives you this:</p><pre>CPU:              42%<br>Memory:           51%<br>Pods:             Healthy<br>Nodes:            Healthy<br>HTTP 5xx:         Increasing<br>Latency:          4.8 seconds<br>Database:         Healthy<br>Load Balancer:    Healthy<br>Customer:         &quot;THE APPLICATION IS DOWN&quot;</pre><p>Congratulations.</p><p>You now have a problem.</p><p>And approximately 37 possible explanations.</p><p>Maybe it’s the application.</p><p>Maybe it’s the database.</p><p>Maybe it’s DNS.</p><p>Maybe it’s a network policy.</p><p>Maybe it’s connection exhaustion.</p><p>Maybe it’s a downstream dependency.</p><p>Maybe it’s a load balancer issue.</p><p>Maybe it’s a recent deployment.</p><p>Maybe it’s an innocent-looking configuration change from two hours ago.</p><p>Maybe the dashboards are telling you half the truth.</p><p>This is where experienced engineers separate themselves.</p><p>Not because they know 500 commands.</p><p>But because they know <strong>which question to ask first</strong>.</p><h3>🔍 3️⃣ Skill #1: Deciding What to Investigate First</h3><p>One of the most valuable DevOps skills is prioritization.</p><p>Imagine your production application suddenly becomes slow.</p><p>You could run:</p><pre>kubectl get pods<br>kubectl get nodes<br>kubectl top pods<br>kubectl top nodes<br>kubectl describe pod<br>kubectl get events<br>kubectl get svc<br>kubectl get endpoints<br>kubectl get ingress</pre><p>You could inspect Grafana.</p><p>Then CloudWatch.</p><p>Then application logs.</p><p>Then database metrics.</p><p>Then network metrics.</p><p>You can spend an hour collecting information.</p><p>Or you can ask a better question:</p><blockquote><strong><em>What evidence will reduce the number of possible causes the fastest?</em></strong></blockquote><p>That’s a very different mindset.</p><p>Suppose customers in only one region are experiencing latency.</p><p>Immediately, the investigation changes.</p><p>You don’t need to treat the entire global platform as equally suspicious.</p><p>Suppose only one API endpoint is slow.</p><p>Now you can narrow the investigation to that request path.</p><p>Suppose latency increased immediately after a deployment.</p><p>Now the deployment becomes a high-value hypothesis.</p><p>This is troubleshooting as <strong>hypothesis reduction</strong>.</p><p>You start with:</p><pre>50 possible causes</pre><p>Then eliminate possibilities:</p><pre>50<br> ↓<br>20<br> ↓<br>8<br> ↓<br>3<br> ↓<br>1 likely root cause</pre><p>That’s much more valuable than simply knowing more commands.</p><h3>🌍 4️⃣ Skill #2: Understanding Blast Radius</h3><p>Here’s a question experienced engineers ask early:</p><blockquote><strong><em>How much of the system is actually affected?</em></strong></blockquote><p>One Pod?</p><p>One Deployment?</p><p>One namespace?</p><p>One node?</p><p>One availability zone?</p><p>One region?</p><p>Every customer?</p><p>Only a particular customer segment?</p><p>This is called understanding the <strong>blast radius</strong>.</p><p>Imagine a Kubernetes application running across three availability zones.</p><p>Suddenly, some Pods become unhealthy.</p><p>You could immediately restart them.</p><p>But first ask:</p><p><strong>Where are the failing Pods located?</strong></p><p>If they’re concentrated in one AZ, that’s valuable information.</p><p>If they’re spread evenly across all three AZs, that’s a different story.</p><p>If only one node is affected, that’s another.</p><p>If every application in the cluster is affected, you’ve moved into a completely different investigation.</p><p>The same symptom can have completely different meanings depending on its scope.</p><h3>🧩 Think in Layers</h3><p>A useful way to visualize production troubleshooting is:</p><pre>User<br>  ↓<br>DNS<br>  ↓<br>CDN / Load Balancer<br>  ↓<br>Ingress<br>  ↓<br>Service<br>  ↓<br>Pod<br>  ↓<br>Application<br>  ↓<br>Database / Dependency<br>  ↓<br>External Service</pre><p>When something breaks, don’t randomly inspect everything.</p><p>Ask:</p><blockquote><strong><em>At which layer does the failure begin?</em></strong></blockquote><p>If DNS is failing, Kubernetes Pods may be perfectly healthy.</p><p>If the Service has no endpoints, the application might be perfectly healthy.</p><p>If the application is returning 500 errors, the network may be completely fine.</p><p>If the application is waiting 10 seconds for a database response, increasing Kubernetes CPU limits isn’t going to magically fix the database.</p><p>This sounds obvious.</p><p>During a 2 AM incident, apparently it becomes advanced mathematics. 😄</p><h3>📊 5️⃣ Skill #3: Challenging Your Telemetry</h3><p>One of the most dangerous assumptions in operations is:</p><blockquote><strong><em>“The dashboard says everything is healthy, so everything must be healthy.”</em></strong></blockquote><p>Not necessarily.</p><p>A dashboard is a representation of reality.</p><p>It isn’t reality itself.</p><p>Imagine:</p><pre>Grafana:<br>CPU       🟢<br>Memory    🟢<br>Pods      🟢<br>Nodes     🟢</pre><p>Meanwhile customers are saying:</p><blockquote><em>“The application is extremely slow.”</em></blockquote><p>Who is wrong?</p><p>Possibly neither.</p><p>Your infrastructure metrics could genuinely be healthy while the user experience is broken.</p><p>Maybe:</p><ul><li>A downstream API is slow.</li><li>Database queries are degrading.</li><li>DNS resolution is intermittent.</li><li>One endpoint has a problem.</li><li>Connection pools are exhausted.</li><li>A particular customer workflow is failing.</li><li>Your monitoring isn’t measuring the actual failure.</li></ul><p>This is why strong engineers don’t simply <strong>read dashboards</strong>.</p><p>They <strong>challenge what the dashboards are telling them</strong>.</p><h3>🔥 6️⃣ The Difference Between System Health and User Experience</h3><p>Consider an application with:</p><pre>CPU: 30%<br>Memory: 45%<br>Pod Restarts: 0<br>Node Health: Good</pre><p>Looks healthy.</p><p>Now consider:</p><pre>P95 latency: 8 seconds<br>Checkout failures: 14%<br>Payment API timeout: 23%</pre><p>Suddenly the picture changes.</p><p>Infrastructure health is not the same thing as application health.</p><p>And application health is not necessarily the same thing as <strong>user experience</strong>.</p><p>A mature DevOps engineer thinks across all three.</p><pre>Infrastructure<br>      ↓<br>Application<br>      ↓<br>User Experience</pre><p>That’s why observability is not simply about having more dashboards.</p><p>It’s about having the <strong>right signals</strong>.</p><h3>⚠️ 7️⃣ Skill #4: Making Risky Decisions Carefully</h3><p>Production gives you several tempting buttons.</p><p>Restart.</p><p>Rollback.</p><p>Failover.</p><p>Scale.</p><p>Drain.</p><p>Delete.</p><p>Disable.</p><p>Sometimes the fastest solution is also the most dangerous one.</p><p>Imagine a Kubernetes node is behaving strangely.</p><p>You could immediately:</p><pre>kubectl drain &lt;node&gt;</pre><p>But first ask:</p><ul><li>How many workloads are running there?</li><li>Are there enough healthy nodes?</li><li>Are PodDisruptionBudgets configured?</li><li>Are there stateful workloads?</li><li>Is the problem isolated to this node?</li><li>Will draining create another capacity problem?</li></ul><p>Similarly, suppose a new deployment appears to be causing errors.</p><p>You could roll it back.</p><p>But first determine:</p><ul><li>Is the deployment actually responsible?</li><li>Did the issue begin at the same time?</li><li>Are only new Pods affected?</li><li>Could the dependency have failed independently?</li><li>Would rollback remove an unrelated security fix?</li></ul><p>The point isn’t to avoid action.</p><p>The point is to understand the <strong>consequence of action before taking it</strong>.</p><h3>🚨 8️⃣ AI Can Suggest Five Actions. You Own the Sixth Question.</h3><p>Imagine an AI assistant tells you:</p><blockquote><em>“Restart the Pods.”</em></blockquote><p>Another suggestion says:</p><blockquote><em>“Rollback the deployment.”</em></blockquote><p>Another says:</p><blockquote><em>“Scale the application.”</em></blockquote><p>Another says:</p><blockquote><em>“Fail over to another region.”</em></blockquote><p>All four might be technically reasonable.</p><p>But which one is safest?</p><p>That’s not merely a syntax problem.</p><p>It’s a decision problem.</p><p>And the engineer making that decision owns the consequence.</p><p>This is why AI doesn’t eliminate the need for senior engineers.</p><p>In many cases, it increases the importance of them.</p><p>Someone has to decide:</p><p><strong>What should we do?</strong></p><p><strong>What shouldn’t we do?</strong></p><p><strong>What information are we missing?</strong></p><p><strong>What is the safest reversible action?</strong></p><p><strong>What happens if our hypothesis is wrong?</strong></p><p>That’s engineering judgment.</p><h3>🧠 9️⃣ Skill #5: Explaining Chaos Clearly</h3><p>Technical ability isn’t enough during a major incident.</p><p>Communication matters.</p><p>Imagine a SEV-1 incident.</p><p>Ten engineers join the bridge.</p><p>Everyone starts talking.</p><p>Someone says:</p><blockquote><em>“I think it’s DNS.”</em></blockquote><p>Another says:</p><blockquote><em>“Maybe it’s the database.”</em></blockquote><p>Someone else starts restarting Pods.</p><p>Another engineer is looking at Grafana.</p><p>The incident gets noisier.</p><p>A strong incident commander can bring structure to the chaos.</p><p>They might say:</p><blockquote><strong><em>“Here’s what we know.”</em></strong></blockquote><blockquote><strong><em>“These customers are affected.”</em></strong></blockquote><blockquote><strong><em>“The issue started at 14:07 UTC.”</em></strong></blockquote><blockquote><strong><em>“Our current hypothesis is increased latency from the payment dependency.”</em></strong></blockquote><blockquote><strong><em>“We’re testing that hypothesis by checking request latency and dependency error rates.”</em></strong></blockquote><blockquote><strong><em>“We’re not restarting workloads yet because the Pods themselves appear healthy.”</em></strong></blockquote><p>That’s powerful.</p><p>Not because it’s complicated.</p><p>Because it’s clear.</p><h3>🗣️ The Four Sentences Every Engineer Should Be Able to Say</h3><p>During an incident, practice communicating in this format:</p><h3>1. What is impacted?</h3><blockquote><em>“Checkout requests are failing for customers in one region.”</em></blockquote><h3>2. What is our current hypothesis?</h3><blockquote><em>“Our current hypothesis is connectivity between the application and payment dependency.”</em></blockquote><h3>3. What are we testing?</h3><blockquote><em>“We’re checking request latency, connection errors, and network path from affected Pods.”</em></blockquote><h3>4. What is the safest next action?</h3><blockquote><em>“We’ll validate the dependency path before making any workload changes.”</em></blockquote><p>This turns chaos into a process.</p><p>And that process is extremely difficult to automate completely because it requires context, judgment, risk assessment, and communication.</p><h3>🤖 So, What Should DevOps Engineers Learn in the AI Era?</h3><p>This doesn’t mean you should stop learning Kubernetes.</p><p>Please don’t interpret this article as permission to abandon Kubernetes and become a motivational speaker on LinkedIn.</p><p>You still need technical depth.</p><p>But the learning priorities should evolve.</p><p>Don’t only learn:</p><blockquote><strong><em>How does this tool work?</em></strong></blockquote><p>Also learn:</p><blockquote><strong><em>Why would I use it?</em></strong></blockquote><p>Then:</p><blockquote><strong><em>What can fail?</em></strong></blockquote><p>Then:</p><blockquote><strong><em>How would I detect that failure?</em></strong></blockquote><p>Then:</p><blockquote><strong><em>What is the blast radius?</em></strong></blockquote><p>Then:</p><blockquote><strong><em>What’s the safest remediation?</em></strong></blockquote><p>Then:</p><blockquote><strong><em>How would I explain the incident to another engineer?</em></strong></blockquote><p>That’s the progression from <strong>tool knowledge to engineering judgment</strong>.</p><h3>🏗️ Build Skills Around Failure, Not Just Success</h3><p>When learning Kubernetes, don’t stop after:</p><blockquote><em>“I deployed an application.”</em></blockquote><p>Ask:</p><p><strong>What happens if the Pod can’t schedule?</strong></p><p><strong>What happens if the image can’t be pulled?</strong></p><p><strong>What happens if DNS fails?</strong></p><p><strong>What happens if the Service has no endpoints?</strong></p><p><strong>What happens if the node becomes NotReady?</strong></p><p><strong>What happens if the deployment gets stuck?</strong></p><p><strong>What happens if the application can’t reach the database?</strong></p><p><strong>What happens if one AZ fails?</strong></p><p><strong>What happens if the monitoring system itself is wrong?</strong></p><p>This is where your engineering maturity develops.</p><h3>🚀 The DevOps Engineer of the AI Era</h3><p>The future DevOps engineer isn’t necessarily the person who can write the most YAML.</p><p>It’s the engineer who can look at a messy production environment and say:</p><blockquote><strong><em>“Here’s what I know.”</em></strong></blockquote><blockquote><strong><em>“Here’s what I don’t know.”</em></strong></blockquote><blockquote><strong><em>“Here’s the most likely failure domain.”</em></strong></blockquote><blockquote><strong><em>“Here’s the evidence supporting that hypothesis.”</em></strong></blockquote><blockquote><strong><em>“Here’s the safest thing we can test next.”</em></strong></blockquote><blockquote><strong><em>“Here’s the potential blast radius if we’re wrong.”</em></strong></blockquote><p>That is a very different skill set from simply knowing commands.</p><p>And ironically, the more AI improves at generating implementation, the more valuable these human engineering skills become.</p><h3>🎯 The New DevOps Skill Stack</h3><p>Think of your career as five layers:</p><pre>🧠 JUDGMENT<br>          Decision Making<br>                 ↑<br>          🔍 TROUBLESHOOTING<br>        Root Cause Analysis<br>                 ↑<br>          📊 OBSERVABILITY<br>       Metrics • Logs • Traces<br>                 ↑<br>          ⚙️ AUTOMATION<br>      Terraform • CI/CD • IaC<br>                 ↑<br>          ☸️ TECHNOLOGY<br>   AWS • Kubernetes • Linux • Git</pre><p>AI is increasingly helping with the lower layers.</p><p>That doesn’t make those layers irrelevant.</p><p>It means you should keep building upward.</p><p>Because the engineer who understands both the technology <strong>and</strong> the consequences of changing it will always have an advantage over someone who can only generate configuration.</p><h3>💡 Final Thought</h3><p>Your Kubernetes YAML may become cheaper.</p><p>Your Terraform code may become faster to produce.</p><p>Your CI/CD pipeline may take minutes instead of hours.</p><p>Your Bash script may be generated before you finish typing the prompt.</p><p>That’s not necessarily bad news.</p><p>It means engineers can spend less time writing repetitive configuration and more time thinking about the systems they’re building.</p><p>The real career risk isn’t:</p><blockquote><strong><em>“AI can write Kubernetes YAML.”</em></strong></blockquote><p>The real risk is:</p><blockquote><strong><em>“I only know how to write Kubernetes YAML.”</em></strong></blockquote><p>Because production doesn’t care how quickly you can generate a manifest.</p><p>It cares whether you can determine <strong>why the system is failing, what is affected, what should happen next, and what could go wrong if you’re wrong.</strong></p><p>That is the part of DevOps that gets harder to automate.</p><p>And probably the part worth getting very good at. 🚀</p><h3>🔑 Key Takeaways</h3><p><strong>1.</strong> AI is reducing the cost of implementation.</p><p><strong>2.</strong> Production judgment remains highly valuable.</p><p><strong>3.</strong> Learn to prioritize investigations instead of running commands randomly.</p><p><strong>4.</strong> Always understand the blast radius before taking action.</p><p><strong>5.</strong> Treat dashboards as evidence, not absolute truth.</p><p><strong>6.</strong> Understand the difference between infrastructure health and user experience.</p><p><strong>7.</strong> Prefer safe, reversible actions when uncertainty is high.</p><p><strong>8.</strong> During incidents, communicate impact, hypothesis, evidence, and next action clearly.</p><p><strong>9.</strong> Learn failure modes, not just happy paths.</p><p><strong>10.</strong> Don’t compete with AI at generating YAML. Become better at deciding what the YAML should actually do.</p><h3>🔖Hashtags</h3><p>#DevOps #Kubernetes #AI #AIOps #CloudComputing #SRE #PlatformEngineering #DevOpsEngineer #KubernetesEngineer #AWS #Terraform #CloudNative #Observability #IncidentManagement #ProductionEngineering #SiteReliabilityEngineering #DevSecOps #Automation #TechCareer #Engineering</p><h3>👋 Follow Me</h3><p>If you enjoyed this article and would like more practical DevOps, Cloud, Kubernetes, AWS, Terraform, CI/CD, and Software Engineering content, feel free to connect with me on LinkedIn:</p><p><strong>LinkedIn:</strong><br><a href="https://www.linkedin.com/in/arvindverma021/">https://www.linkedin.com/in/arvindverma021/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cabf2adfc61a" width="1" height="1" alt=""><hr><p><a href="https://blog.devops.dev/your-kubernetes-knowledge-is-becoming-cheaper-your-production-judgment-isnt-cabf2adfc61a">🚨 Your Kubernetes Knowledge Is Becoming Cheaper. Your Production Judgment Isn’t.</a> was originally published in <a href="https://blog.devops.dev">DevOps.dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Build Multi-Account AWS Observability with CloudWatch OAM and Grafana]]></title>
            <link>https://blog.devops.dev/how-to-build-multi-account-aws-observability-with-cloudwatch-oam-and-grafana-00d6a1f10768?source=rss----33f8b2d9a328---4</link>
            <guid isPermaLink="false">https://medium.com/p/00d6a1f10768</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[cloudwatch]]></category>
            <category><![CDATA[logging-and-monitoring]]></category>
            <dc:creator><![CDATA[Anil Raj Rimal]]></dc:creator>
            <pubDate>Mon, 16 Nov 2026 13:25:14 GMT</pubDate>
            <atom:updated>2026-11-16T13:25:13.539Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>A practical Terraform guide: one sink, one link per account, one Grafana workspace</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b89Nc_S9eRQXbAJcww3ZKA.png" /></figure><p>If you run workloads across several AWS accounts, you know the routine. Something breaks, you log into the right account, open the CloudWatch console, and read log groups one by one. Need to compare two accounts? Open two browser sessions.</p><p>There is no single view, and it gets worse with every account you add.</p><p>This guide shows how to fix that with <strong>CloudWatch Observability Access Manager (OAM)</strong>, <strong>Amazon Managed Grafana</strong> and <strong>Terraform</strong>, plus the gotchas worth knowing before you start.</p><p><strong>What is CloudWatch OAM?</strong></p><p>OAM is AWS’s built-in <strong>cross-account observability</strong> feature. It has two parts:</p><p>- <strong>Sink:</strong> lives in the <strong>monitoring account</strong>, the one that should see everything.</p><p>- <strong>Link:</strong> lives in each <strong>source account</strong> and shares its data with the sink.</p><p>Once linked, CloudWatch in the monitoring account can query the source account’s metrics and logs directly. You pass the account ID in the query, and AWS handles the rest.</p><p><strong>Why not one Grafana data source per account?</strong></p><p>The common alternative is a read-only IAM role in every account and one Grafana data source per account. It works, but every new account means a new role, trust policy and data source to maintain.</p><p>With OAM, a source account needs <strong>one resource</strong>, and Grafana uses <strong>one IAM role in one account</strong>.</p><p><strong>Architecture</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ynBuO3w5xCuWIQU3A4q51w.png" /></figure><p>- <strong>Monitoring account:</strong> holds the OAM sink and the Grafana workspace.</p><p>- <strong>Source accounts:</strong> one OAM link each, sharing <strong>metrics</strong> and <strong>log groups</strong>.</p><p>- <strong>Grafana:</strong> users sign in with IAM Identity Center. Grafana queries CloudWatch in the monitoring account, and OAM serves data from the linked accounts.</p><p>There are <strong>no cross-account IAM roles</strong> in the read path. The only cross-account trust is the sink policy.</p><p><strong>Step 1: Create the Sink</strong></p><p>In the monitoring account, create the sink and a policy that lets accounts in your <strong>AWS Organization</strong> link to it, only for metrics and logs:</p><pre>resource &quot;aws_oam_sink&quot; &quot;this&quot; {<br>  name = &quot;example-monitoring-sink&quot;<br>}<br><br>resource &quot;aws_oam_sink_policy&quot; &quot;this&quot; {<br>  sink_identifier = aws_oam_sink.this.arn<br><br>  policy = jsonencode({<br>    Version = &quot;2012-10-17&quot;<br>    Statement = [{<br>      Effect    = &quot;Allow&quot;<br>      Principal = { AWS = &quot;*&quot; }<br>      Action    = [&quot;oam:CreateLink&quot;, &quot;oam:UpdateLink&quot;]<br>      Resource  = &quot;*&quot;<br>      Condition = {<br>        StringEquals = { &quot;aws:PrincipalOrgID&quot; = var.org_id }<br>        &quot;ForAllValues:StringEquals&quot; = {<br>          &quot;oam:ResourceTypes&quot; = [&quot;AWS::CloudWatch::Metric&quot;, &quot;AWS::Logs::LogGroup&quot;]<br>        }<br>      }<br>    }]<br>  })<br>}</pre><p>Scoping by Organization ID means new accounts can link without changing the sink. If you want tighter control, list specific account IDs instead.</p><p><strong>Step 2: Create a Link in Each Source Account</strong></p><pre>resource &quot;aws_oam_link&quot; &quot;this&quot; {<br>  label_template  = &quot;$AccountName&quot;<br>  resource_types  = [&quot;AWS::CloudWatch::Metric&quot;, &quot;AWS::Logs::LogGroup&quot;]<br>  sink_identifier = var.sink_arn<br>}</pre><p>That is the entire footprint in a source account.</p><p>If you keep a separate Terraform state per account, Terraform cannot see the dependency between the sink and the links. Apply the sink first, then pass its ARN to the links. Storing it in <strong>SSM Parameter Store</strong> or reading it from remote state is safer than copying it by hand.</p><p><strong>Step 3: Set Up Amazon Managed Grafana</strong></p><p>Create the workspace with CURRENT_ACCOUNTaccess, since OAM already handles cross-account access, and use IAM Identity Center for login.</p><p><strong>Write the IAM role policy yourself.</strong> Do not assume the SERVICE_MANAGEDpermission type will attach everything you need. Attach Grafana’s documented CloudWatch policy explicitly, and add:</p><p>- oam:ListSinks</p><p>- oam:ListAttachedLinks</p><p>These let Grafana discover linked accounts, and they are not part of a standard CloudWatch read policy.</p><p><strong>Step 4: Build Cross-Account Dashboards</strong></p><p><strong>Put the account ID on every query.</strong> Without it, CloudWatch returns the monitoring account’s own data. Add an account dropdown variable and use it everywhere.</p><p><strong>Use full log group ARNs for Logs Insights.</strong> A log group name alone is not enough across accounts:</p><pre>{<br>  &quot;accountId&quot;: &quot;$account&quot;,<br>  &quot;logGroups&quot;: [<br>    { &quot;arn&quot;: &quot;arn:aws:logs:&lt;region&gt;:$account:log-group:/aws/lambda/example-fn:*&quot; }<br>  ]<br>}</pre><p><strong>Do not hardcode the data source UID.</strong> Use a data source variable resolved by type cloudwatchso your dashboard JSON works in any workspace.</p><p><strong>Step 5: Manage Dashboards as Code (and Rotate the Token)</strong></p><p>You can manage folders and dashboards with the <strong>Grafana Terraform provider</strong>. That provider does not use IAM. It uses a <strong>Grafana service account token</strong>, and in Amazon Managed Grafana those tokens last <strong>30 days</strong> at most.</p><p>To keep deployments working, automate the renewal:</p><p>1. Store the token in <strong>AWS</strong> <strong>Secrets Manager or any secrets holder</strong>.</p><p>2. Run a <strong>Lambda</strong> on a schedule shorter than 30 days.</p><p>3. The Lambda creates a new token through the AWS Grafana API (CreateWorkspaceServiceAccountToken), saves it to the secret, and only then deletes the old one.</p><p>4. Set a CloudWatch alarm on the Lambda’s Errors metric and send it to <strong>SNS</strong>.</p><p>5. Have your CI pipeline read the secret on every run.</p><p>Two things to get right in the Lambda:</p><p>- <strong>Raise exceptions on failure.</strong> If the function catches errors and returns normally, Lambda counts it as a success, and your alarm never fires.</p><p>- <strong>If you use Secrets Manager rotation as the trigger,</strong> it calls the function once per rotation step (createSecret, setSecret, testSecret, finishSecret). Either handle each step, or use <strong>EventBridge Scheduler </strong>for a simpler single call.</p><p>Remember, this token is only for deploying dashboards. If rotation fails, dashboards keep working; only deployments stop.</p><p><strong>Gotchas Worth Knowing</strong></p><p><strong>Sink policy details matter.</strong> Common mistakes: using the sink ARN as Resource instead of &quot;*”, using a bare &quot;*”principal instead of {AWS = &quot;*&quot;}, and missing oam:UpdateLink.</p><p><strong>Workspace creation can fail on retry.</strong> If a retry returns ConflictException: Duplicate requesteven though no workspace exists, the API is deduplicating on request contents. A unique description, such as description = uuid()with lifecycle { ignore_changes = [description]}, gets around it.</p><p><strong>Your deployment role needs SSO permissions</strong> to create a workspace that uses IAM Identity Center.</p><p><strong>Check the Lambda runtime’s boto3 version.</strong>The Grafana service account token APIs are fairly new. Packaging the Lambda as a container image with its own boto3 avoids surprises.</p><p><strong>OAM shares everything by default.</strong> Without link filters, every log group and metric is visible to the monitoring account. Use filters to share only what you need.</p><p><strong>Security Checklist</strong></p><p>- Restrict the sink policy to your organization or specific accounts.</p><p>- Share only the telemetry types you use.</p><p>- Use link filters for sensitive or noisy log groups.</p><p>- Treat Grafana viewer access as read access to every linked account’s logs.</p><p>- Keep the monitoring account more locked down than any source account.</p><p>- Keep CI permissions scoped to what the deployment actually needs.</p><p><strong>When to Use This</strong></p><p><strong>Good fit:</strong> many AWS accounts with similar workloads, telemetry already in CloudWatch, and a team that needs one view.</p><p><strong>Overkill:</strong> two or three accounts, or you already send everything to a third-party observability tool.</p><p>Note that OAM centralizes <strong>viewing</strong>, not <strong>alerting</strong>. Alarms still live in each account unless you move alerting into Grafana.</p><p><strong>Key Takeaways</strong></p><p>1. OAM moves cross-account access into CloudWatch. Source accounts need one link, and Grafana needs one role.<br>2. The sink policy is your only cross-account trust, so get it exactly right.<br>3. Do not rely on managed IAM defaults. Write the Grafana role policy yourself.<br>4. Put the account ID on every dashboard query.<br>5. Decide what to share before you link, and use filters.<br>6. Automated rotation is only useful if failures are visible.</p><p>#AWS #CloudWatch #Grafana #Terraform #DevOps #Observability #PlatformEngineering</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=00d6a1f10768" width="1" height="1" alt=""><hr><p><a href="https://blog.devops.dev/how-to-build-multi-account-aws-observability-with-cloudwatch-oam-and-grafana-00d6a1f10768">How to Build Multi-Account AWS Observability with CloudWatch OAM and Grafana</a> was originally published in <a href="https://blog.devops.dev">DevOps.dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[10 Golden Rules for Reliable CI/CD]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://blog.devops.dev/10-golden-rules-for-reliable-ci-cd-534aad0f3185?source=rss----33f8b2d9a328---4"><img src="https://cdn-images-1.medium.com/max/1280/1*40ODKSF1nw6o6r0K0uwnSg.png" width="1280"></a></p><p class="medium-feed-snippet">Shipping software should never be a gamble</p><p class="medium-feed-link"><a href="https://blog.devops.dev/10-golden-rules-for-reliable-ci-cd-534aad0f3185?source=rss----33f8b2d9a328---4">Continue reading on DevOps.dev »</a></p></div>]]></description>
            <link>https://blog.devops.dev/10-golden-rules-for-reliable-ci-cd-534aad0f3185?source=rss----33f8b2d9a328---4</link>
            <guid isPermaLink="false">https://medium.com/p/534aad0f3185</guid>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[ci-cd-pipeline]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[bektiaw]]></dc:creator>
            <pubDate>Sat, 14 Nov 2026 20:46:01 GMT</pubDate>
            <atom:updated>2026-11-14T20:46:01.491Z</atom:updated>
        </item>
    </channel>
</rss>