<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[AWS Tip - Medium]]></title>
        <description><![CDATA[Best AWS, DevOps, Serverless, and more from top Medium writers . - Medium]]></description>
        <link>https://awstip.com?source=rss----b4c1e34ed5e---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>AWS Tip - Medium</title>
            <link>https://awstip.com?source=rss----b4c1e34ed5e---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 01:49:51 GMT</lastBuildDate>
        <atom:link href="https://awstip.com/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com" rel="hub"/>
        <item>
            <title><![CDATA[Getting Started with AWS Emulator: Run AWS Locally Without an AWS Account]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://awstip.com/getting-started-with-aws-emulator-run-aws-locally-without-an-aws-account-ebbea1ac22db?source=rss----b4c1e34ed5e---4"><img src="https://cdn-images-1.medium.com/max/600/0*p0S9Im5H4zEmIACL" width="600"></a></p><p class="medium-feed-snippet">Friend Link&#x2026;</p><p class="medium-feed-link"><a href="https://awstip.com/getting-started-with-aws-emulator-run-aws-locally-without-an-aws-account-ebbea1ac22db?source=rss----b4c1e34ed5e---4">Continue reading on AWS Tip »</a></p></div>]]></description>
            <link>https://awstip.com/getting-started-with-aws-emulator-run-aws-locally-without-an-aws-account-ebbea1ac22db?source=rss----b4c1e34ed5e---4</link>
            <guid isPermaLink="false">https://medium.com/p/ebbea1ac22db</guid>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[emulator]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[cloud]]></category>
            <dc:creator><![CDATA[Harshal Jethwa]]></dc:creator>
            <pubDate>Mon, 16 Nov 2026 13:31:11 GMT</pubDate>
            <atom:updated>2026-11-16T13:31:10.530Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[AWS Security Hub at Scale: Building an AI Inventory Across 100+ Accounts]]></title>
            <link>https://awstip.com/aws-security-hub-at-scale-building-an-ai-inventory-across-100-accounts-ac432cabee51?source=rss----b4c1e34ed5e---4</link>
            <guid isPermaLink="false">https://medium.com/p/ac432cabee51</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[aila-agentic-architect]]></category>
            <category><![CDATA[aws-security-hub]]></category>
            <category><![CDATA[cloud-security]]></category>
            <category><![CDATA[ai-security]]></category>
            <dc:creator><![CDATA[Alaa Rahimi]]></dc:creator>
            <pubDate>Mon, 16 Nov 2026 13:31:09 GMT</pubDate>
            <atom:updated>2026-11-16T13:31:08.478Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>Your agents are spinning up models, calling LLM APIs, and running pentests for you. Do you actually know where all your AI lives?</em></h4><p>This week, I finally had time to explore the latest Security Hub upgrades for AI, and AWS shipped one of the year&#39;s most useful organization-wide security features: <strong>AI inventory</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jjK3379bW0YE74PP" /><figcaption>Photo by <a href="https://unsplash.com/@markuswinkler?utm_source=medium&amp;utm_medium=referral">Markus Winkler</a> on <a href="https://unsplash.com?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>I’ve written about AWS Organizations and OUs in <a href="https://medium.com/@alaa-rahimi/list/aws-organisations-chronicles-74bf96eb87c3">this full series</a>. If you’re still running every workload in a single OU, start there. Many of us have built out OUs for 100+ accounts, and some have gone further with a dedicated AI OU: sandbox, production, and MLOps accounts, Bedrock guardrail policies, and extra SCPs.</p><p>That’s a great start. But an OU tells you where AI is <em>allowed</em> to run. It doesn’t tell you where AI is <em>actually</em> running.</p><p>Until now, answering “which AI workloads do we have?” meant digging through AWS Config, chasing Bedrock lines on the bill, or, for the lucky ones with a Slack integration 😉, asking in Slack. Security Hub now answers it for you, across every account and Region.</p><blockquote><strong><em>A note for new AWS customers:</em></strong><em> AWS recently launched AWS Projects, a simplified sign-up where each project is an isolated account with a hard spend limit. It’s a great sandbox for AI agents, but Security Hub, GuardDuty and custom SCPs aren’t available there until you activate advanced features. Everything below assumes a classic AWS Organization.</em></blockquote><h3>🔍 What AI inventory actually is</h3><p>Released in July 2026, AI inventory is a continuously updated, organization-wide view of your AI assets and their security posture. It’s included in Security Hub Essentials at no extra cost.</p><p>It discovers AI from three sources:</p><ol><li><strong>AWS Config → managed AI.</strong> A supported subset of Amazon Bedrock, Bedrock AgentCore, and SageMaker resources.</li><li><strong>Amazon Inspector SBOM → self-hosted AI on EC2 and in ECR images.</strong> <br>A Software Bill of Materials (SBOM) lists the software packages installed on your AWS compute resources. Security Hub reads it to spot Hugging Face and Ollama models, inference servers such as vLLM and llama.cpp, and agents.</li><li><strong>GuardDuty DNS telemetry → external AI APIs.</strong> The OpenAI, Anthropic, Mistral, Cohere, and other AI endpoints your EC2 instances call.</li></ol><p>The third one is my favorite. It’s a shadow AI detector: the developer who wired an OpenAI key into a forgotten EC2 instance now shows up in your inventory.</p><p>Every self-hosted asset is linked to the host it runs on, and managed assets show their finding counts. A <strong>canonical ID</strong> like model/pkg:huggingface/meta-llama/llama-3-8b identifies <em>what</em> a resource is, independent of <em>where</em> it runs. So you can ask &quot;where is llama-3-8b running across all 100+ accounts?&quot; and get one answer.</p><h3>🎯 Why this matters at scale</h3><ul><li><strong>You can’t secure what you don’t know exists.</strong> AWS tells the story of a team that only discovered a compromised account calling a foundation model thousands of times because finance questioned the bill.</li><li><strong>Your OU design gets verified.</strong> If AI assets show up outside your AI OU, either your SCPs have a gap or a team is going around them.</li><li><strong>It helps with compliance.</strong> If you’re under the EU AI Act, an automatic org-wide AI inventory is useful evidence for your AI register.</li><li><strong>It pairs with GuardDuty AI Protection</strong>, launched alongside it, which detects unusual model calls, stolen credentials being used to run inference on your bill, and prompt injection. When GuardDuty fires, the inventory shows you the blast radius.</li></ul><h3>⚙️ Configuring Security Hub across 100+ accounts</h3><ol><li><strong>Use central configuration with a delegated admin.</strong> The admin account sees AI assets from every account; member accounts see only their own.</li><li><strong>Make sure AWS Config records AI resource types.</strong> Managed AI discovery reads Config configuration items. If your recorder skips AWS::Bedrock::* or AWS::SageMaker::*, those assets won&#39;t appear.</li><li><strong>Turn on Inspector with the right scan mode.</strong> For EC2, use enhanced, agentless, or hybrid scanning. Plain agent-based scanning doesn’t collect enough software detail to detect AI. This is the one most people will miss.</li><li><strong>Turn on GuardDuty in every account and Region</strong> to get external-endpoint detection.</li><li><strong>Configure custom model paths in Inspector</strong> if your teams store models outside the default Hugging Face or Ollama caches.</li></ol><h3>🧭 Getting started: finding your AI resources</h3><p>Once the prerequisites are on, here’s how to go from an empty screen to a full picture of AI across your organization.</p><h4>Step 1: Sign in to the right account and Region</h4><p>Sign in to your <strong>Security Hub delegated admin account</strong> and switch to your <strong>home (aggregation) Region</strong>. Member accounts only see their own AI, and cross-Region data only shows up in the home Region. Make sure you’re in the new Security Hub, not the Security Hub CSPM view with standards and controls.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WJOpU6uUMnIRUBwXSmHxYg.png" /><figcaption>Central configuration in the delegated admin account</figcaption></figure><blockquote><em>The new Security Hub: a “Switch to classic view” button at the top right means you’re in the right experience, and AI inventory sits under Inventory in the left navigation.</em></blockquote><h4>Step 2: Open the AI inventory page</h4><p>In the left navigation, open <strong>Inventory</strong>, then <strong>AI inventory</strong>. The summary bar at the top shows the total number of AI assets, split by discovery type (managed or self-hosted) and sub-category (model, agent, inference endpoint, external endpoint). Choose any number to filter the table to those assets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8ao6xDQPmFXrCbxOjwu4TA.jpeg" /><figcaption>The AI inventory summary</figcaption></figure><h4>Step 3: Group and filter the table</h4><p>The table groups by resource type by default. Two groupings are worth trying first:</p><ul><li><strong>Account ID</strong>: shows which accounts run AI. Compare this list with the accounts in your AI OU.</li><li><strong>Canonical ID</strong>: shows every deployment of the same model or endpoint across the org. This only applies to self-hosted resources, so managed AI disappears from this view.</li></ul><p>Use the quick filters to show only self-hosted or only managed resources.</p><h4>Step 4: Drill into a resource and its signals</h4><p>Choose a row to open the detail panel. For managed resources, you see the configuration JSON. For self-hosted resources, you see the <strong>contributing signals</strong> that led to the detection, such as the SBOM components and inference software behind a model, or the DNS domain behind an external endpoint. The panel also names the host, so you can filter the table by that host to see everything else running next to it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j71701k8WNCh6QpwOYaEPw.jpeg" /><figcaption>Resource summary</figcaption></figure><h4>Step 5: Start from the host instead</h4><p>On the <strong>Resources</strong> page, EC2 instances that run self-hosted AI show an AI count badge, and managed AI/ML resources show an AI icon. Turn on the quick filter to show only hosts with self-hosted AI. Choose a host, open the <strong>AI assets</strong> tab, then choose <strong>View all</strong>.</p><h4>Step 6: Automate it with the API</h4><p>The console is great for exploring; the API is how you keep watching. This call counts AI assets by type across the org from the delegated admin:</p><pre>aws securityhub get-resources-statistics-v2 \<br>  --group-by-rules &#39;[{&quot;GroupByField&quot;:&quot;ResourceType&quot;,<br>    &quot;Filters&quot;:{&quot;CompositeFilters&quot;:[{&quot;Operator&quot;:&quot;AND&quot;,&quot;StringFilters&quot;:[<br>      {&quot;FieldName&quot;:&quot;ResourceCategory&quot;,&quot;Filter&quot;:{&quot;Comparison&quot;:&quot;EQUALS&quot;,&quot;Value&quot;:&quot;AI/ML&quot;}}]}]}}]&#39;</pre><p>And this one lists every self-hosted model:</p><pre>aws securityhub get-resources-v2 --filters &#39;{<br>  &quot;CompositeFilters&quot;:[{&quot;Operator&quot;:&quot;AND&quot;,&quot;StringFilters&quot;:[<br>    {&quot;FieldName&quot;:&quot;ResourceCategory&quot;,&quot;Filter&quot;:{&quot;Comparison&quot;:&quot;EQUALS&quot;,&quot;Value&quot;:&quot;AI/ML&quot;}},<br>    {&quot;FieldName&quot;:&quot;ResourceSubCategory&quot;,&quot;Filter&quot;:{&quot;Comparison&quot;:&quot;EQUALS&quot;,&quot;Value&quot;:&quot;Model&quot;}}]}]}&#39;</pre><p>When you filter on ResourceSubCategory, you must also filter on ResourceCategory = AI/ML, or the call fails with a ValidationException. Run these on a schedule and alert on anything new outside your AI OU.</p><h3>📂 Why your inventory might be empty</h3><p>When I first opened AI inventory, I saw nothing. If that happens to you, check these in order:</p><ol><li><strong>Wrong account or Region.</strong> Only the delegated admin sees the whole organization, and only in the home Region.</li><li><strong>AWS Config isn’t recording AI resource types.</strong> The docs say managed discovery needs no extra configuration, but it reads Config configuration items. Check with:</li></ol><pre>aws configservice describe-configuration-recorders<br><br>aws configservice get-discovered-resource-counts | grep -iE &quot;bedrock|sagemaker&quot;</pre><p><strong>3. Inspector is in plain agent-based mode on EC2.</strong> Without enhanced, agentless, or hybrid scanning, there’s no deep SBOM, so there’s no self-hosted detection. Check with aws inspector2 get-configuration.</p><p><strong>4. GuardDuty is off</strong> in that account or Region, so there are no external endpoints.</p><p><strong>5. The confidence threshold is real.</strong> A Python library on its own doesn’t count. Security Hub needs a primary signal, such as model files in a cache, plus corroborating signals before an asset appears. That’s good for false positives but confusing on the first day.</p><p><strong>6. Patience.</strong> After you fix scan modes, give Inspector up to 24 hours to rescan and correlate.</p><p>To confirm the pipeline end to end, seed a test in a <strong>sandbox account</strong>: launch a small EC2 instance managed by SSM, then run:</p><pre>curl -fsSL https://ollama.com/install.sh | sh<br>ollama pull tinyllama        # model in the default cache<br>ollama serve &amp;               # inference server<br>curl -s https://api.openai.com/v1/models &gt; /dev/null   # DNS to an external AI API</pre><p>After the next scan, the instance should show up as a model, an inference endpoint, and an external endpoint. Terminate it when you’re done.</p><h3>What about the agent running your pentest?</h3><p>Back to the question in the title. When an AI agent runs pentests or provisions infrastructure for you, it’s an AI workload itself, and it leaves a footprint: the model it calls, the host it runs on, the endpoints it reaches. With AI inventory:</p><ol><li><strong>Run the agent in a dedicated account inside your AI OU</strong>, never in production.</li><li><strong>Check the inventory before and after the run.</strong> Any new model, inference server, or external endpoint the agent created should be expected and documented.</li><li><strong>Watch for external-endpoint detections outside the testing account.</strong> That’s a sign the agent, or a tool it installed, is reaching further than intended.</li><li><strong>Combine this with EventBridge rules on Security Hub findings</strong>, so a high-severity finding during the run can automatically remove the agent’s permissions.</li></ol><h3>⚠️ Limitations</h3><ul><li>Self-hosted assets don’t have findings yet; only managed AI shows finding counts.</li><li>Self-hosted detection covers EC2 and ECR only, not Lambda, Fargate, or EKS-managed compute.</li><li>ECR images aren’t attributed to a parent host, so host AI counts don’t include them.</li><li>External-endpoint detection works from EC2 only, so a Lambda calling OpenAI is invisible to it.</li><li>Models are detected only in default cache folders unless you configure custom paths in Inspector.</li></ul><h3>🐾 A Personal Note</h3><p>First of all, if you’re already here, thanks for reading this far! As usual, I want to leave you with a small personal note.</p><p>The past few weeks have been great. I’ve started working on an automation project that will save our team a lot of time and cut maintenance work to a minimum. It’s a good reminder that while everyone is focused on new AI integrations, we still need solid organization-wide security and automation underneath it all. That’s what gives us room to grow, and to stay secure when times are tough.</p><p>On a personal level, the highlight of my year was attending the baptism of the most precious little member of our family at Gamla Uppsala Church — a medieval stone church beside burial mounds older than the Viking Age! <br>Standing in a place with centuries of history, welcoming the newest generation, and gathering with the whole family was truly special.</p><p>As the year winds down and the dark winter months settle in, I’m reminded how much the people around us matter. Encouragement from friends, colleagues, and loved ones gives you the courage to try new things and not be afraid of change.</p><p>Tell the people who support you how much you appreciate them — that’s how we keep our communities strong.</p><h3>📚 More Reads</h3><ul><li><a href="https://aws.amazon.com/about-aws/whats-new/2026/07/aws-security-hub-ai/">AWS What’s New — Security Hub AI inventory</a></li><li><a href="https://aws.amazon.com/blogs/security/security-hub-adds-ai-workload-protection-and-multicloud-support-for-microsoft-azure/">AWS Security Blog — AI workload protection and Azure support</a></li><li><a href="https://docs.aws.amazon.com/securityhub/latest/userguide/securityhub-v2-ai-inventory.html">AWS Docs — AI Inventory in Security Hub</a></li><li><a href="https://docs.aws.amazon.com/accounts/latest/reference/supported-services-sign-up-new.md">AWS Docs — Services not supported in the new AWS experience</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ac432cabee51" width="1" height="1" alt=""><hr><p><a href="https://awstip.com/aws-security-hub-at-scale-building-an-ai-inventory-across-100-accounts-ac432cabee51">AWS Security Hub at Scale: Building an AI Inventory Across 100+ Accounts</a> was originally published in <a href="https://awstip.com">AWS Tip</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Secure SFTP Server on AWS with SFTPGo, Amazon S3, MySQL RDS, and HTTPS]]></title>
            <link>https://awstip.com/building-a-secure-sftp-server-on-aws-with-sftpgo-amazon-s3-mysql-rds-and-https-979eefc125e1?source=rss----b4c1e34ed5e---4</link>
            <guid isPermaLink="false">https://medium.com/p/979eefc125e1</guid>
            <category><![CDATA[amazon-s3]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[cyber-security-awareness]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[sftp]]></category>
            <dc:creator><![CDATA[Subhomay Banerjee]]></dc:creator>
            <pubDate>Mon, 16 Nov 2026 13:31:09 GMT</pubDate>
            <atom:updated>2026-11-16T13:31:08.248Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lSJ5Pr3I2bERczYfwwqA-Q.png" /></figure><p><strong>A practical DevOps guide to deploying SFTPGo on Ubuntu, integrating Amazon S3 as file storage, configuring SSH key authentication, and implementing centralized audit logging using MySQL and SFTPGo plugins.</strong></p><p>File transfer is still an important part of many business applications. Whether it’s exchanging financial documents, sharing reports, receiving customer files, or integrating with external systems, organizations need a reliable and secure way to transfer and manage files.</p><p>Traditionally, setting up an SFTP server meant provisioning an EC2 instance, configuring OpenSSH, creating Linux users, managing directories, and worrying about disk capacity and backups.</p><p>But what if we could simplify file transfer management while keeping the actual file storage on Amazon S3?</p><p>Recently, I worked on implementing <strong>SFTPGo on AWS</strong>, using Amazon S3 as the storage backend. We also configured SSH key-based authentication, enabled audit logging through SFTPGo’s EventStore and EventSearch plugins, connected the audit database to Amazon RDS MySQL, and secured the web administration interface using Nginx and Let’s Encrypt SSL certificates.</p><p>The interesting part is that SFTPGo provides a web-based administration interface, so managing users, storage, and file activities becomes much easier.</p><p>In this article, I’ll walk through the implementation step by step.</p><h3>1. Understanding the Architecture</h3><p>Before jumping into the installation, let’s understand what we’re building.</p><p>The architecture consists of:</p><ul><li><strong>Amazon EC2:</strong> Hosts the SFTPGo application.</li><li><strong>SFTPGo:</strong> Handles SFTP connections, user management, file access, and administration.</li><li><strong>Amazon S3:</strong> Acts as the primary file storage backend.</li><li><strong>Amazon RDS MySQL:</strong> Stores audit events generated by SFTPGo plugins.</li><li><strong>SFTPGo EventStore Plugin:</strong> Captures file operations and other configured events.</li><li><strong>SFTPGo EventSearch Plugin:</strong> Makes stored audit events searchable through the administration interface.</li><li><strong>Nginx:</strong> Acts as a reverse proxy for the SFTPGo web interface.</li><li><strong>Certbot:</strong> Provides and renews HTTPS certificates through Let’s Encrypt.</li></ul><p>The SFTP service listens on port <strong>2022</strong>, while the web administration interface runs on port <strong>8080</strong> internally and is accessed through an HTTPS-enabled domain.</p><h3>2. Installing SFTPGo on Ubuntu EC2</h3><p>First, connect to your Ubuntu EC2 instance and update the packages.</p><pre>sudo apt-get update -y<br>sudo apt-get upgrade -y</pre><pre>sudo apt install -y curl gnupg \<br>    lsb-release unzip zip wget</pre><p>Next, add the SFTPGo repository signing key:</p><pre>curl -fsSL https://packages.sftpgo.com/apt/gpg.key \<br>| sudo gpg --dearmor \<br>-o /usr/share/keyrings/sftpgo.gpg</pre><p>For this implementation, we downloaded the SFTPGo Debian package directly from GitHub.</p><pre>wget https://github.com/drakkan/sftpgo/releases/download/v2.7.6/sftpgo_2.7.6-1_amd64.deb</pre><p>Install the downloaded package:</p><pre>sudo dpkg -i sftpgo_2.7.6-1_amd64.deb</pre><p><em>Note: Make sure the filename passed to </em><em>dpkg matches the package you actually downloaded. If you choose a newer release, use its corresponding package filename.</em></p><p>Verify the installation:</p><pre>sftpgo --version</pre><p>Initialize the provider:</p><pre>sudo sftpgo initprovider</pre><p>Enable and restart the SFTPGo service:</p><pre>sudo systemctl enable sftpgo<br>sudo systemctl restart sftpgo</pre><p>Check the service status:</p><pre>sudo systemctl status sftpgo</pre><p>At this point, SFTPGo should be running.</p><h3>Accessing the SFTPGo administration interface</h3><p>By default, the web administration interface is available at:</p><pre>http://&lt;EC2-PUBLIC-IP&gt;:8080</pre><p>The SFTP service listens on:</p><pre>&lt;EC2-PUBLIC-IP&gt;:2022</pre><p>Open the web interface and create your initial administrator credentials.</p><p>Once logged in, you can create SFTP users and configure their storage backends.</p><h3>3. Configuring Amazon S3 as the Storage Backend</h3><p>This is one of the main reasons I chose SFTPGo for this implementation.</p><p>Instead of storing uploaded files directly on the EC2 instance, we use Amazon S3 as the underlying storage system.</p><p>Why does this matter?</p><p>When files are stored locally on EC2, you have to consider disk capacity, volume expansion, backups, and storage management.</p><p>With S3, file storage is separated from the application server.</p><p>This provides several advantages:</p><ul><li>Storage capacity can grow independently of EC2.</li><li>Files remain outside the application server’s local filesystem.</li><li>S3 provides high durability.</li><li>Storage lifecycle policies can help manage older files.</li><li>IAM policies can control access to the bucket.</li></ul><h3>Configuring S3 in SFTPGo</h3><p>Log in to the SFTPGo administration interface.</p><p>Navigate to the user management section and create a new user.</p><p>While configuring the user’s filesystem, SFTPGo provides multiple storage backend options, including:</p><ul><li>Local Disk</li><li>Amazon S3</li><li>Azure Blob Storage</li><li>Google Cloud Storage</li><li>Other supported storage providers</li></ul><p>Select <strong>Amazon S3</strong> and configure the required bucket, region, and storage settings.</p><p>For AWS authentication, my recommendation is to use an EC2 IAM instance role with the minimum required S3 permissions rather than hardcoding long-lived AWS access keys.</p><p>The IAM policy should be restricted to the required bucket and operations.</p><p>For example, the permissions may include:</p><ul><li>s3:ListBucket</li><li>s3:GetObject</li><li>s3:PutObject</li><li>s3:DeleteObject</li></ul><p>The exact permissions depend on the operations you want SFTPGo users to perform.</p><p>Once configured, uploaded files are stored in the designated S3 bucket instead of consuming local EC2 disk space.</p><p>This makes the storage architecture much easier to maintain as the number of users and uploaded files grows.</p><h3>4. Configuring SSH Key-Based Authentication</h3><p>Security is particularly important when exposing an SFTP service to external users.</p><p>SFTPGo supports different authentication methods, including password authentication and SSH public-key authentication.</p><p>For this implementation, we preferred SSH key-based authentication.</p><p>The general approach is:</p><ol><li>The user generates an SSH key pair.</li><li>The public key is added to the corresponding SFTPGo user account.</li><li>The private key remains with the user.</li><li>During authentication, SFTPGo validates the key pair.</li></ol><p>This avoids depending on reusable passwords for SFTP access.</p><p>You can generate an SSH key pair using:</p><pre>ssh-keygen -t ed25519 -C &quot;sftp-user&quot;</pre><p>The command generates a private key and a corresponding public key.</p><p>The public key can then be added to the SFTPGo user’s SSH public keys configuration.</p><p><strong>Important:</strong> Do not distribute private keys. Keep them securely stored with the authorized user.</p><p>Also, password authentication should only be disabled globally if all required access methods and user workflows support key-based authentication.</p><h3>5. Customizing the SFTPGo Logo and Favicon</h3><p>A small but useful improvement when deploying SFTPGo for a business or client environment is customizing the administration interface.</p><p>SFTPGo allows you to replace the default logo and favicon with your own images.</p><p>The files used in our installation were:</p><p><strong>Favicon:</strong></p><pre>/usr/share/sftpgo/static/favicon.png</pre><p><strong>Logo:</strong></p><pre>/usr/share/sftpgo/static/img/logo.png</pre><p>Replace these files with your custom images, keeping appropriate dimensions and formats.</p><p>I recommend keeping a backup of the original files before replacing them.</p><p>One thing to remember: package upgrades may replace customized static assets, so keep your branding files separately for future restoration.</p><h3>6. Configuring Centralized Audit Logging with SFTPGo Plugins</h3><p>Now comes one of the most important parts of this implementation.</p><p>Managing file transfers is not only about uploading and downloading files. In many business environments, it is equally important to know:</p><ul><li>Who uploaded a file?</li><li>Who downloaded a file?</li><li>When was a file deleted?</li><li>Which user renamed or copied a file?</li><li>When was a user account created or modified?</li><li>What administrative activities took place?</li></ul><p>These activities are important for troubleshooting, operational visibility, and security auditing.</p><p>SFTPGo provides a plugin architecture that allows us to extend its capabilities.</p><p>For our implementation, we used two plugins:</p><p><strong>EventStore:</strong> Captures and stores configured events.</p><p><strong>EventSearch:</strong> Allows administrators to search the stored events through the SFTPGo administration interface.</p><h3>Installing the SFTPGo plugins</h3><p>First, update the package information:</p><pre>sudo apt update</pre><p>Check the available plugin package:</p><pre>apt-cache policy sftpgo-plugins</pre><p>Install the plugins:</p><pre>sudo apt install sftpgo-plugins</pre><p>Verify that both plugin executables are available:</p><pre>ls -l /usr/bin/sftpgo-plugin-*</pre><p>We should see:</p><pre>/usr/bin/sftpgo-plugin-eventstore<br>/usr/bin/sftpgo-plugin-eventsearch</pre><p>Make sure the installed plugin versions are compatible with your SFTPGo installation.</p><h3>7. Creating the Audit Database in Amazon RDS MySQL</h3><p>For audit event storage, we decided to use Amazon RDS MySQL instead of keeping the information locally on the EC2 instance.</p><p>This separates audit data from the SFTPGo application server and provides a managed database service.</p><p>Connect to your RDS instance:</p><pre>mysql -h &lt;RDS-ENDPOINT&gt; -u admin -p</pre><p>Create a dedicated database:</p><pre>CREATE DATABASE sftpgo_events CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;</pre><p>Next, create a dedicated database user:</p><pre>CREATE USER &#39;sftpgo&#39;@&#39;%&#39; IDENTIFIED BY &#39;&lt;STRONG-PASSWORD&gt;&#39;;</pre><p>Grant the required database permissions:</p><pre>GRANT ALL PRIVILEGES ON sftpgo_events.* TO &#39;sftpgo&#39;@&#39;%&#39;;</pre><pre>FLUSH PRIVILEGES;</pre><p>For a production deployment, review the required database privileges and restrict the database user’s host and network access wherever possible.</p><p>Also ensure that the RDS security group allows MySQL traffic on port 3306 only from the SFTPGo EC2 security group.</p><p>The database endpoint should not be publicly accessible.</p><h3>8. Configuring EventStore and EventSearch Plugins</h3><p>Once the database and plugins are ready, we need to configure SFTPGo to use them.</p><p>SFTPGo supports environment-variable-based configuration, which makes this process straightforward.</p><p>Open the environment configuration file:</p><pre>sudo vi /etc/sftpgo/sftpgo.env</pre><p>Add the following configuration.</p><h3>A. MySQL database connection</h3><pre># ==========================================<br># SFTPGo Audit Database - MySQL<br># ==========================================<br><br>SFTPGO_PLUGIN_EVENTSTORE_DSN=&quot;sftpgo:&lt;DB-PASSWORD&gt;@tcp(&lt;RDS-ENDPOINT&gt;:3306)/sftpgo_events?charset=utf8mb4&amp;interpolateParams=true&amp;timeout=10s&amp;writeTimeout=10s&amp;readTimeout=10s&quot;<br>SFTPGO_PLUGIN_EVENTSEARCH_DSN=&quot;sftpgo:&lt;DB-PASSWORD&gt;@tcp(&lt;RDS-ENDPOINT&gt;:3306)/sftpgo_events?charset=utf8mb4&amp;interpolateParams=true&amp;timeout=10s&amp;writeTimeout=10s&amp;readTimeout=10s&quot;</pre><p>Both plugins use the same MySQL database in this configuration.</p><p>The EventStore plugin writes the configured events, while EventSearch retrieves them for the administration interface.</p><h3>B. EventStore plugin configuration</h3><pre># ==========================================<br># EventStore Plugin<br># ==========================================<br><br>SFTPGO_PLUGINS__0__TYPE=notifier<br>SFTPGO_PLUGINS__0__NOTIFIER_OPTIONS__FS_EVENTS=&quot;download,upload,delete,rename,mkdir,rmdir,ssh_cmd,copy&quot;<br>SFTPGO_PLUGINS__0__NOTIFIER_OPTIONS__PROVIDER_EVENTS=&quot;add,update,delete&quot;<br>SFTPGO_PLUGINS__0__NOTIFIER_OPTIONS__PROVIDER_OBJECTS=&quot;user,folder,group,admin,api_key,share,event_action,event_rule,role,ip_list_entry,configs&quot;<br>SFTPGO_PLUGINS__0__NOTIFIER_OPTIONS__LOG_EVENTS=&quot;1,2,3,4,5,6,7,8,9&quot;<br>SFTPGO_PLUGINS__0__NOTIFIER_OPTIONS__RETRY_MAX_TIME=300<br>SFTPGO_PLUGINS__0__NOTIFIER_OPTIONS__RETRY_QUEUE_MAX_SIZE=10000<br>SFTPGO_PLUGINS__0__CMD=&quot;/usr/bin/sftpgo-plugin-eventstore&quot;<br>SFTPGO_PLUGINS__0__ARGS=&quot;serve,--driver,mysql&quot;<br>SFTPGO_PLUGINS__0__AUTO_MTLS=1</pre><p>Let’s understand some of these settings.</p><p><strong>FS_EVENTS</strong></p><p>Defines the filesystem activities we want to capture, such as uploads, downloads, deletions, renames, directory creation, and copy operations.</p><p><strong>PROVIDER_EVENTS</strong></p><p>Controls which provider-related events are captured, including additions, updates, and deletions.</p><p><strong>PROVIDER_OBJECTS</strong></p><p>Specifies the SFTPGo objects whose management activities we want to monitor, such as users, folders, groups, administrators, API keys, and configurations.</p><p><strong>RETRY_MAX_TIME and RETRY_QUEUE_MAX_SIZE</strong></p><p>These settings configure retry handling and the maximum retry queue size for the notifier plugin.</p><p><strong>CMD and ARGS</strong></p><p>Specify the EventStore plugin executable and instruct it to run using the MySQL driver.</p><p><strong>AUTO_MTLS</strong></p><p>Enables automatic mutual TLS for communication between SFTPGo and its plugin.</p><h3>C. EventSearch plugin configuration</h3><p>Next, configure the EventSearch plugin:</p><pre># ==========================================<br># EventSearch Plugin<br># ==========================================<br><br>SFTPGO_PLUGINS__1__TYPE=eventsearcher<br>SFTPGO_PLUGINS__1__CMD=&quot;/usr/bin/sftpgo-plugin-eventsearch&quot;<br>SFTPGO_PLUGINS__1__ARGS=&quot;serve,--driver,mysql&quot;<br>SFTPGO_PLUGINS__1__AUTO_MTLS=1</pre><p>The EventSearch plugin uses the same MySQL driver and database connection configuration.</p><p>Together, these plugins allow us to capture configured activities and search the stored audit events through the SFTPGo interface.</p><p><strong>Important:</strong> Ensure the SFTPGo systemd service actually loads /etc/sftpgo/sftpgo.env, and protect this file with appropriate permissions. Also, verify the plugin configuration and database connectivity before relying on the audit trail in production.</p><h3>9. Restart SFTPGo and Verify Audit Logging</h3><p>Once the configuration is complete, restart the service:</p><pre>sudo systemctl restart sftpgo</pre><p>Check the service:</p><pre>sudo systemctl status sftpgo</pre><p>If you encounter any issues, inspect the service logs:</p><pre>sudo journalctl -u sftpgo -n 100 --no-pager</pre><p>You should also check the database connectivity and plugin startup messages.</p><h3>Where to view audit logs?</h3><p>Log in to the SFTPGo administration interface.</p><p>Navigate to:</p><p><strong>Admin UI → Server Manager → Logs</strong></p><p>The configured audit events should now be available for searching.</p><p>To verify the implementation, perform a few test operations:</p><ol><li>Upload a file using an SFTP user.</li><li>Download the uploaded file.</li><li>Rename a file.</li><li>Delete a test file.</li><li>Create or update an SFTPGo user.</li></ol><p>Then revisit the Logs section and verify that the corresponding activities have been recorded.</p><p>This is an important step. Installing plugins is only part of the job; validating that the events are actually being stored and retrieved is what makes the implementation useful.</p><h3>10. Configuring Nginx and HTTPS with Let’s Encrypt</h3><p>The SFTPGo administration interface initially runs over HTTP on port 8080.</p><p>For a production environment, we don’t want administrators accessing the interface over an unencrypted HTTP connection.</p><p>Instead, we configured Nginx as a reverse proxy and used Certbot to obtain an SSL certificate from Let’s Encrypt.</p><p>The resulting access flow is:</p><p><strong>Administrator → HTTPS Domain → Nginx → SFTPGo (localhost:8080)</strong></p><h3>Install Nginx</h3><pre>sudo apt update<br>sudo apt install nginx -y</pre><p>Create a server configuration for your domain:</p><pre>sudo nano /etc/nginx/sites-available/sftpgo</pre><p>Example configuration:</p><pre>server {<br>    listen 80;<br>    server_name files.example.com;</pre><pre>    location / {<br>        proxy_pass <a href="http://127.0.0.1:8080;">http://127.0.0.1:8080;</a></pre><pre>        proxy_set_header Host $host;<br>        proxy_set_header X-Real-IP $remote_addr;<br>        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;<br>        proxy_set_header X-Forwarded-Proto $scheme;<br>    }<br>}</pre><p>Replace files.example.com with your actual domain.</p><p>Enable the configuration:</p><pre>sudo ln -s \<br>/etc/nginx/sites-available/sftpgo \<br>/etc/nginx/sites-enabled/sftpgo</pre><p>Test the configuration:</p><pre>sudo nginx -t</pre><p>Reload Nginx:</p><pre>sudo systemctl reload nginx</pre><h3>Install Certbot</h3><p>Install Certbot and its Nginx integration:</p><pre>sudo apt install certbot python3-certbot-nginx -y</pre><p>Request an SSL certificate:</p><pre>sudo certbot --nginx -d files.example.com</pre><p>Follow the prompts to complete the certificate issuance.</p><p>Certbot will configure the HTTPS virtual host and can also redirect HTTP traffic to HTTPS.</p><p>Verify automatic renewal:</p><pre>sudo certbot renew --dry-run</pre><p>Now the administration interface should be accessible using:</p><pre>https://files.example.com</pre><p>A few important networking considerations:</p><ul><li>Port 443 should be accessible to authorized administrators.</li><li>Port 80 is needed for the HTTP challenge and optional HTTP-to-HTTPS redirection.</li><li>SFTP port 2022 must remain separately accessible to authorized SFTP clients.</li><li>Port 8080 should not be exposed publicly if Nginx is the intended access path. Bind the SFTPGo web service to localhost or restrict it through firewall rules.</li></ul><p>Remember that HTTPS for the web interface and SSH encryption for SFTP are two separate things. Both need to be secured.</p><h3>11. Final Architecture and Security Considerations</h3><p>After completing the configuration, we have a setup where file storage, application management, and audit logging are separated into different components.</p><p>Here are some of the benefits of this approach:</p><p><strong>1. Centralized File Storage</strong></p><p>Amazon S3 handles the actual file storage, reducing dependency on the EC2 instance’s local disk.</p><p><strong>2. Secure Authentication</strong></p><p>SSH public-key authentication provides a more controlled approach to SFTP user access.</p><p><strong>3. Centralized Audit Trail</strong></p><p>The EventStore plugin captures configured activities and stores them in Amazon RDS MySQL.</p><p><strong>4. Searchable Logs</strong></p><p>The EventSearch plugin allows administrators to search audit events directly from the SFTPGo administration interface.</p><p><strong>5. Encrypted Web Administration</strong></p><p>Nginx and Let’s Encrypt provide HTTPS access to the administration interface.</p><p><strong>6. Separation of Responsibilities</strong></p><p>The EC2 instance runs SFTPGo, S3 stores files, and RDS stores audit events. This separation makes the environment easier to maintain and extend.</p><p>For production, I would additionally recommend:</p><ul><li>Enable S3 encryption at rest and enforce TLS for S3 requests.</li><li>Enable S3 versioning if file recovery is required.</li><li>Configure appropriate S3 lifecycle and retention policies.</li><li>Use IAM roles instead of static AWS credentials.</li><li>Keep RDS in private subnets.</li><li>Restrict security group access to the required ports and sources.</li><li>Monitor SFTPGo and Nginx logs.</li><li>Back up the audit database and test restoration.</li><li>Keep SFTPGo and its plugins updated.</li><li>Protect SSH private keys and administrator credentials.</li></ul><p>One important distinction: S3 versioning, lifecycle policies, and backups serve different purposes. A lifecycle policy manages object transitions or expiration; it is not by itself a complete backup strategy.</p><h3>Final Thoughts</h3><p>SFTP infrastructure doesn’t have to be complicated or dependent on local disk storage.</p><p>By combining SFTPGo with Amazon S3, we can build a flexible file transfer platform where storage capacity is independent of the application server.</p><p>Adding SSH key-based authentication, centralized audit logging through MySQL RDS, and HTTPS administration makes the solution more suitable for business environments where security and visibility matter.</p><p>What I particularly like about this implementation is that SFTPGo gives us a web-based interface for managing users and file activities, while AWS handles the underlying storage and database services.</p><p>It’s a practical architecture for organizations that need managed SFTP capabilities without having to build and maintain every component themselves.</p><p><strong>If you’re planning to build a secure file transfer platform on AWS, this architecture is a good starting point.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/146/1*vMyFTOWtr9c252J4X1FW6g.gif" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=979eefc125e1" width="1" height="1" alt=""><hr><p><a href="https://awstip.com/building-a-secure-sftp-server-on-aws-with-sftpgo-amazon-s3-mysql-rds-and-https-979eefc125e1">Building a Secure SFTP Server on AWS with SFTPGo, Amazon S3, MySQL RDS, and HTTPS</a> was originally published in <a href="https://awstip.com">AWS Tip</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Medallion Architecture on AWS]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://awstip.com/building-a-medallion-architecture-on-aws-899034d44213?source=rss----b4c1e34ed5e---4"><img src="https://cdn-images-1.medium.com/max/1042/1*WQ3K8lDEzCS6iBhF-_LhLw.png" width="1042"></a></p><p class="medium-feed-snippet">DataForge AWS Data Engineering Series - Blog 13 of 20</p><p class="medium-feed-link"><a href="https://awstip.com/building-a-medallion-architecture-on-aws-899034d44213?source=rss----b4c1e34ed5e---4">Continue reading on AWS Tip »</a></p></div>]]></description>
            <link>https://awstip.com/building-a-medallion-architecture-on-aws-899034d44213?source=rss----b4c1e34ed5e---4</link>
            <guid isPermaLink="false">https://medium.com/p/899034d44213</guid>
            <category><![CDATA[medallion-architecture]]></category>
            <category><![CDATA[data-lake]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[data-architecture]]></category>
            <dc:creator><![CDATA[The Data Forge]]></dc:creator>
            <pubDate>Mon, 16 Nov 2026 13:31:06 GMT</pubDate>
            <atom:updated>2026-11-16T13:31:05.007Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[AWS Emulator vs LocalStack: Is LocalStack Getting Kicked Out of Its Own House? ]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://awstip.com/aws-emulator-vs-localstack-is-localstack-getting-kicked-out-of-its-own-house-8e9eefd9e10c?source=rss----b4c1e34ed5e---4"><img src="https://cdn-images-1.medium.com/max/1537/1*bBqBRsgJcJDrXwGfWd3Edw.png" width="1537"></a></p><p class="medium-feed-snippet">AWS has entered the local cloud development space, and things just got interesting.</p><p class="medium-feed-link"><a href="https://awstip.com/aws-emulator-vs-localstack-is-localstack-getting-kicked-out-of-its-own-house-8e9eefd9e10c?source=rss----b4c1e34ed5e---4">Continue reading on AWS Tip »</a></p></div>]]></description>
            <link>https://awstip.com/aws-emulator-vs-localstack-is-localstack-getting-kicked-out-of-its-own-house-8e9eefd9e10c?source=rss----b4c1e34ed5e---4</link>
            <guid isPermaLink="false">https://medium.com/p/8e9eefd9e10c</guid>
            <category><![CDATA[emulator]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[Harshal Jethwa]]></dc:creator>
            <pubDate>Mon, 16 Nov 2026 13:31:05 GMT</pubDate>
            <atom:updated>2026-11-16T13:31:03.872Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Building a Cloud-Native File Upload and Analytics Platform Using AWS (Step-by-Step Guide)]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://awstip.com/building-a-cloud-native-file-upload-and-analytics-platform-using-aws-step-by-step-guide-ed5950c228a3?source=rss----b4c1e34ed5e---4"><img src="https://cdn-images-1.medium.com/max/788/0*Crs8AAtqRxnGSkEd.png" width="788"></a></p><p class="medium-feed-snippet">I recently built a cloud-native project where users can upload videos or files through a web page, have them stored securely in Amazon S3&#x2026;</p><p class="medium-feed-link"><a href="https://awstip.com/building-a-cloud-native-file-upload-and-analytics-platform-using-aws-step-by-step-guide-ed5950c228a3?source=rss----b4c1e34ed5e---4">Continue reading on AWS Tip »</a></p></div>]]></description>
            <link>https://awstip.com/building-a-cloud-native-file-upload-and-analytics-platform-using-aws-step-by-step-guide-ed5950c228a3?source=rss----b4c1e34ed5e---4</link>
            <guid isPermaLink="false">https://medium.com/p/ed5950c228a3</guid>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[filesystem]]></category>
            <dc:creator><![CDATA[Harshal Jethwa]]></dc:creator>
            <pubDate>Mon, 16 Nov 2026 13:31:02 GMT</pubDate>
            <atom:updated>2026-11-16T13:31:01.603Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[EKS Makes Kubernetes Easier. It Doesn’t Make It Simpler.]]></title>
            <link>https://awstip.com/eks-makes-kubernetes-easier-it-doesnt-make-it-simpler-3cc546d5fe0f?source=rss----b4c1e34ed5e---4</link>
            <guid isPermaLink="false">https://medium.com/p/3cc546d5fe0f</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[aws-eks]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[Karan Singh Saini]]></dc:creator>
            <pubDate>Mon, 16 Nov 2026 13:31:00 GMT</pubDate>
            <atom:updated>2026-11-16T13:30:59.650Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/627/1*nijWsv06Af4za3DA_4ONyA.png" /></figure><p>You have an EKS cluster, your applications are deployed, and AWS is managing the Kubernetes control plane. You don’t have to maintain the API server, worry about its availability, or manage the infrastructure behind it.</p><p>So how much Kubernetes do you actually need to worry about?</p><p>Quite a lot, as it turns out.</p><p>Imagine your application suddenly starts receiving more traffic. Karpenter provisions additional nodes, but some of your pods are still stuck in <em>Pending</em>. You check CPU and memory, and neither appears to be a problem.</p><p>Or perhaps a deployment completes successfully, but users start reporting intermittent 502 errors. Kubernetes says everything is healthy, yet requests are failing.</p><p>These aren’t necessarily problems with EKS. They’re examples of what happens when the parts of Kubernetes that AWS doesn’t manage start affecting your applications.</p><p>I think this is one of the most important things to understand before running Kubernetes in production on AWS: <strong>a managed Kubernetes control plane doesn’t mean a managed Kubernetes environment.</strong></p><h3>The first surprise: scaling isn’t always about compute</h3><p>Let’s go back to that application receiving more traffic.</p><p>Your cluster needs additional capacity, so Karpenter provisions new EC2 instances. The nodes join the cluster, and you’d expect the pending pods to get scheduled.</p><p>But they don’t.</p><p>One possible reason is networking.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/690/1*z4NS6Zp3IGwTBdcyniGjeQ.jpeg" /></figure><p>With the VPC CNI, every pod gets a real VPC IP address, and each instance type has a limit on how many it can hold. If a node hits that limit, new pods stay <em>Pending</em> even though CPU and memory are free. And if the subnet itself runs out of addresses, the pod gets scheduled but sits in <em>ContainerCreating</em> with a “failed to assign an IP address” event. Same root cause, different symptom, and it’s worth knowing which one you’re looking at.</p><p>This creates an interesting situation: you can have plenty of CPU and memory available across your cluster and still be unable to schedule additional pods.</p><p>Adding more nodes might help if the existing nodes have reached their networking limits. But if the underlying subnets are running out of addresses, adding nodes can make the situation worse rather than solve it.</p><p>You may need to look at prefix delegation, custom networking, warm IP pool settings, or even your overall VPC address allocation.</p><p>This is something worth thinking about before your first scaling event. How many pods do you expect to run? How much IP address capacity will they consume? Do your subnets have enough room for future growth?</p><p>When planning EKS capacity, CPU and memory are only part of the equation.</p><h3>Your deployment succeeded. So why are users getting errors?</h3><p>Here’s another situation that can be confusing, particularly when teams are relatively new to running production workloads on Kubernetes.</p><p>You deploy a new application version. The new pods become ready, the rollout completes, and Kubernetes reports that the deployment is healthy.</p><p>Then your monitoring starts showing 502 errors.</p><p>What happened?</p><p>One possible explanation is a race between pod termination, Kubernetes endpoint updates, and load balancer target deregistration.</p><p>When Kubernetes terminates a pod, several things happen around the same time. The pod begins its shutdown process, its endpoint status changes, and the load balancing infrastructure needs to stop sending traffic to it.</p><p>These events aren’t necessarily instantaneous. If the application stops accepting requests before the load balancer has stopped forwarding traffic, users can experience failed requests during an otherwise successful deployment.</p><p>A <em>preStop</em> hook can introduce a delay before the container receives its termination signal:</p><pre>lifecycle:<br>  preStop:<br>    exec:<br>      command: [&quot;/bin/sh&quot;, &quot;-c&quot;, &quot;sleep 15&quot;]</pre><p>But don’t treat this as a magic fix. The right configuration depends on your target type, readiness checks, deregistration delay, termination grace period, ingress setup, and application shutdown behaviour.</p><p>The bigger lesson is that Kubernetes reporting a successful rollout doesn’t necessarily mean your users experienced an uninterrupted deployment.</p><p>Have you actually tested what happens to active requests when Kubernetes terminates one of your application pods? It’s worth finding out before a production deployment does it for you.</p><h3>When saving money creates an availability problem</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/854/1*53C6X-qErjVTN9IulWtW_g.gif" /></figure><p>Karpenter is one of the reasons EKS can be attractive from a cost and operational perspective. It provisions nodes based on workload requirements and can consolidate underutilized capacity.</p><p>But think about what consolidation actually involves: existing nodes may be terminated, and their workloads need to move somewhere else.</p><p>What happens if several replicas of your application happen to be running on the same node?</p><p>If those replicas are concentrated on one node, an unexpected node failure could take down multiple replicas at once. Even during planned consolidation, restrictive or poorly configured disruption budgets can create problems. The application might recover, but that doesn’t mean users won’t notice the disruption.</p><p>This is where PodDisruptionBudgets and topology spread constraints become important. A PDB helps control how many replicas can be voluntarily disrupted, while topology spread constraints help distribute workloads across nodes and availability zones.</p><p>Neither is a replacement for designing applications to tolerate failures. A restrictive PDB can also prevent node maintenance from progressing, so its settings need to reflect how the application actually works.</p><p>The interesting part is that your autoscaling configuration and your application’s availability configuration are closely connected.</p><p>You can configure Karpenter perfectly and still have an application that isn’t resilient to the infrastructure changes Karpenter makes.</p><h3>The IAM problem hiding inside your pods</h3><p>Now let’s look at something that can easily be overlooked when setting up an EKS cluster.</p><p>An application needs to access an S3 bucket. Someone attaches the required permissions to the EC2 instance role, and the application starts working.</p><p>It seems straightforward, until you consider what else might be able to use those permissions.</p><p>If a pod can access the EC2 Instance Metadata Service and the appropriate protections aren’t in place, it may be able to retrieve credentials associated with the node’s IAM role. That role might have permissions well beyond what the application needs.</p><p>This is why workload identity matters.</p><p>EKS supports IAM Roles for Service Accounts (IRSA), which allows you to associate an IAM role with a Kubernetes service account. EKS Pod Identity provides another mechanism for associating IAM roles with workloads.</p><p>Both approaches help avoid relying on broad permissions attached to the worker nodes, although they have different configuration and compatibility considerations.</p><p>If you’re running multiple applications or teams in the same cluster, this becomes especially important. You don’t want every workload inheriting permissions simply because it happens to run on the same EC2 instance.</p><p>A useful question to ask when reviewing an EKS environment is: <strong>if one of my pods were compromised, what AWS permissions could it access?</strong></p><p>The answer might tell you more about your security posture than looking at the node role alone.</p><h3>And then comes the Kubernetes upgrade</h3><p>At some point, you’ll need to upgrade your cluster. AWS makes the control plane upgrade relatively straightforward, but the rest of the environment still needs to be compatible.</p><p>Your managed add-ons, self-managed components, node groups, networking configuration, and applications all need to be considered.</p><p>Even something as simple as draining a node can become a problem if a PodDisruptionBudget prevents Kubernetes from evicting a workload.</p><p>There is also a financial reason not to postpone upgrades indefinitely. EKS provides 14 months of standard support for a Kubernetes version, followed by 12 months of extended support. Extended support costs $0.60 per cluster-hour compared with $0.10 during standard support.</p><p>For a cluster spending a full year in extended support, that’s an additional $4,380.</p><p>So, who is tracking Kubernetes version support dates? Who checks add-on compatibility? Who tests whether workloads can survive node replacement?</p><p>If the answer is nobody in particular, upgrading is likely to become a much bigger exercise than it needs to be.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/450/1*BIrBoCOThv_QHoFE4IyaxQ.jpeg" /></figure><h3>Does EKS Auto Mode change the equation?</h3><p>To be fair, AWS is continuing to reduce the amount of infrastructure teams need to manage.</p><p>EKS Auto Mode can take responsibility for compute provisioning and node lifecycle management, along with integrations for networking, DNS, storage, and load balancing.</p><p>That’s a meaningful improvement for teams that don’t want to manage every part of the infrastructure supporting their clusters.</p><p>But it doesn’t eliminate the responsibilities we’ve discussed. You still need to plan subnet capacity, configure workload permissions, make applications tolerate disruptions, and test deployments and upgrades.</p><p>EKS Auto Mode enforces a maximum node lifetime of 21 days, after which nodes are replaced automatically. This means workloads need to tolerate regular node rotation.</p><p>The amount of infrastructure you manage changes, but understanding how your applications behave on Kubernetes remains important.</p><h3>So, how managed is your managed Kubernetes?</h3><p>I don’t think any of this makes EKS a bad choice. Quite the opposite. Having AWS manage the control plane removes a significant operational burden, and features such as Karpenter and Auto Mode make running Kubernetes on AWS more accessible.</p><p>But there’s a difference between having AWS manage Kubernetes infrastructure and having AWS manage the operational behaviour of your applications.</p><p>Networking can still prevent pods from starting. Deployments can still interrupt traffic. Autoscaling can still expose availability problems. IAM permissions can still be broader than they should be, and upgrades can still become difficult.</p><p>These are not unusual edge cases that disappear once a cluster is configured correctly. They’re responsibilities that come with running workloads on Kubernetes.</p><p>Before choosing EKS, it’s worth thinking beyond the cost of the cluster and the effort required to provision it. You also need people who understand what happens when workloads scale, nodes disappear, networking runs out of capacity, or an upgrade doesn’t go as planned.</p><p><strong>AWS manages the control plane. Your team still needs to understand everything running on top of it.</strong></p><p>That’s what makes EKS easier than managing Kubernetes yourself, but not necessarily simple.</p><p>I’d love to hear from people running EKS in production: what are the problems that made you realise a managed Kubernetes service still needs quite a bit of operational work?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3cc546d5fe0f" width="1" height="1" alt=""><hr><p><a href="https://awstip.com/eks-makes-kubernetes-easier-it-doesnt-make-it-simpler-3cc546d5fe0f">EKS Makes Kubernetes Easier. It Doesn’t Make It Simpler.</a> was originally published in <a href="https://awstip.com">AWS Tip</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Amazon Athena: SQL on S3 Without Servers]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://awstip.com/amazon-athena-sql-on-s3-without-servers-755b215500e7?source=rss----b4c1e34ed5e---4"><img src="https://cdn-images-1.medium.com/max/954/1*_51C4fg-VKHWP0t3OJXJBg.png" width="954"></a></p><p class="medium-feed-snippet">DataForge AWS Data Engineering Series - Blog 12 of 20</p><p class="medium-feed-link"><a href="https://awstip.com/amazon-athena-sql-on-s3-without-servers-755b215500e7?source=rss----b4c1e34ed5e---4">Continue reading on AWS Tip »</a></p></div>]]></description>
            <link>https://awstip.com/amazon-athena-sql-on-s3-without-servers-755b215500e7?source=rss----b4c1e34ed5e---4</link>
            <guid isPermaLink="false">https://medium.com/p/755b215500e7</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[amazon-athena]]></category>
            <category><![CDATA[cost-optimization]]></category>
            <dc:creator><![CDATA[The Data Forge]]></dc:creator>
            <pubDate>Mon, 16 Nov 2026 13:30:59 GMT</pubDate>
            <atom:updated>2026-11-16T13:30:57.873Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[From cloud engineer to forward deployed engineer on AWS]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://awstip.com/from-cloud-engineer-to-forward-deployed-engineer-on-aws-78f85897ffec?source=rss----b4c1e34ed5e---4"><img src="https://cdn-images-1.medium.com/max/2600/1*vldLAib1YZiO8ni5gcsM_w.png" width="3000"></a></p><p class="medium-feed-snippet">TL;DR</p><p class="medium-feed-link"><a href="https://awstip.com/from-cloud-engineer-to-forward-deployed-engineer-on-aws-78f85897ffec?source=rss----b4c1e34ed5e---4">Continue reading on AWS Tip »</a></p></div>]]></description>
            <link>https://awstip.com/from-cloud-engineer-to-forward-deployed-engineer-on-aws-78f85897ffec?source=rss----b4c1e34ed5e---4</link>
            <guid isPermaLink="false">https://medium.com/p/78f85897ffec</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[cloud-architecture]]></category>
            <category><![CDATA[forward-deployed-engineer]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[ai-agent]]></category>
            <dc:creator><![CDATA[Ioannis Moustakis]]></dc:creator>
            <pubDate>Mon, 16 Nov 2026 13:30:57 GMT</pubDate>
            <atom:updated>2026-11-16T13:30:56.538Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[AWS Step Functions for Data Pipeline Orchestration]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://awstip.com/aws-step-functions-for-data-pipeline-orchestration-06fd844ca956?source=rss----b4c1e34ed5e---4"><img src="https://cdn-images-1.medium.com/max/954/1*Zx06BitJDopbbc38oeF1jQ.png" width="954"></a></p><p class="medium-feed-snippet">DataForge AWS Data Engineering Series - Blog 11 of 20</p><p class="medium-feed-link"><a href="https://awstip.com/aws-step-functions-for-data-pipeline-orchestration-06fd844ca956?source=rss----b4c1e34ed5e---4">Continue reading on AWS Tip »</a></p></div>]]></description>
            <link>https://awstip.com/aws-step-functions-for-data-pipeline-orchestration-06fd844ca956?source=rss----b4c1e34ed5e---4</link>
            <guid isPermaLink="false">https://medium.com/p/06fd844ca956</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[workflow-automation]]></category>
            <category><![CDATA[orchestration]]></category>
            <category><![CDATA[aws-step-functions]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[The Data Forge]]></dc:creator>
            <pubDate>Mon, 16 Nov 2026 13:30:56 GMT</pubDate>
            <atom:updated>2026-11-16T13:30:54.579Z</atom:updated>
        </item>
    </channel>
</rss>