<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Towards Data Engineering - Medium]]></title>
        <description><![CDATA[Explore top articles on data engineering, cloud architecture, agentic AI, and MLOps. Join our thriving Medium community of data professionals to discover curated insights, share your expertise, and build the future of AI and data-driven ecosystems. - Medium]]></description>
        <link>https://medium.com/towards-data-engineering?source=rss----37f58dd42be7---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Towards Data Engineering - Medium</title>
            <link>https://medium.com/towards-data-engineering?source=rss----37f58dd42be7---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 00:52:27 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/towards-data-engineering" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com" rel="hub"/>
        <item>
            <title><![CDATA[The Hidden Data Privacy Risks Behind Enterprise AI Agents]]></title>
            <link>https://medium.com/towards-data-engineering/the-hidden-data-privacy-risks-behind-enterprise-ai-agents-61d0c4627c31?source=rss----37f58dd42be7---4</link>
            <guid isPermaLink="false">https://medium.com/p/61d0c4627c31</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[cybersecurity]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ai-agent]]></category>
            <dc:creator><![CDATA[Satyam Sahu]]></dc:creator>
            <pubDate>Tue, 17 Nov 2026 14:27:46 GMT</pubDate>
            <atom:updated>2026-11-17T14:27:45.926Z</atom:updated>
            <content:encoded><![CDATA[<h4>Understand how AI agents expose enterprise data, where privacy risks emerge, and how to secure every model call before production.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k9bc7MRNnxErZiyA81rKUQ.png" /><figcaption>Cover image created using AI</figcaption></figure><p>Getting an enterprise AI project into production isn’t just about choosing the best model or having enough budget. As we head into 2027, there’s another question companies are taking much more seriously: <strong>Where does our data actually go?</strong></p><p>Think about it. A single AI agent might call several different models to complete one task. Along the way, sensitive information like customer records, contracts, or even source code could end up passing through multiple third-party providers. And that’s where things get tricky.</p><p>So, how do you keep that data safe without giving up the benefits of AI agents?</p><blockquote><em>In this article, we’ll look at where your data goes, compare different deployment options, and walk through a practical privacy checklist to help you prepare for your next security review.</em></blockquote><h3>Why Is Privacy the Deciding Factor Now?</h3><p>Here’s the thing: companies are adopting AI faster than they’re putting the right security controls in place. And the numbers make that pretty clear.</p><p>Several industry surveys from 2026 highlight just how serious the problem has become:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*opIA5sT_VLN1h1-MbbuTIA.png" /></figure><p>The pattern is hard to ignore. Companies want to move faster with AI, but their security and governance practices aren’t keeping up.</p><p>Regulation is adding another layer to the problem. The EU AI Act deadlines for high-risk systems have been extended, but the requirements haven’t disappeared. Under the <a href="https://artificialintelligenceact.eu/ai-act-explorer/digital-omnibus/">Digital Omnibus on AI</a>, standalone high-risk systems listed in Annex III, including hiring tools, credit scoring, and education systems, now have until December 2, 2027, to comply, instead of August 2, 2026.</p><p>So, privacy isn’t something teams can leave for the final security review anymore. <strong>If you’re building an AI agent for enterprise use, data privacy needs to be part of the design from day one.</strong></p><h3>Why Do AI Agents Make the Privacy Problem Bigger?</h3><p>Here’s where things get a little more complicated.</p><p>A typical AI agent doesn’t rely on just one model to complete a task. It often makes several model calls, with each one handling a different part of the job.</p><p>A good example is the <a href="https://superlinked.com/blog/contract-review-agent">contract review agent workflow</a> documented by Superlinked, which involves ten separate model calls. These cover everything from initial document triage and vision processing to OCR, embeddings, reranking, entity extraction, text-to-SQL, risk reasoning, orchestration, and safety checks.</p><p>The problem? Each of these steps could involve a different model or service provider. That means your data might leave your infrastructure several times during a single agent workflow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EMswR_HiuLoIuWjZkrdCOA.png" /><figcaption><em>An AI agent doesn’t send data to just one model. Information can pass through multiple external providers, tools, and observability systems during a single workflow.</em></figcaption></figure><p>Let’s look at where that happens:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wIRWT__oLFLdOPSuFOVb9A.png" /></figure><p>And that’s the part many teams overlook. <strong>Data doesn’t just leave your system when someone sends a prompt to an LLM. It can travel through embeddings, search, external tools, safety checks, and even monitoring systems.</strong></p><p>The more moving parts your agent has, the more carefully you need to track where that data goes.</p><h3>Hosted API vs. Self-Hosted vs. Hybrid: Which Keeps Your Data Safer?</h3><p>Now that we know how many places an agent can send your data, let’s look at the three main ways to deploy these models: hosted APIs, self-hosted open models, and hybrid deployments.</p><p>But before comparing them, there’s something worth understanding. <strong>Using a hosted API doesn’t mean your data disappears as soon as the model responds.</strong> Providers have their own data retention policies, and those policies matter.</p><p>Let’s look at the data retention policies documented by two widely used LLM providers, <strong>OpenAI and Anthropic</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QeDK3_0aX3WR_BiOtryJiQ.png" /></figure><p>This doesn’t mean these providers are careless with customer data. In fact, OpenAI states that API data hasn’t been used to train its models since March 2023, unless customers explicitly opt in.</p><p>Still, if you’re building an enterprise AI agent, you need to understand what happens to your data after every API call.</p><p>So, which deployment approach makes the most sense?</p><p>Let’s compare all three:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JDgTebHt92_ussSwuv82Sg.png" /></figure><p>There’s no single winner here. <strong>Hosted APIs</strong> give you access to powerful models without managing infrastructure. <strong>Self-hosting</strong> gives you greater control over where your data is processed, but also makes security your responsibility.</p><p><strong>Hybrid deployments </strong>offer a middle ground. You can keep sensitive workloads inside your own infrastructure while still using hosted frontier models for tasks that need them.</p><p>And for most enterprise AI agents, that balance is worth considering.</p><figure><img alt="Hosted vs. Self-Hosted vs. Hybrid: Three Ways to Deploy Enterprise AI" src="https://cdn-images-1.medium.com/max/1024/1*-DO-osLV7TszAj9HFs7-HA.png" /></figure><h3>How Do You Route Agent Tasks by Data Sensitivity?</h3><p>For most teams, I’d go with a hybrid approach. You don’t need to run everything locally or send every request to a hosted model. Instead, decide where each task should run based on the kind of data it handles.</p><p>Here’s a simple way to think about it:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WQXbZ0tv5Uws-Yoa-pzsLw.png" /></figure><p>The idea is simple: keep sensitive data inside your infrastructure and send only the minimum necessary context to external models.</p><p>You don’t even need a complicated routing system to get started. Many open-model servers, including<strong> vLLM and SIE</strong>, expose OpenAI-compatible endpoints. This means you can use the same <strong>OpenAI Python SDK</strong> to connect to both <strong>local </strong>and <strong>hosted models.</strong></p><p>Here’s a small example:</p><pre>from openai import OpenAI<br><br>local = OpenAI(<br>    base_url=&quot;http://inference.internal:8080/v1&quot;,<br>    api_key=&quot;not-needed&quot;<br>)<br>frontier = OpenAI()  # Uses OPENAI_API_KEY<br><br>def pick_client(task):<br>    if task.has_sensitive_data or not task.needs_frontier:<br>        return local<br>    return frontier</pre><p>That’s pretty much all you need for a basic routing setup. The code itself isn’t particularly complicated.</p><p><strong>The tricky part is deciding what actually counts as sensitive data (</strong>has_sensitive_data<strong>).</strong></p><p>A database record might look harmless on its own, but combine it with an <strong><em>email address, customer ID, or contract detail</em></strong>, and you’ve got information that shouldn’t leave your infrastructure.</p><p>That’s why data classification needs to happen before routing. Your agent should know what it’s handling before deciding which model gets to process it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*45BOMcUV1sXKWdWwGFA-nQ.png" /><figcaption><strong>Data Sensitivity-Based Routing for AI Agents:</strong> A simple decision-making workflow.</figcaption></figure><h3>Privacy Checklist for AI Agent Developers</h3><p>Before taking your AI agent to production, there are a few things worth checking. You don’t want to discover a data privacy issue during a security review, especially when most of these problems are easier to prevent early on.</p><p>Here’s a practical checklist you can run through:</p><blockquote><strong><em>▣ Map every model call</em></strong><em> your agent makes. Include embeddings, guardrails, and sub-agents. Don’t overlook the smaller calls.</em></blockquote><blockquote><strong><em>▣ Classify the data</em></strong><em> each call handles. Is it public, internal, confidential, or regulated?</em></blockquote><blockquote><strong><em>▣ Check the data retention policy</em></strong><em> for every hosted endpoint. Stateful endpoints can retain data longer than regular completion requests.</em></blockquote><blockquote><strong><em>▣ Apply for Zero Data Retention (ZDR)</em></strong><em> wherever you’re eligible and still need to send data to external providers.</em></blockquote><blockquote><strong><em>▣ Verify the processing region</em></strong><em> to make sure it meets your data residency requirements.</em></blockquote><blockquote><strong><em>▣ Redact Personally Identifiable Information (PII)</em></strong><em> before data leaves your infrastructure, not after.</em></blockquote><blockquote><strong><em>▣ Review your logs, traces, and evaluation datasets.</em></strong><em> They can contain real prompts and sensitive information too.</em></blockquote><blockquote><strong><em>▣ Limit tool permissions.</em></strong><em> Give each tool access only to what it needs. Read-only database access is a good starting point. For example, in the </em><a href="https://superlinked.com/blog/contract-review-agent"><em>contract review agent </em></a><em>workflow discussed earlier, the text-to-SQL component is restricted to SELECT queries, preventing it from modifying database records.</em></blockquote><blockquote><strong><em>▣ Pin model versions</em></strong><em> so unexpected provider updates don’t change your agent’s behavior.</em></blockquote><blockquote><strong><em>▣ Disable vendor telemetry</em></strong><em> in air-gapped or restricted environments.</em></blockquote><blockquote><strong><em>▣ Test open models on your own data</em></strong><em> before moving any workload away from a hosted provider.</em></blockquote><blockquote><strong><em>▣ Review your setup regularly</em></strong><em>, especially when a provider changes its data retention or privacy policies.</em></blockquote><p>You don’t need to treat this as a one-time exercise. Your agent will evolve, new tools will get added, and providers will update their policies. Your privacy review needs to keep up.</p><h3>When Does a Hosted API Still Make Sense?</h3><p>By now, you might be thinking that self-hosting is always the safer option. But that’s not necessarily true.</p><p>Running an open model on your own infrastructure doesn’t automatically make your data more private. A poorly configured cluster with exposed logs can be less secure than a properly managed hosted API with Zero Data Retention.</p><p>There are still plenty of situations where a hosted API makes more sense:</p><ul><li><strong>You need a frontier model.</strong> Your task requires capabilities that open models haven’t been able to match in your own evaluations.</li><li><strong>Your workload is small or unpredictable.</strong> Running your own GPU infrastructure might not make financial sense if it sits idle most of the day.</li><li><strong>You don’t have an infrastructure team.</strong> Self-hosting means taking care of capacity, upgrades, security patches, and incident response.</li><li><strong>The provider already meets your security requirements.</strong> For example, it offers ZDR and regional processing that satisfy your organization’s policies.</li></ul><p>This is an important distinction when comparing hosted APIs with self-hosted inference. Superlinked makes a similar point in its <a href="https://superlinked.com/blog/sie-vs-openai">SIE vs. OpenAI comparison</a> too. When you self-host, you also take full responsibility for protecting the data, managing access controls, and maintaining the infrastructure. There’s no external provider managing those security controls for you.</p><p>So, don’t choose self-hosting just because it sounds more private, or a hosted API just because it’s easier.</p><p><strong>Choose the deployment model that gives you the right balance of data protection, model quality, operational effort, and cost, and one your team can actually manage properly.</strong></p><h3>Skipping the Infrastructure Work</h3><p>If you’ve gone through the checklist above, you might be thinking: we want to run sensitive workloads on our own infrastructure, but we don’t really want to build and maintain an entire model-serving platform, that obviously introduces another layer of operational work.</p><p>That’s a fair concern. Managing inference infrastructure takes time, and not every team has the resources to do it from scratch.</p><p>One option in this space is <a href="https://superlinked.com/docs"><strong>SIE (Superlinked Inference Engine)</strong></a><strong>,</strong> an open-source inference server designed to run models within your own infrastructure.</p><p>It supports running encoders, rerankers, entity extractors, and generation models across different environments, from a local laptop to a production Kubernetes cluster.</p><p>It supports more than 100 open models through four core primitives: <strong>Encode, Score, Extract, and Generate</strong>.</p><p>For teams that want to experiment without setting up their own inference infrastructure, <a href="https://superlinked.com/serverless">SIE Serverless</a> provides a hosted alternative.</p><p>It offers a hosted version with SOC 2 Type 2 compliance and US or EU data residency. Just remember that it’s still a hosted service, so avoid sending sensitive data unless its processing and retention controls meet your requirements. You can start with non-sensitive workloads and move more sensitive ones to your own cloud.</p><p>For teams that prefer managing inference within their own cloud environment, <a href="https://superlinked.com/native-inference">Native Inference</a> is another deployment option.</p><h3>Frequently Asked Questions (FAQs)</h3><h4>◉ Is data privacy really the biggest barrier to enterprise AI adoption?</h4><p>It’s certainly one of the biggest. Several 2026 surveys from AvePoint, Okta, and KPMG identify data security and privacy as major concerns holding back enterprise AI and agent adoption.</p><h4>◉ Do OpenAI and Anthropic use API data to train their models?</h4><p>Neither provider uses API data for model training by default. However, both have data retention policies that can allow inputs and outputs to be retained for up to 30 days, subject to specific exceptions. Zero Data Retention arrangements are available under certain conditions.</p><h4>◉ Is self-hosting automatically more private?</h4><p>No. Self-hosting gives you more control over where your data is processed, but it also makes you responsible for access controls, logging, and data retention. A poorly secured self-hosted setup can be just as risky as an external API.</p><h4>◉ Can I use frontier models and still pass an enterprise privacy review?</h4><p>Yes. You can still use hosted frontier models without exposing unnecessary sensitive information. Redact PII before sending data, apply for Zero Data Retention where eligible, and share only the minimum context the model needs to complete its task.</p><h3>References and Citations (as used in this article)</h3><p>▢ <a href="https://www.avepoint.com/shifthappens/reports/artificial-intelligence-report-2026">AvePoint State of AI 2026</a></p><p>▢ <a href="https://www.avepoint.com/blog/strategy-blog/the-state-of-ai-2026-security-insights-cisos-need-to-know">AvePoint’s The State of AI 2026: Security Insights CISOs Need to Know</a></p><p>▢ <a href="https://www.okta.com/newsroom/articles/enterprise-buyer-survey-ai-agent-security/">Okta enterprise buyer survey</a></p><p>▢ <a href="https://kpmg.com/kpmg-us/content/dam/kpmg/pdf/2026/ai-quarterly-pulse-survey-q1-2026.pdf">KPMG AI Pulse Survey, Q1 2026</a></p><p>▢ <a href="https://www.ibm.com/reports/data-breach">IBM Cost of a Data Breach 2026</a></p><p>▢ <a href="https://www.cisco.com/c/en/us/about/trust-center/data-privacy-benchmark-study.html">Cisco Data and Privacy Benchmark Study 2026</a></p><p>▢ <a href="https://developers.openai.com/api/docs/guides/your-data">OpenAI API: Data controls in the OpenAI platform</a></p><p>▢ <a href="https://privacy.claude.com/en/articles/7996866-how-long-do-you-store-my-organization-s-data">Anthropic API: How long do you store my organization’s data?</a></p><p>▢ <a href="https://superlinked.com/docs">Superlinked Docs</a>, <a href="https://superlinked.com/blog/sie-vs-openai">Superlinked SIE vs OpenAI</a>, <a href="https://superlinked.com/serverless">Superlinked Serverless</a>, <a href="https://superlinked.com/native-inference">Superlinked Native Inference</a>, <a href="https://superlinked.com/blog/contract-review-agent">Superlinked Contract Review Agent</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=61d0c4627c31" width="1" height="1" alt=""><hr><p><a href="https://medium.com/towards-data-engineering/the-hidden-data-privacy-risks-behind-enterprise-ai-agents-61d0c4627c31">The Hidden Data Privacy Risks Behind Enterprise AI Agents</a> was originally published in <a href="https://medium.com/towards-data-engineering">Towards Data Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Genie Ontology Learns on Its Own. Here’s the App I’d Build to Keep It Honest.]]></title>
            <link>https://medium.com/towards-data-engineering/genie-ontology-learns-on-its-own-heres-the-app-i-d-build-to-keep-it-honest-f9a3ff2948e5?source=rss----37f58dd42be7---4</link>
            <guid isPermaLink="false">https://medium.com/p/f9a3ff2948e5</guid>
            <category><![CDATA[ontology]]></category>
            <category><![CDATA[ontology-manager]]></category>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[THE BRICK LEARNING]]></dc:creator>
            <pubDate>Tue, 17 Nov 2026 14:24:20 GMT</pubDate>
            <atom:updated>2026-11-17T14:24:21.909Z</atom:updated>
            <content:encoded><![CDATA[<h4>New in The Databricks Now — Pro: a whitepaper and concept screens for an Ontology Steward Workbench on Databricks</h4><p>Last week I wrote about Genie Ontology going GA, and about why Ontology Manager matters more than the headline. This week I wanted to go one step further and ask a practical question:</p><p><strong>If I were a data steward tomorrow, what tool would I actually want?</strong></p><p><strong>The uncomfortable truth about a self-learning ontology.</strong></p><p>Genie Ontology learns business context from your dashboards, SQL, notebooks and BI models. That’s powerful. But spend some time in the docs and three facts stand out:</p><ul><li><strong>You can’t edit a learned snippet directly.</strong> You fix the source asset, or define the concept yourself as a governed Page.</li><li><strong>There’s no changelog of what it learned.</strong> You can see which sources informed a single answer, but there’s no diff of what changed in the ontology over time, and no public export.</li><li><strong>Ontology Manager isn’t shipped yet.</strong> It’s announced as coming soon in Beta.</li></ul><p>So when Genie gives a wrong answer about “net revenue”, the steward is left with a hard question: <em>where do I even fix this?</em></p><p><strong>The idea: an Ontology Steward Workbench</strong></p><p>A Databricks App, built only on public APIs, that treats the ontology as something you <strong>test from the outside</strong>. It asks Genie a fixed set of probe questions every night, records what it answered and what it cited, and flags anything that drifts or disagrees.</p><p>It has six modules:</p><ul><li><strong>Health:</strong> a scorecard per domain, with the highest-impact fixes one click away</li><li><strong>Probe:</strong> scheduled questions per business concept</li><li><strong>Conflicts and gaps:</strong> every definition of a term, side by side, with an explanation of how they differ</li><li><strong>Fix Studio:</strong> routes each problem to the right fix (draft a Page, certify or deprecate an asset, restrict a stale one, or ticket the owner)</li><li><strong>Regression Lab:</strong> golden questions run before and after every fix</li><li><strong>Changelog:</strong> the diff of what Genie believes that the platform doesn’t give you today</li></ul><p>My favourite screen shows the trap a self-learning ontology walks into. <strong>The most-cited definition is not the correct one.</strong> A dashboard used by 41 assets quietly outranks the certified metric view. The Workbench surfaces that, suggests the fix, and then <em>proves</em> the fix worked with a before-and-after test run, including any question it broke along the way.</p><p><a href="https://topmate.io/akash_wadhankar/2404189">Latest freshly brewed... with Akash A Wadhankar | Topmate</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aGzo8qEYt2TOGYdh5JXBDQ.png" /></figure><p><strong>What’s in the whitepaper</strong></p><ul><li>The six failure modes of a learned ontology, and where each fix actually lives</li><li>Personas mapped to real Unity Catalog roles (curators, Page managers, owners)</li><li><strong>Four art-of-the-possible app screens</strong></li><li>A reference architecture: Databricks Apps with on-behalf-of-user auth, Lakebase, Genie APIs, the Pages REST API and system tables</li><li>A 10-table data model</li><li>The detect, triage, fix, test, publish loop</li><li>A four-phase build plan, where phase 1 needs no Genie APIs at all</li><li>Build vs wait: what to build now and what to leave to Ontology Manager</li><li>Nine risks, including the one open question to prove before you start</li></ul><p>To be clear, this is a design paper and the screens are concepts, not a shipped product. Every API it relies on is listed, and the one I couldn’t confirm is called out plainly.</p><p><strong>Join The Databricks Now — Pro</strong></p><p><em>Get the latest freshly brewed knowledge on Databricks.</em></p><p>Built for architects and leads: every new Databricks feature, simplified, with a point of view and the gotchas spelled out. This whitepaper sits alongside the Genie Ontology &amp; Ontology Manager paper and the rest of the library.</p><p><a href="https://topmate.io/akash_wadhankar/2404189">Latest freshly brewed... with Akash A Wadhankar | Topmate</a></p><p><em>Would you build this, or wait for Ontology Manager? Tell me in the comments.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f9a3ff2948e5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/towards-data-engineering/genie-ontology-learns-on-its-own-heres-the-app-i-d-build-to-keep-it-honest-f9a3ff2948e5">Genie Ontology Learns on Its Own. Here’s the App I’d Build to Keep It Honest.</a> was originally published in <a href="https://medium.com/towards-data-engineering">Towards Data Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Databricks Just Gave Its Agents a Brain for Business Context.]]></title>
            <link>https://medium.com/towards-data-engineering/databricks-just-gave-its-agents-a-brain-for-business-context-31a5424d6fc4?source=rss----37f58dd42be7---4</link>
            <guid isPermaLink="false">https://medium.com/p/31a5424d6fc4</guid>
            <category><![CDATA[ontology]]></category>
            <category><![CDATA[ontology-manager]]></category>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[THE BRICK LEARNING]]></dc:creator>
            <pubDate>Tue, 17 Nov 2026 14:01:39 GMT</pubDate>
            <atom:updated>2026-11-17T14:01:41.177Z</atom:updated>
            <content:encoded><![CDATA[<h3>Databricks Just Gave Its Agents a Brain for Business Context. Ontology Manager Is How You Keep It Honest.</h3><h4><strong>Subtitle:</strong> New in The Databricks Now — Pro: an architect’s whitepaper on Genie Ontology and Ontology Manager</h4><p>On 11 November 2026, Databricks made Genie Ontology generally available. Most of the coverage has focused on the ontology. I think the more important announcement is the one still marked “coming soon”: <strong>Ontology Manager</strong>.</p><p>Here’s why.</p><p><strong>Agents can write SQL. They still don’t know what your business means.</strong></p><p>Ask an agent for “net revenue” and it has to guess. Which of the five tables? Which of the three dashboards? Which of the two formulas Finance and Sales disagree on? RAG doesn’t fix this, because it retrieves what looks similar. It can’t tell you what is authoritative.</p><p>Genie Ontology goes after this directly. It learns business context from work already happening in your workspace: dashboards, SQL, notebooks, Power BI models, documents, even your org chart. A ranking algorithm called OntoRank then decides which definition wins. Every agent reads from that one layer, including external agents like Claude, ChatGPT and Copilot through Genie MCP.</p><p>On Databricks’ own benchmark, Genie with the ontology passed 91% of tasks. The best frontier agent managed 58%, while using more than twice the tokens.</p><p><strong>But here’s the catch.</strong></p><p>An ontology that builds itself learns what’s <em>popular</em>, not what’s <em>correct</em>. A flawed formula copied into twenty dashboards can outrank the right one sitting in a single notebook.</p><p>That’s the gap Ontology Manager fills. It’s the human control point:</p><ul><li><strong>See</strong> what the ontology knows and how healthy it is</li><li><strong>Find</strong> gaps, such as heavily used concepts with no governed definition</li><li><strong>Promote</strong> learned knowledge into official, owned Unity Catalog Pages</li></ul><p>In short, agents write the draft and your stewards sign it off.</p><p><strong>What’s in the new whitepaper</strong></p><p>I’ve added a detailed architect’s whitepaper on this to <strong>The Databricks Now — Pro</strong>. Inside:</p><ul><li>A reference architecture diagram showing how sources, OntoRank, Pages and Ontology Manager fit together</li><li>What is GA, what is Beta and what is still coming, so you don’t over-promise to stakeholders</li><li>A curation operating model: roles, review loop and cadence</li><li>Seven use cases mapped to real customer outcomes</li><li>A critical read of the benchmark: what it proves, and what it doesn’t</li><li>How it compares with Palantir, Microsoft Fabric IQ and Snowflake</li><li>Where it sits on the ontology maturity curve, and what’s still missing</li><li>Nine gotchas I’d want every architect to know before rollout</li><li>A readiness checklist and a five-step adoption roadmap</li></ul><p>The biggest takeaway: most of the work that makes Genie Ontology succeed is governance you can start <strong>today</strong>, before Ontology Manager ships.</p><p><strong>Join The Databricks Now — Pro</strong></p><p><a href="https://topmate.io/akash_wadhankar/community">Member posts by Akash Wadhankar | Topmate</a></p><p><em>Get the latest freshly brewed knowledge on Databricks.</em></p><p>The membership is built for architects and leads. Each new Databricks feature arrives simplified, with a clear point of view and the gotchas spelled out. This whitepaper joins the existing library, which includes:</p><ul><li>Unity Catalog</li><li>Databricks Genie vs Snowflake Cortex Analyst</li><li>Declarative Pipelines vs dbt on Snowflake</li><li>Metric views vs Snowflake semantic views</li><li>The Unity AI Gateway review</li></ul><p><a href="https://topmate.io/akash_wadhankar/community">Member posts by Akash Wadhankar | Topmate</a></p><p><em>What’s your take: will a self-learning ontology reduce governance work, or just move it? Tell me in the comments.</em></p><p><strong>Suggested tags:</strong> Databricks, Data Architecture, AI Agents, Data Governance, Unity Catalog</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=31a5424d6fc4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/towards-data-engineering/databricks-just-gave-its-agents-a-brain-for-business-context-31a5424d6fc4">Databricks Just Gave Its Agents a Brain for Business Context.</a> was originally published in <a href="https://medium.com/towards-data-engineering">Towards Data Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[First, can you really become a Data Engineer in 90 days?]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/towards-data-engineering/first-can-you-really-become-a-data-engineer-in-90-days-08f52005fab9?source=rss----37f58dd42be7---4"><img src="https://cdn-images-1.medium.com/max/2600/1*6SfSw5PQYtx7ydgWZMZlPQ.png" width="2752"></a></p><p class="medium-feed-snippet">90 Days to Become a Production-Minded Data Engineer: The Nov &#x2192; Feb Roadmap (12 Weeks, 12 Projects)</p><p class="medium-feed-link"><a href="https://medium.com/towards-data-engineering/first-can-you-really-become-a-data-engineer-in-90-days-08f52005fab9?source=rss----37f58dd42be7---4">Continue reading on Towards Data Engineering »</a></p></div>]]></description>
            <link>https://medium.com/towards-data-engineering/first-can-you-really-become-a-data-engineer-in-90-days-08f52005fab9?source=rss----37f58dd42be7---4</link>
            <guid isPermaLink="false">https://medium.com/p/08f52005fab9</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[B V Sarath Chandra]]></dc:creator>
            <pubDate>Tue, 17 Nov 2026 13:55:23 GMT</pubDate>
            <atom:updated>2026-11-17T13:55:22.161Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Your SQL Works on 1,000 Rows. Will It Survive 100 Million?]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/towards-data-engineering/your-sql-works-on-1-000-rows-will-it-survive-100-million-a0111e78d30f?source=rss----37f58dd42be7---4"><img src="https://cdn-images-1.medium.com/max/2600/1*otGb9uxd_71TLU5BEnapWA.png" width="2752"></a></p><p class="medium-feed-snippet">The query that takes 0.2 seconds on your laptop might take 20 minutes in production. Here&#x2019;s what changes when your data grows, and how to&#x2026;</p><p class="medium-feed-link"><a href="https://medium.com/towards-data-engineering/your-sql-works-on-1-000-rows-will-it-survive-100-million-a0111e78d30f?source=rss----37f58dd42be7---4">Continue reading on Towards Data Engineering »</a></p></div>]]></description>
            <link>https://medium.com/towards-data-engineering/your-sql-works-on-1-000-rows-will-it-survive-100-million-a0111e78d30f?source=rss----37f58dd42be7---4</link>
            <guid isPermaLink="false">https://medium.com/p/a0111e78d30f</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[B V Sarath Chandra]]></dc:creator>
            <pubDate>Tue, 17 Nov 2026 13:55:11 GMT</pubDate>
            <atom:updated>2026-11-17T13:55:09.987Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Building a Snowflake Production Operations Toolkit: 5 SQL Patterns Every Data Engineer Should Know]]></title>
            <link>https://medium.com/towards-data-engineering/building-a-snowflake-production-operations-toolkit-5-sql-patterns-every-data-engineer-should-know-e08610d9cda5?source=rss----37f58dd42be7---4</link>
            <guid isPermaLink="false">https://medium.com/p/e08610d9cda5</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[snowflake-data-cloud]]></category>
            <category><![CDATA[snowflake-computing]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Thiru Gankidi]]></dc:creator>
            <pubDate>Tue, 17 Nov 2026 13:55:00 GMT</pubDate>
            <atom:updated>2026-11-17T13:54:59.037Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>Using SQL, Account Usage views and generated data we can build a practical toolkit for monitoring Snowflake. Let’s take a look.</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xzdGiruoalHtA8lTaJS1Ug.png" /></figure><h4>Introduction</h4><p>Whilst building data pipelines on Snowflake is a key challenge, ensuring they are reliable, secure, fast and cost effective is also important.</p><p>As your Snowflake environment grows you will face problems such as:<br>Why has my warehouse credit increased suddenly?<br>What queries are causing performance issues?<br>What tasks are failing regularly?<br>Are sensitive permissions being given to the correct roles?<br>How much are we using for Cortex AI?</p><p>Snowflake provides rich metadata within the ACCOUNT_USAGE schema. But it can be tricky to extract insights from this data. This typically involves running multiple queries, comparing historical data, and understanding what each metric represents.</p><p>I have created a set of 5 patterns to monitor your Snowflake environment using SQL. This open source tool is called the Snowflake Production Operations Toolkit.</p><p>This toolkit can help you identify issues with warehouse credit, query performance, task failures, access control and Cortex AI usage.</p><p><strong>GitHub Repository:</strong> <a href="https://github.com/thgankidi-stack/snowflake-production-operations-toolkit">https://github.com/thgankidi-stack/snowflake-production-operations-toolkit</a></p><h3>Why I Used Synthetic Data</h3><p>A problem I came across whilst creating this toolkit was that there was limited historical data available in the Snowflake trial environment.<br>If there is not much historical data then the monitor queries will return empty results.</p><p>To avoid wasting warehouse and AI usage, I created some mock tables to represent a realistic scenario.</p><p>This data is made up and is just to demonstrate how the monitoring would work. The real scripts will pull from the real system metadata tables in Snowflake. Let’s go through each of the monitoring patterns and see what they are doing.</p><h3>1. Warehouse Credit Spike Detective</h3><h3>The Problem</h3><p>One of the most common issues that can occur with a cloud data warehouse is unexpected compute credits usage.<br>This could be due to increased workload, warehouse size, number of clusters or changed execution patterns.<br>Looking at your total daily credits may not show you if there is any unusual usage.<br>A better way is to compare current credits usage with some historical data.</p><h3>The Approach</h3><p>The Warehouse Credit Spike Detective uses Snowflake’s WAREHOUSE_METERING_HISTORY view.</p><p>The SQL compares hourly warehouse compute credits against historical consumption for the same hour of day.</p><p>It then identifies hours where consumption exceeds twice the baseline.</p><h3>SQL Implementation</h3><pre>WITH hourly AS (<br>    SELECT<br>        warehouse_name,<br>        start_time,<br>        DATE_PART(&#39;hour&#39;, start_time) AS hour_of_day,<br>        credits_used_compute AS compute_credits<br>    FROM SNOWFLAKE.ACCOUNT_USAGE.WAREHOUSE_METERING_HISTORY<br>    WHERE start_time &gt;= DATEADD(day, -21, CURRENT_TIMESTAMP())<br>      AND start_time &lt; DATEADD(hour, -6, CURRENT_TIMESTAMP())<br>),<br>scored AS (<br>    SELECT<br>        warehouse_name,<br>        start_time,<br>        compute_credits,<br>        AVG(compute_credits) OVER (<br>            PARTITION BY warehouse_name, hour_of_day<br>            ORDER BY DATE_PART(epoch_second, start_time)<br>            RANGE BETWEEN 1209600 PRECEDING AND 1 PRECEDING<br>        ) AS baseline_credits,<br>        COUNT(*) OVER (<br>            PARTITION BY warehouse_name, hour_of_day<br>            ORDER BY DATE_PART(epoch_second, start_time)<br>            RANGE BETWEEN 1209600 PRECEDING AND 1 PRECEDING<br>        ) AS baseline_observations<br>    FROM hourly<br>)<br>SELECT<br>    warehouse_name,<br>    start_time,<br>    ROUND(compute_credits, 3) AS compute_credits,<br>    ROUND(baseline_credits, 3) AS baseline_credits,<br>    ROUND(<br>        compute_credits / NULLIF(baseline_credits, 0), 2<br>    ) AS spike_ratio<br>FROM scored<br>WHERE start_time &gt;= DATEADD(day, -7, CURRENT_TIMESTAMP())<br>  AND baseline_observations &gt;= 3<br>  AND baseline_credits &gt; 0<br>  AND compute_credits &gt;= baseline_credits * 2<br>ORDER BY spike_ratio DESC, start_time DESC;</pre><h3>Results from the Synthetic Dataset</h3><p>The demo results identified several unusually high credit consumption periods for ETL_WH.</p><p>MetricExample resultWarehouseETL_WHObserved credits8.00Historical baseline1.09Spike ratio7.34×</p><p>The synthetic warehouse consumed approximately 7.34 times its historical baseline.</p><p>This provides a clear signal that the warehouse deserves further investigation.</p><p><strong>Operational insight:</strong> An anomaly detection query should identify where to investigate, not automatically conclude why consumption increased.</p><p>A production investigation should also consider warehouse resizing, multi-cluster activity, workload changes, and query execution patterns.</p><h3>2. Expensive Query Investigator</h3><h3>The Problem</h3><p>Long-running queries can affect reporting performance, pipeline execution times, and warehouse utilization.</p><p>However, execution duration alone does not explain the underlying problem.</p><p>A query might be slow because it scans large volumes of data, spills intermediate results to remote storage, or waits for warehouse resources.</p><p>Understanding these signals helps engineering teams determine where optimization efforts should begin.</p><h3>The Approach</h3><p>The Expensive Query Investigator uses Snowflake’s QUERY_HISTORY view to examine execution metrics.</p><p>It focuses on:</p><ul><li>Total elapsed time</li><li>Execution time</li><li>Data scanned</li><li>Remote storage spilling</li><li>Warehouse queueing time</li></ul><h3>SQL Implementation</h3><pre>SELECT<br>    query_id,<br>    start_time,<br>    warehouse_name,<br>    user_name,<br>    query_type,<br>    ROUND(total_elapsed_time / 1000.0, 2)<br>        AS elapsed_seconds,<br>    ROUND(execution_time / 1000.0, 2)<br>        AS execution_seconds,<br>    ROUND(bytes_scanned / POWER(1024, 3), 2)<br>        AS scanned_gb,<br>    ROUND(<br>        bytes_spilled_to_remote_storage / POWER(1024, 3),<br>        2<br>    ) AS remote_spill_gb,<br>    ROUND(queued_overload_time / 1000.0, 2)<br>        AS queue_seconds,<br>    query_hash<br>FROM SNOWFLAKE.ACCOUNT_USAGE.QUERY_HISTORY<br>WHERE start_time &gt;= DATEADD(day, -7, CURRENT_TIMESTAMP())<br>  AND execution_status = &#39;success&#39;<br>  AND warehouse_name IS NOT NULL<br>ORDER BY total_elapsed_time DESC<br>LIMIT 50;</pre><h3>Results from the Synthetic Dataset</h3><p>The sample results include queries exhibiting substantial execution time and resource usage.</p><p>MetricExample resultElapsed time900 secondsData scanned100 GBRemote storage spill10 GBQueueing timeUp to 30 seconds</p><p>A query taking 900 seconds while spilling 10 GB to remote storage warrants investigation.</p><p>Remote spilling can indicate that intermediate processing exceeded available memory, although further examination of the query profile is necessary to establish the cause.</p><p>Queueing time can also indicate resource contention or warehouse concurrency limitations.</p><h3>Practical Recommendations</h3><p>When investigating slow queries, I typically consider the following questions:</p><ol><li>Is the query scanning significantly more data than expected?</li><li>Are joins or aggregations producing large intermediate datasets?</li><li>Is remote spilling occurring consistently?</li><li>Is queueing contributing materially to elapsed time?</li><li>Does the query profile suggest an SQL optimization or warehouse configuration issue?</li></ol><p><strong>Important distinction:</strong> The script identifies expensive-looking queries based on performance characteristics. It does not calculate actual query-level credit cost.</p><h3>3. Snowflake Task Failure Analyzer</h3><h3>The Problem</h3><p>Scheduled tasks are an important part of Snowflake pipeline orchestration.</p><p>In production environments, failures can occur because of upstream data availability, SQL errors, permission changes, or other execution problems.</p><p>A single failed task may be easy to investigate. Repeated failures across multiple pipelines are more difficult to manage without a consolidated operational view.</p><h3>The Approach</h3><p>The Task Failure Analyzer uses TASK_HISTORY to summarize execution outcomes over a 30-day period.</p><p>It calculates the total number of executions, failed executions, failure percentage, and latest failure information.</p><h3>SQL Implementation</h3><pre>SELECT<br>    database_name,<br>    schema_name,<br>    name AS task_name,<br>    COUNT(*) AS total_runs,<br>    COUNT_IF(state = &#39;FAILED&#39;) AS failed_runs,<br>    ROUND(<br>        100.0 * COUNT_IF(state = &#39;FAILED&#39;)<br>        / NULLIF(COUNT(*), 0),<br>        2<br>    ) AS failure_pct,<br>    MAX(<br>        IFF(state = &#39;FAILED&#39;, scheduled_time, NULL)<br>    ) AS latest_failure_time,<br>    MAX_BY(<br>        IFF(state = &#39;FAILED&#39;, error_message, NULL),<br>        IFF(state = &#39;FAILED&#39;, scheduled_time, NULL)<br>    ) AS latest_failure_message<br>FROM SNOWFLAKE.ACCOUNT_USAGE.TASK_HISTORY<br>WHERE scheduled_time &gt;= DATEADD(day, -30, CURRENT_TIMESTAMP())<br>  AND state IN (<br>      &#39;SUCCEEDED&#39;,<br>      &#39;FAILED&#39;,<br>      &#39;CANCELLED&#39;,<br>      &#39;SKIPPED&#39;<br>  )<br>GROUP BY 1, 2, 3<br>HAVING COUNT_IF(state = &#39;FAILED&#39;) &gt; 0<br>ORDER BY failed_runs DESC, latest_failure_time DESC;</pre><h3>Results from the Synthetic Dataset</h3><p>The demo includes three fictional ETL tasks.</p><p>Each task failed four times out of twenty executions.</p><p>A 20% failure rate is a strong indication that the task requires investigation, although the appropriate threshold depends on the workload’s reliability requirements.</p><h3>Operational Insight</h3><p>Failure rates are more useful when combined with business impact.</p><p>For example, a daily customer data load failing once may be more consequential than several failures in a noncritical development pipeline.</p><p>A future enhancement would be to add task criticality, ownership, recovery time, and downstream dependency information.</p><h3>4. Snowflake RBAC Access Auditor</h3><h3>The Problem</h3><p>Role-based access control is fundamental to operating Snowflake securely.</p><p>As organizations grow, permissions may be granted across multiple teams, applications, and service accounts.</p><p>Without periodic review, sensitive privileges can become difficult to track.</p><h3>The Approach</h3><p>The RBAC Access Auditor examines active grants in Snowflake’s GRANTS_TO_ROLES view.</p><p>It highlights selected sensitive privileges, including ownership, grant management, user creation, and role creation.</p><h3>SQL Implementation</h3><pre>SELECT<br>    grantee_name AS role_name,<br>    granted_on AS object_type,<br>    name AS object_name,<br>    privilege,<br>    grant_option,<br>    granted_by,<br>    created_on<br>FROM SNOWFLAKE.ACCOUNT_USAGE.GRANTS_TO_ROLES<br>WHERE deleted_on IS NULL<br>  AND granted_to = &#39;ACCOUNT ROLE&#39;<br>  AND (<br>      privilege IN (<br>          &#39;OWNERSHIP&#39;,<br>          &#39;MANAGE GRANTS&#39;,<br>          &#39;CREATE USER&#39;,<br>          &#39;CREATE ROLE&#39;<br>      )<br>      OR grant_option = TRUE<br>  )<br>ORDER BY<br>    role_name,<br>    object_type,<br>    object_name,<br>    privilege;</pre><h3>Results from the Synthetic Dataset</h3><p>The synthetic results identify several fictional roles with sensitive permissions.</p><p>RolePrivilegeDEMO_ADMINMANAGE GRANTSDEMO_ENGINEEROWNERSHIPDEMO_PLATFORMCREATE ROLEDEMO_SECURITYCREATE USER</p><p>These grants are not automatically security violations.</p><p>For example, a platform administration role may legitimately require elevated privileges.</p><p>The objective is to make potentially sensitive access visible so that it can be reviewed against organizational policies.</p><h3>Operational Insight</h3><p>Effective access governance requires more than identifying privileged grants.</p><p>Organizations should also evaluate role inheritance, business justification, separation of duties, and periodic access certification.</p><p>This script provides an initial review of direct grants. It does not recursively resolve the full role hierarchy.</p><h3>5. Snowflake Cortex AI Usage Monitor</h3><h3>The Problem</h3><p>As AI capabilities become more deeply integrated into enterprise data platforms, monitoring AI consumption is becoming increasingly important.</p><p>Teams need visibility into which AI functions and models are being used, how consumption changes over time, and where unexpected usage patterns may occur.</p><p>Traditional warehouse monitoring alone does not provide a complete picture of AI-related consumption.</p><h3>The Approach</h3><p>The Cortex AI Usage Monitor analyzes historical records from Snowflake’s CORTEX_AI_FUNCTIONS_USAGE_HISTORY view.</p><p>It summarizes function-level usage by date and model.</p><h3>SQL Implementation</h3><pre>SELECT<br>    DATE_TRUNC(&#39;day&#39;, start_time) AS usage_day,<br>    function_name,<br>    model_name,<br>    ROUND(SUM(credits), 4)<br>        AS ai_function_credits,<br>    COUNT(DISTINCT query_id)<br>        AS distinct_queries,<br>    COUNT(*) AS usage_records<br>FROM SNOWFLAKE.ACCOUNT_USAGE.CORTEX_AI_FUNCTIONS_USAGE_HISTORY<br>WHERE start_time &gt;= DATEADD(day, -30, CURRENT_TIMESTAMP())<br>  AND is_completed = TRUE<br>GROUP BY 1, 2, 3<br>ORDER BY<br>    usage_day DESC,<br>    ai_function_credits DESC;</pre><h3>Results from the Synthetic Dataset</h3><p>The demo contains fictional AI function consumption records for three example function categories.</p><p>FunctionSynthetic modelAI_COMPLETEdemo-llm-modelAI_CLASSIFYdemo-classification-modelAI_EMBEDdemo-embedding-model</p><p>The sample output shows daily aggregated usage, including simulated credit consumption and query counts.</p><p>Some generated records show increased credit usage, allowing the monitoring query to illustrate how consumption trends can be investigated.</p><h3>Operational Insight</h3><p>AI cost governance should evolve alongside AI adoption.</p><p>Useful monitoring capabilities include identifying consumption trends, comparing model usage, and investigating unexpected increases.</p><p>This example is limited to Cortex AI Functions usage. It should not be interpreted as a complete measure of all Snowflake AI spending.</p><h3>6. Making the Toolkit Reproducible with Synthetic Data</h3><p>One of my main objectives was to make the examples accessible to engineers who do not have a production Snowflake environment.</p><p>A new trial account may have limited query history, task executions, or AI consumption records.</p><p>To address this, the toolkit includes synthetic monitoring tables representing the same operational concepts as the production Account Usage views.</p><p>For example, a simplified warehouse credit dataset can be generated using Snowflake’s GENERATOR function.</p><pre>SELECT<br>    &#39;ETL_WH&#39; AS warehouse_name,<br>    DATEADD(<br>        hour,<br>        -SEQ4(),<br>        DATE_TRUNC(&#39;hour&#39;, CURRENT_TIMESTAMP())<br>    ) AS start_time,<br>    1.0 AS credits_used_compute<br>FROM TABLE(GENERATOR(ROWCOUNT =&gt; 24));</pre><p>This produces example hourly warehouse consumption records.</p><p>The full demo extends this pattern to generate multiple warehouses, historical observations, and intentionally abnormal consumption.</p><p>The same approach is used to simulate long-running queries, task failures, sensitive grants, and AI usage records.</p><p>The benefit is straightforward: engineers can test the monitoring logic without generating artificial production workloads.</p><p>The synthetic data is not actual Snowflake system history, and its credit values are not billing measurements.</p><h3>7. From SQL Scripts to a Production Monitoring Framework</h3><p>These five examples provide a starting point, but a mature monitoring solution requires more than individual SQL queries.</p><p>A practical evolution would involve four layers.</p><p><strong>Layer 1 — Metadata collection</strong></p><p>Collect relevant operational metadata from Snowflake Account Usage views.</p><p><strong>Layer 2 — Monitoring and analysis</strong></p><p>Apply SQL-based rules to detect unusual consumption, performance degradation, failures, or potentially sensitive access patterns.</p><p><strong>Layer 3 — Operational reporting</strong></p><p>Expose the results through dashboards or reporting interfaces so engineering teams can investigate trends.</p><p><strong>Layer 4 — Response and improvement</strong></p><p>Introduce ownership, notifications, investigation workflows, and documented remediation procedures.</p><p>A future version of this toolkit could use Streamlit in Snowflake to bring the monitoring results into a single operational dashboard.</p><p>It could also incorporate configurable thresholds and automated notifications for selected events.</p><p>These are potential enhancements rather than capabilities included in the initial SQL toolkit.</p><h3>8. Lessons Learned</h3><p>Developing these examples reinforced several important principles.</p><p><strong>First, monitoring should focus on actionable signals.</strong> A large volume of operational metrics is not necessarily useful unless teams can identify what requires attention.</p><p><strong>Second, historical context matters.</strong> Comparing current consumption against a baseline provides more information than reporting isolated credit totals.</p><p><strong>Third, performance and cost are related but different.</strong> A long-running query does not automatically represent the highest credit consumption.</p><p><strong>Fourth, governance requires interpretation.</strong> Identifying privileged roles is the beginning of an access review, not proof of a security problem.</p><p><strong>Finally, reproducibility improves technical knowledge sharing.</strong> Synthetic datasets allow engineers to explore operational patterns even when they do not have access to production history.</p><h3>Conclusion</h3><p>Snowflake provides rich operational metadata, but extracting practical insights requires the right monitoring patterns.</p><p>The Snowflake Production Operations Toolkit brings together five SQL examples addressing common challenges in data platform operations:</p><ul><li>Warehouse credit spike detection</li><li>Query performance investigation</li><li>Task failure analysis</li><li>RBAC access auditing</li><li>Cortex AI usage monitoring</li></ul><p>By combining read-only production queries with synthetic demo data, the toolkit provides a practical foundation for learning, troubleshooting, and extending Snowflake operational monitoring.</p><p>The goal is not to replace enterprise monitoring platforms. It is to provide reusable SQL building blocks that data engineers and platform teams can understand, adapt, and improve.</p><p><strong>Explore the open-source toolkit:</strong></p><p>GitHub: <a href="https://github.com/thgankidi-stack/snowflake-production-operations-toolkit">https://github.com/thgankidi-stack/snowflake-production-operations-toolkit</a></p><p>Medium: <a href="https://medium.com/@thgankidi">https://medium.com/@thgankidi</a></p><p>LinkedIn: <a href="https://www.linkedin.com/in/thiru-gankidi/">https://www.linkedin.com/in/thiru-gankidi/</a></p><p>If you work with Snowflake, I’d be interested to hear which operational monitoring challenges you encounter most frequently and what additional SQL patterns would be useful.</p><p><em>Disclaimer: All screenshots and example results in this article use synthetic monitoring data. The production SQL examples are provided for educational and operational reference and should be validated against the applicable Snowflake metadata schema, privileges, and account configuration before production use.</em></p><p><strong>Tags:</strong> Snowflake, Data Engineering, SQL, DataOps, Cloud Data Warehousing</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e08610d9cda5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/towards-data-engineering/building-a-snowflake-production-operations-toolkit-5-sql-patterns-every-data-engineer-should-know-e08610d9cda5">Building a Snowflake Production Operations Toolkit: 5 SQL Patterns Every Data Engineer Should Know</a> was originally published in <a href="https://medium.com/towards-data-engineering">Towards Data Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Media Agencies can scale multi-tenant, Advertiser data platforms without sacrificing client…]]></title>
            <link>https://medium.com/towards-data-engineering/how-media-agencies-can-scale-multi-tenant-advertiser-data-platforms-without-sacrificing-client-228b0aa1416a?source=rss----37f58dd42be7---4</link>
            <guid isPermaLink="false">https://medium.com/p/228b0aa1416a</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[bigquery]]></category>
            <category><![CDATA[data-architecture]]></category>
            <category><![CDATA[medallion-architecture]]></category>
            <dc:creator><![CDATA[Feroz Mulla]]></dc:creator>
            <pubDate>Tue, 17 Nov 2026 13:15:54 GMT</pubDate>
            <atom:updated>2026-11-17T13:15:52.822Z</atom:updated>
            <content:encoded><![CDATA[<h3><em>How Media Agencies can scale multi-tenant, Advertiser data platforms without sacrificing client isolation</em></h3><p>Media and marketing agencies sit on one of the trickier architectural problems in data engineering. An agency data team often manages analytics, attribution, and media mix modeling across 10s or 100s of enterprise advertisers, each bringing feeds from Google Marketing Platform, third-party ad networks, and sensitive first-party CRM systems.</p><p>At one end of the spectrum, landing every client inside a single shared BigQuery (BQ) project and relying on logical table prefixes or row-level filters is asking for trouble. One misconfigured view or over-privileged service account can expose one advertiser’s performance data to another, noisy-neighbor queries compete for resources, and client offboarding turns into a surgical data deletion exercise.</p><p>At the other end of the spectrum, standing up standalone codebases, orchestrators, and pipelines per advertiser kills engineering velocity, increases operational overhead, and spikes ongoing costs. For anyone who has operated client data platforms long enough, maintaining several forked copies of the same SQL transformation pipeline quickly turns routine schema updates into weeks of repetitive toil.</p><p>The practical middle ground is to <strong>centralize control and orchestration while physically isolating storage, compute identity, and sharing per advertiser</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cPNBRyMH4K1Mp9i9L5LB9w.png" /><figcaption>Reference Architecture: Multi-tenant Agency-Advertisers Data Platform</figcaption></figure><p>Here is a breakdown of how to design and build this reference architecture on Google Cloud using BigQuery, Dataform, Cloud Workflows, and Terraform:</p><h3>1. Core Design Principle: Separate the Control Plane from the Data Plane</h3><p>To get both operational leverage and strict tenant boundaries, split the platform into two distinct project tiers.</p><h3>1.1 The Agency Admin Project (Control Plane)</h3><p>The central admin project holds zero advertiser business data. Instead, it hosts the shared machinery that one agency platform team operates across every client:</p><ul><li><strong>Control Plane Metadata (platform_admin dataset):</strong> Small BQ configuration and audit tables tracking active tenants, ingestion sources, SLA schedules, and pipeline run_log history.</li><li><strong>Orchestration:</strong> Cloud Scheduler, Eventarc, and Cloud Workflows to coordinate daily runs and retries. Managed Airflow (Cloud Composer) is another alternative when you need more complex orchestration capabilities, though its constant-uptime stateful compute can add to baseline costs.</li><li><strong>Transformation Codebase:</strong> A single Git-linked Dataform repository containing the parameterized Bronze to Silver to Gold SQLX models.</li><li><strong>Governance and Operations:</strong> Knowledge Catalog for cross-tenant lineage and glossary management, Cloud Monitoring for SLA alerting and FinOps, and a Cloud Build + Terraform tenant factory.</li></ul><h3>1.2 Dedicated Advertiser Projects (Isolated Data Planes)</h3><p>Every advertiser (advertiser-01, advertiser-02, up to advertiser-N) gets a dedicated Google Cloud project provisioned from an identical Terraform template. Each tenant project owns its Cloud Storage landing bucket, BigLake connection, bronze, silver, and gold BQ datasets, and a dedicated execution service account (sa-dataform@advertiser-01).</p><ul><li><strong>Good to know:</strong> Using a Google Cloud project as the tenant boundary gives you four enterprise requirements out of the box: hard IAM isolation, independent project quotas, clean per-advertiser cloud billing exports for FinOps chargeback, and straightforward client offboarding (revoke the sharing listing and delete the project).</li></ul><h3>2. Single-Repo Medallion Pipelines with Dataform</h3><p>Maintaining a consistent data model across dozens or hundreds of advertisers requires keeping transformation logic in a single repository while ensuring execution stays scoped to each client’s project.</p><h3>2.1 One SQLX Codebase, Compiled per Advertiser</h3><p>Dataform separates code authoring from compilation. Instead of hardcoding project IDs inside SQL queries, every Bronze source declaration and downstream Silver or Gold model references ${dataform.projectConfig.defaultDatabase} (along with optional vars.tenant_id compilation variables).</p><p>At runtime, the orchestrator calls the Dataform API to create a compilation result for a specific advertiser, overriding defaultDatabase with that advertiser’s GCP project ID (e.g. advertiser-01) and executing the compiled DAG as sa-dataform@advertiser-01. One pull request updates the transformation logic across all advertisers without duplicating a single SQL file.</p><h3>2.2 Bronze, Silver, and Gold Layers (With Early PII Hashing)</h3><p>Inside each advertiser project, data moves through three standardized BQ datasets:</p><ul><li><strong>Bronze (bronze):</strong> Raw, append-only tables matching source feeds as received. First-party CSV or Parquet drops in the advertiser’s GCS landing bucket are exposed via BigLake external tables, alongside native tables landed by BigQuery Data Transfer Service (for Google Ads, CM360, DV360, SA360, and GA4) and Cloud Run API connectors (for Meta, TikTok, Amazon Ads, and The Trade Desk).</li><li><strong>Silver (silver):</strong> Cleansed, deduplicated, and conformed native BQ tables (or Lakehouse Iceberg tables) with standardized currencies, time zones, and channel taxonomies. Dataform assertions (nonNull, uniqueKey, and rowConditions) validate data hygiene on every run. First-party CRM identifiers such as email addresses and phone numbers are normalized and hashed with SHA256 at the Bronze-to-Silver boundary so cleartext PII never enters Silver or Gold.</li><li><strong>Gold (gold):</strong> Business-ready marts (as native BQ tables or Lakehouse Iceberg tables) partitioned by date and clustered where supported, powering cross-channel performance, budget pacing, multi-touch attribution, and media mix modeling (MMM) features.</li><li><strong>Good to know:</strong> When a centralized Dataform repository in the Admin project executes a workflow invocation using a custom tenant service account (sa-dataform@advertiser-01), Dataform submits the BQ job via the host repository project while reading and writing tables in advertiser-01. Grant the tenant service account roles/bigquery.jobUser (bigquery.jobs.create only, with zero dataset access) on the Admin project, and verify constraints/iam.disableCrossProjectServiceAccountUsage is configured appropriately for your organization.</li></ul><h3>References</h3><ul><li><a href="https://docs.cloud.google.com/dataform/docs/configure-compilation">Configure Dataform compilation overrides and release configurations</a></li><li><a href="https://docs.cloud.google.com/dataform/docs/test-data">Test tables with Dataform assertions</a></li><li><a href="https://docs.cloud.google.com/bigquery/docs/biglake-intro">BigLake external tables on Cloud Storage</a></li></ul><h3>3. Metadata-Driven Orchestration with Cloud Workflows</h3><p>When SQL transformations push down into BigQuery via Dataform and standard media ingestion runs via BQ DTS or serverless jobs, your orchestration layer primarily coordinates API calls, retries, and audit logging. Cloud Workflows paired with Cloud Scheduler handles this lightweight control loop with serverless pay-per-step pricing.</p><h3>3.1 Parent-Child Parallel Fan-Out</h3><p>The orchestration plane uses a two-level workflow design (Flow A in the architecture diagram):</p><ol><li><strong>Parent Orchestrator Workflow:</strong> Triggered by Cloud Scheduler per SLA tier (or Eventarc on file arrival), the parent workflow queries platform_admin.tenants WHERE is_active = TRUE and runs a parallel loop across active advertisers.</li><li><strong>Per-Tenant Child Workflow:</strong> Invoked once per advertiser with runtime parameters (tenant_id, gcp_project_id, landing_bucket, dataform_sa). It writes a RUNNING entry to platform_admin.run_log, triggers ingestion checks, compiles the central Dataform repo with tenant overrides, polls the Dataform workflow invocation running under sa-dataform@&lt;tenant&gt;, and closes out the run log with SUCCEEDED or FAILED status.</li></ol><h3>3.2 Keeping Both Workflows in the Admin Project</h3><p>A common design question is whether the Child Workflow itself should be deployed inside each advertiser project or kept alongside the Parent Workflow in the Agency Admin project.</p><p>Keeping both workflow definitions in the Admin project is cleaner in practice. First, a single parameterized Child Workflow definition can serve 10 or 100 tenants without redeploying YAML state machines into every client project. Second, the Child Workflow has to call two resources owned by the Admin project: the shared Dataform repository and the platform_admin.run_log table. Hosting the orchestration state machine in the Admin project avoids granting advertiser service accounts reverse write permissions into the central control plane, while all data-plane reads and writes in BQ and GCS still execute strictly as sa-dataform@&lt;advertiser-project&gt;.</p><p><strong>Good to know:</strong> Isolate failure handling inside the Child Workflow’s try / except block so a malformed source feed or failed assertion in advertiser-01 marks only that tenant’s run as FAILED and alerts the ops channel without blocking advertiser-02 through advertiser-N.</p><h3>References</h3><ul><li><a href="https://docs.cloud.google.com/workflows/docs/reference/syntax/parallel-steps">Execute parallel steps in Cloud Workflows</a></li><li><a href="https://docs.cloud.google.com/workflows/docs/schedule-workflow">Schedule Workflows with Cloud Scheduler</a></li></ul><h3>4. Zero-Copy Sharing, Open Lakehouse Federation, and Agent Readiness</h3><p>Modern advertisers rarely treat an agency’s BI portal as the final destination for their data. Enterprise clients typically run their own cloud data platforms (BigQuery, Snowflake, Databricks, etc.) and want direct, governed access to their curated Gold tables.</p><h3>4.1 BigQuery Sharing (Analytics Hub) and Lakehouse Iceberg Tables</h3><p>To support client consumption without building brittle SFTP or CSV export pipelines, the serving layer exposes Gold marts through two complementary patterns:</p><ul><li><strong>BigQuery Sharing (Analytics Hub):</strong> Each advertiser project publishes a dedicated Data Exchange and Listing pointing to its gold views or marts. Subscribers using BigQuery link the dataset read-only and zero-copy into their own GCP organization, paying for their own query compute while the agency retains instant revocation control.</li><li><strong>Lakehouse Iceberg Tables:</strong> For advertisers consuming data from external or cross-cloud engines (such as Spark, Trino, Databricks, or Snowflake), curated tables can be stored as Lakehouse Apache Iceberg tables backed by Google Cloud’s Lakehouse runtime catalog. Underlying Parquet data files and Iceberg metadata (metadata.json, manifest lists, and manifest files) live directly in the advertiser’s GCS bucket, while the standard Apache Iceberg REST catalog endpoint handles snapshot discovery and multi-engine interoperability without duplicating transformation logic.</li><li><strong>Good to know:</strong> Pairing Lakehouse Iceberg tables with the Apache Iceberg REST catalog endpoint and credential vending lets authorized external engines access Parquet files in the advertiser’s GCS bucket using short-lived vended tokens rather than granting broad, long-lived storage IAM permissions.</li></ul><h3>4.2 Preparing the Data Plane for BQML and Data Agents</h3><p>Standardizing Gold schemas across tenants pays dividends beyond BI reporting. Because every advertiser’s gold dataset follows the same conformed schema and semantic conventions inside its own isolated project, agencies can develop BQML pipelines (for audience propensity, customer lifetime value, or media mix modeling) once and deploy them per client, writing scores straight back to gold.</p><p>The same principle applies as agencies roll out conversational data agents (e.g. BQ native agents surfaced through Gemini Enterprise or orchestrated via ADK). Grounding each client’s agent instance on their isolated gold dataset and tenant-scoped identity ensures the agent has clean semantic context while eliminating the risk of cross-client data leakage during multi-step reasoning.</p><h3>References</h3><ul><li><a href="https://docs.cloud.google.com/bigquery/docs/analytics-hub-introduction">Introduction to BigQuery sharing (Analytics Hub)</a></li><li><a href="https://docs.cloud.google.com/lakehouse/docs/introduction">What is borderless Lakehouse?</a></li><li><a href="https://docs.cloud.google.com/lakehouse/docs/about-lakehouse-catalogs">About the Lakehouse runtime catalog and Apache Iceberg REST endpoint</a></li></ul><h3>5. The Tenant Factory: Onboarding Advertiser N+1 in Minutes</h3><p>A multi-tenant architecture is only as scalable as its Day-2 onboarding and security model. Adding a new client should be a declarative configuration change rather than an engineering project.</p><h3>5.1 GitOps Onboarding with Terraform (Flow B)</h3><p>In the tenant factory pattern, onboarding advertiser-03 requires zero changes to Dataform SQLX files or Cloud Workflows definitions:</p><ol><li><strong>Declare:</strong> Open a pull request adding the new advertiser block (project_id, display_name, region) to terraform.tfvars.</li><li><strong>Provision:</strong> Cloud Build runs Terraform (tenant_factory module) to enable APIs and create the GCS landing bucket, BigLake connection, bronze/silver/gold datasets, sa-dataform service account, least-privilege IAM bindings, and Analytics Hub exchange.</li><li><strong>Register:</strong> Insert the new advertiser’s metadata row into platform_admin.tenants (and configure any feed-specific DTS transfers).</li><li><strong>Run:</strong> On the next scheduled trigger, medallion-parent-orchestrator automatically discovers the new row and executes its end-to-end medallion pipeline in parallel with existing clients.</li></ol><h3>5.2 Modular Security Guardrails (Flow C)</h3><p>In regulated industries such as financial services, healthcare, or retail, project IAM alone is often paired with deeper defense-in-depth controls:</p><ul><li><strong>VPC Service Controls (VPC-SC):</strong> Wrap each advertiser project in its own service perimeter to prevent unauthorized data exfiltration, with a narrow ingress rule allowing only the Admin project’s orchestrator and Dataform service agents to trigger jobs.</li><li><strong>Customer-Managed Encryption Keys (CMEK):</strong> Provision per-advertiser Cloud KMS keys so each client’s GCS buckets and BQ datasets are encrypted with keys that can be rotated or revoked independently.</li><li><strong>Catalog Governance:</strong> Use Knowledge Catalog (Dataplex) policy tags to enforce column-level access controls on sensitive attributes and aggregate data lineage centrally across tenants.</li><li><strong>Good to know:</strong> Keep infrastructure guardrails like VPC-SC perimeters, CMEK, and third-party connectors behind modular Terraform feature flags (e.g. enable_vpc_sc, enable_cmek). This allows engineers to stand up and test functional medallion pipelines quickly in sandbox environments without getting blocked on organization-level perimeter policies, while keeping production hardening codified in the same repository.</li></ul><h3>Conclusion</h3><p>To state the obvious, multi-tenant data architecture is not a one-size-fits-all decision. An early-stage shop with three small clients may get by with simpler setups, while a global agency managing dozens or hundreds of enterprise brands has to treat isolation, auditability, and onboarding speed as first-class design constraints.</p><p>By pairing a thin, metadata-driven control plane with dedicated per-advertiser data projects, single-repo Dataform SQLX, and zero-copy BQ and Lakehouse Iceberg sharing, a single platform team can scale across dozens or hundreds of clients without trading away governance or engineering agility.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=228b0aa1416a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/towards-data-engineering/how-media-agencies-can-scale-multi-tenant-advertiser-data-platforms-without-sacrificing-client-228b0aa1416a">How Media Agencies can scale multi-tenant, Advertiser data platforms without sacrificing client…</a> was originally published in <a href="https://medium.com/towards-data-engineering">Towards Data Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[I Compared DuckDB, Polars, DataFusion, and Apache Spark on the Same Dataset.]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/towards-data-engineering/i-compared-duckdb-polars-datafusion-and-apache-spark-on-the-same-dataset-747090c60047?source=rss----37f58dd42be7---4"><img src="https://cdn-images-1.medium.com/max/1408/1*eYgdKqgtnacdTVj2G7-uLg.jpeg" width="1408"></a></p><p class="medium-feed-snippet">I implemented identical ingestion, filtering, joins, aggregations, and analytical transformations across four processing engines. I&#x2026;</p><p class="medium-feed-link"><a href="https://medium.com/towards-data-engineering/i-compared-duckdb-polars-datafusion-and-apache-spark-on-the-same-dataset-747090c60047?source=rss----37f58dd42be7---4">Continue reading on Towards Data Engineering »</a></p></div>]]></description>
            <link>https://medium.com/towards-data-engineering/i-compared-duckdb-polars-datafusion-and-apache-spark-on-the-same-dataset-747090c60047?source=rss----37f58dd42be7---4</link>
            <guid isPermaLink="false">https://medium.com/p/747090c60047</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[polar]]></category>
            <category><![CDATA[duckdb]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Dr.Michael]]></dc:creator>
            <pubDate>Tue, 17 Nov 2026 01:13:01 GMT</pubDate>
            <atom:updated>2026-11-17T01:13:00.273Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[7 Python Libraries for Bloom Filters, Logs, Configs and Crawl Links (One Misses Its Target)]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/towards-data-engineering/7-python-libraries-for-bloom-filters-logs-configs-and-crawl-links-one-misses-its-target-1def7b1faa49?source=rss----37f58dd42be7---4"><img src="https://cdn-images-1.medium.com/max/1672/1*yYggEYpeUUhEol70_zq5uA.png" width="1672"></a></p><p class="medium-feed-snippet">Some Python libraries take a job that every project rebuilds and make it a few clean lines.</p><p class="medium-feed-link"><a href="https://medium.com/towards-data-engineering/7-python-libraries-for-bloom-filters-logs-configs-and-crawl-links-one-misses-its-target-1def7b1faa49?source=rss----37f58dd42be7---4">Continue reading on Towards Data Engineering »</a></p></div>]]></description>
            <link>https://medium.com/towards-data-engineering/7-python-libraries-for-bloom-filters-logs-configs-and-crawl-links-one-misses-its-target-1def7b1faa49?source=rss----37f58dd42be7---4</link>
            <guid isPermaLink="false">https://medium.com/p/1def7b1faa49</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Abdul Ahad]]></dc:creator>
            <pubDate>Tue, 17 Nov 2026 00:01:02 GMT</pubDate>
            <atom:updated>2026-11-17T00:01:02.489Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[6 Python Libraries for Markov Text, PO Files, CSS, Packets and Web Dates (One Fails Quietly)]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/towards-data-engineering/6-python-libraries-for-markov-text-po-files-css-packets-and-web-dates-one-fails-quietly-bb77b2dc57c8?source=rss----37f58dd42be7---4"><img src="https://cdn-images-1.medium.com/max/1672/1*CbUMCiczsmIBpcps0tNCDQ.png" width="1672"></a></p><p class="medium-feed-snippet">Some Python libraries read one awkward format so well that you never need to hand parse it again.</p><p class="medium-feed-link"><a href="https://medium.com/towards-data-engineering/6-python-libraries-for-markov-text-po-files-css-packets-and-web-dates-one-fails-quietly-bb77b2dc57c8?source=rss----37f58dd42be7---4">Continue reading on Towards Data Engineering »</a></p></div>]]></description>
            <link>https://medium.com/towards-data-engineering/6-python-libraries-for-markov-text-po-files-css-packets-and-web-dates-one-fails-quietly-bb77b2dc57c8?source=rss----37f58dd42be7---4</link>
            <guid isPermaLink="false">https://medium.com/p/bb77b2dc57c8</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[technology]]></category>
            <dc:creator><![CDATA[Hasnainraja]]></dc:creator>
            <pubDate>Mon, 16 Nov 2026 17:23:43 GMT</pubDate>
            <atom:updated>2026-11-16T17:23:41.911Z</atom:updated>
        </item>
    </channel>
</rss>