<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Snowflake Builders Blog: Data Engineers, App Developers, AI, &amp; Data Science - Medium]]></title>
        <description><![CDATA[Best practices, tips &amp; tricks from Snowflake experts and community - Medium]]></description>
        <link>https://medium.com/snowflake?source=rss----34b6daafc07---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Snowflake Builders Blog: Data Engineers, App Developers, AI, &amp;amp; Data Science - Medium</title>
            <link>https://medium.com/snowflake?source=rss----34b6daafc07---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 01:59:50 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/snowflake" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com" rel="hub"/>
        <item>
            <title><![CDATA[From Streamlit to Snowflake App Runtime: A Demo Migration]]></title>
            <link>https://medium.com/snowflake/from-streamlit-to-snowflake-app-runtime-a-demo-migration-19e6e81d9873?source=rss----34b6daafc07---4</link>
            <guid isPermaLink="false">https://medium.com/p/19e6e81d9873</guid>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[semantic-view]]></category>
            <category><![CDATA[streamlit]]></category>
            <category><![CDATA[daanalytics]]></category>
            <category><![CDATA[data-superhero]]></category>
            <dc:creator><![CDATA[Daan Bakboord]]></dc:creator>
            <pubDate>Tue, 17 Nov 2026 01:03:18 GMT</pubDate>
            <atom:updated>2026-11-17T01:03:16.700Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_JL1qb7zCJAg_EtU.png" /></figure><p>Moving a data app should not change what its numbers mean. I moved the Apache Ossie demo from Streamlit in Snowflake to Snowflake App Runtime to try a different interface around the same metric definitions. This walkthrough covers the Next.js rewrite, deployment and the authentication error I encountered.</p><p>In the <a href="https://www.daanalytics.nl/apache-ossie-semantic-layer/">same table, different revenue</a> the demo answered one question twice: 115,550 without a semantic view, 103,200 with it. That demo ran as a Streamlit in Snowflake app. I have now moved it to Snowflake App Runtime, as a Next.js app that Snowflake builds and runs next to the data.</p><p>The numbers did not move. That was the point of the exercise. The Ossie YAML describes the intended definitions; the Snowflake semantic view implements the metrics queried by the app. Keeping those definitions outside the interface meant I could replace the interface without rewriting them.</p><p>This is a migration of a small demo, with five comparison questions. It is not a performance benchmark or a complete production migration. The code is in the <a href="https://github.com/daanalytics/apache-ossie-demo/tree/main/app-runtime?ref=daanalytics.nl">Apache Ossie demo repository</a>.</p><h3>Why try a different runtime?</h3><p>For this demo, the useful change was control over the interface: ordinary links, server-rendered pages and a custom layout. Streamlit had already made it easy to build the original data app. Moving to Next.js let me shape how readers navigate the explanation and share a particular comparison.</p><p>I would make that choice based on the application you need to build. If Python widgets already meet the requirement, keeping Streamlit avoids owning a separate web interface. Snowflake describes the distinction in its <a href="https://docs.snowflake.com/en/developer-guide/snowflake-app-runtime/about-snowflake-app-runtime?ref=daanalytics.nl">App Runtime overview</a>: Streamlit offers a Python-first approach, while App Runtime supports custom web applications.</p><p>For me, the architectural question was whether the metric definitions would survive that change intact.</p><h3>What changed</h3><p>The SQL, the data and the semantic view stayed exactly as they were. Everything between the user and the query changed: one Python app file became a small Next.js project in app-runtime/.</p><p>Every state of the app is now a URL. That sounds minor. It made the app easier to test, to link to from a blog, and to record.</p><h3>The steps</h3><p>I did the rewrite with a Cursor cloud agent, working from the existing repo. The data prerequisites stayed the same: 01_setup.sql and 02_semantic_view.sql run first. To follow the deployment steps, you also need an account configured for App Runtime, a Snowflake CLI version that supports its version 2 manifest, and a role with the required deployment privileges. Follow Snowflake&#39;s <a href="https://docs.snowflake.com/en/developer-guide/snowflake-app-runtime/account-admin-setup?ref=daanalytics.nl">account administrator setup</a> before deploying into a new environment.</p><p>Use the repository version containing the <a href="https://github.com/daanalytics/apache-ossie-demo/commit/506f7a831313ec23ebbd05805eb10e380b8243cf?ref=daanalytics.nl">OAuth authentication fix</a>.</p><ol><li><strong>Port the content.</strong> The text of the four explanatory tabs moved into lib/content.ts and components/sections.tsx. These tabs need no connection.</li><li><strong>Port the comparisons.</strong> The five questions, each with its “without” SQL, its SEMANTIC_VIEW() query and its row preview, became data in lib/scenarios.ts. lib/compare.ts runs them; lib/merge.ts lines up both result sets per dimension.</li><li><strong>Write the connection.</strong> lib/snowflake.ts reads the session token when the app runs in Snowflake, and falls back to connections.toml or environment variables when it runs on a laptop.</li><li><strong>Configure the build.</strong> next.config.ts sets output: &quot;standalone&quot; and keeps snowflake-sdk out of the bundle.</li><li><strong>Describe the app.</strong> app.yml names the service OSSIE_DEMO_SAR in OSSIE_DEMO.ECOMMERCE, uses COMPUTE_WH, and lists the install, build and run commands:</li></ol><pre>version: 2 name: OSSIE_DEMO_SAR database: OSSIE_DEMO schema: ECOMMERCE query_warehouse: COMPUTE_WH label: Apache Ossie + Snowflake description: How Apache Ossie metric context maps to a Snowflake semantic view. icon: public/icon.svg auto_suspend_secs: 900 ignore: - node_modules - .next - .env* - .git install: commands: - [npm, ci] build: commands: - [npm, run, build] - [mkdir, -p, .next/standalone/.next] - [cp, -r, .next/static, .next/standalone/.next/static] - [cp, -r, public, .next/standalone/public] run: command: [node, .next/standalone/server.js]</pre><p>The static-file copy commands matter here: they place the Next.js assets and public/ files alongside the standalone server. The <a href="https://github.com/daanalytics/apache-ossie-demo/blob/main/app-runtime/app.yml?ref=daanalytics.nl">repository manifest</a> is the complete configuration for this demo.</p><ol><li><strong>Deploy.</strong> From app-runtime/, run snow app deploy. Snowflake installs, builds and starts the app.</li><li><strong>Grant access.</strong> The repository includes 06_deploy_app_runtime.sql. For access limited to the demo role, grant service usage as below. That role also needs usage on the parent database and schema:</li></ol><pre>GRANT USAGE ON APPLICATION SERVICE OSSIE_DEMO.ECOMMERCE.OSSIE_DEMO_SAR TO ROLE OSSIE_DEMO_ROLE;</pre><p>The repository script also grants service usage to PUBLIC. Review that broader grant before running the whole script in your own account.</p><p>For a local run, use npm ci and npm run dev from app-runtime/, with credentials supported by this project&#39;s connection helper. Its local path supports username/password settings from environment variables or connections.toml; do not assume it supports every Snowflake CLI authentication method.</p><h3>The snag: one line in the connection</h3><p>The first deploy looked fine. Four tabs rendered. <strong>With vs without</strong>, the only tab that queries Snowflake, showed this for every query:</p><pre>Run 01_setup.sql and 02_semantic_view.sql first so both queries can run. Error: The user you were trying to authenticate as differs from the user tied to the access token.</pre><p>The first sentence is the app’s own fallback message, and it points the wrong way. The tables and the view were there. The second sentence is the real error.</p><p>In App Runtime the app logs in with an OAuth token that Snowflake writes to /snowflake/session/token. The token carries the service identity. The connection code also sent a username, and since none was set in the runtime, it fell back to a placeholder:</p><pre>if (token) { options.authenticator = &quot;OAUTH&quot;; options.token = token; options.username = user || &quot;snowflake-app-runtime&quot;; // the bug }</pre><p>Snowflake compared that name with the token’s user and refused. In this OAuth branch, the fix was to remove the explicit username:</p><pre>if (token) { options.authenticator = &quot;OAUTH&quot;; options.token = token; }</pre><p>Snowflake documents the session-token path and token rotation in its <a href="https://docs.snowflake.com/en/developer-guide/snowflake-app-runtime/runtime-environment?ref=daanalytics.nl">runtime environment guide</a>. The helper reads the token when it creates a connection. After a redeploy, the tab showed 115,550 and 103,200 again.</p><p>The lesson for anyone porting from Streamlit: get_active_session() hid all of this. In App Runtime the connection is your code, so check the login path before you trust the error text around it.</p><h3>Signing in is different from choosing query permissions</h3><p>Opening an authenticated app does not automatically make every SQL statement run as the viewer. In this demo, the comparison calls use the helper’s default service identity. The helper also contains a caller-token path, but the comparison calls do not enable it.</p><p>For an application with different data access per viewer, decide that execution context explicitly. Snowflake’s <a href="https://docs.snowflake.com/en/developer-guide/snowflake-app-runtime/query-snowflake?ref=daanalytics.nl">query access documentation</a> distinguishes owner’s rights from restricted caller’s rights, including the caller grants required for the latter. Keeping metric definitions stable and preserving user-specific access are separate checks in a migration.</p><h3>Same definitions, new interface</h3><p>The App Runtime version returns the same answers as the Streamlit version, question for question. That is the whole argument of the first post, tested from the other side: the definition lives with the data, so the front end is replaceable.</p><p>The second row is the join trap from the first post. The naive query joins order_items and sums orders.total_amount once for every line. The semantic view names line_revenue and sums line totals instead. That metric has no completed-order filter, whereas total_revenue does. The two rows therefore illustrate different definitions; their semantic totals should not be expected to match.</p><p>The walkthrough video covers all five tabs, every YAML section and all five questions. The URL parameters also make individual views easier to open directly.</p><p>Watch it: <a href="https://www.daanalytics.nl/content/files/library/ossie-app-runtime-demo-walkthrough.mp4?v=1793972308445935934">Apache Ossie demo on Snowflake App Runtime</a> (video, about 2 minutes).</p><h3>What I would tell you before you move</h3><ul><li><strong>Keep the contract out of the app.</strong> The app still contains comparison SQL and presentation logic. The named metric expressions queried through the semantic view stayed in Snowflake. If those expressions had lived only in Python, the rewrite would also have involved porting business definitions.</li><li><strong>Budget for the widgets.</strong> Streamlit supplies tabs, pickers, tables and charts as built-in components. In Next.js each one is yours to write; here that meant a GET form and a small SVG chart.</li><li><strong>Own the login path.</strong> Keep the OAuth connection consistent with the service token, and choose the query execution context explicitly.</li><li><strong>Make state a URL.</strong> Server-rendered pages with query parameters were simpler than client state, and they pay off in testing and recording.</li><li><strong>Keep both while you compare.</strong> The repository retains the Streamlit implementation. Compare all five questions, dimension values and filters before treating the new interface as equivalent. Additional local extensions are outside the scope of this five-tab port.</li></ul><p>The code is in the <a href="https://github.com/daanalytics/apache-ossie-demo?ref=daanalytics.nl">apache-ossie-demo repo</a>, with the App Runtime version under app-runtime/.</p><p>The warehouse still knows the amounts. The contract still decides which one is revenue. Only the window around it changed.</p><h3>Code and further reading</h3><p><strong>Transparency note:</strong> AI assisted with restructuring, editing and checking the article against repository code and Snowflake documentation.</p><p><em>Originally published at </em><a href="https://www.daanalytics.nl/from-streamlit-in-snowflake-to-snowflake-app-runtime/"><em>https://www.daanalytics.nl</em></a><em> on November 6, 2026.</em></p><p><a href="https://www.linkedin.com/redir/redirect?url=https%3A%2F%2Ffd.xuwubk.eu.org%3A443%2Fhttps%2Fcommunity.snowflake.com%2Fs%2Fdataheroes&amp;urlhash=bplT&amp;trk=article-ssr-frontend-pulse_little-text-block">Snowflake Data Superhero</a> and Chapter Lead for the <a href="https://www.linkedin.com/groups/13092879/?trk=article-ssr-frontend-pulse_little-text-block">Dutch Snowflake ❄️ User Group</a>. Consulting Partner at <a href="https://bravinci.com/">Bravinci</a>. Online also known as; <a href="https://www.linkedin.com/redir/redirect?url=https%3A%2F%2Ffd.xuwubk.eu.org%3A443%2Fhttps%2Fmedium.com%2F%40daanalytics&amp;urlhash=XbLU&amp;trk=article-ssr-frontend-pulse_little-text-block">DaAnalytics</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*H47te_BEiw8xsPQ5.jpeg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=19e6e81d9873" width="1" height="1" alt=""><hr><p><a href="https://medium.com/snowflake/from-streamlit-to-snowflake-app-runtime-a-demo-migration-19e6e81d9873">From Streamlit to Snowflake App Runtime: A Demo Migration</a> was originally published in <a href="https://medium.com/snowflake">Snowflake Builders Blog: Data Engineers, App Developers, AI, &amp; Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[I benchmarked Snowflake Postgres mirroring for two days. Here are the numbers.]]></title>
            <link>https://medium.com/snowflake/i-benchmarked-snowflake-postgres-mirroring-for-two-days-here-are-the-numbers-b50eba22b8de?source=rss----34b6daafc07---4</link>
            <guid isPermaLink="false">https://medium.com/p/b50eba22b8de</guid>
            <category><![CDATA[postgresql]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[postgresql-replication]]></category>
            <dc:creator><![CDATA[Ondřej Blaha]]></dc:creator>
            <pubDate>Tue, 17 Nov 2026 01:00:34 GMT</pubDate>
            <atom:updated>2026-11-17T01:00:32.793Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>Latency distributions, cost per refresh interval, impact on the source database, bulk loads and operations. Measured on the smallest instance that supports mirroring.</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JNkXMvJD0sYpv1WJUV8CgA.png" /></figure><p>Snowflake Postgres, managed PostgreSQL running inside a Snowflake account, has been generally available since 24 February 2026, and <strong>data mirroring</strong> into Snowflake followed on 15 October 2026. I didn’t just want to see that it works. I wanted real numbers under a simulated load, so that I can tell clients how it actually behaves and compare the cost with what we had expected. The marketing says “seconds” and “sub-minute”, and nobody had published a latency distribution, a cost per interval or the impact on the source database, so I spent two days measuring them.</p><h3>TL;DR</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dbkwS-iGpbvPLc3XAH4COw.png" /></figure><ul><li><strong>Freshness:</strong> new rows show up in TABLE$live after <strong>~26 s (median), ~37 s (p90)</strong>, no matter whether the refresh interval is 30 seconds or 10 minutes. The table itself lags behind $live by <strong>up to one refresh interval</strong>.</li><li><strong>Cost:</strong> the serverless apply task cost <strong>0.063 credits/h idle</strong> at a 30-second interval and <strong>0.25 credits/h</strong> under a steady write load. Switching to 2 minutes cut the loaded cost by <strong>73 %</strong>, to 10 minutes by <strong>88 %</strong>.</li><li><strong>Source impact:</strong> small OLTP transactions barely notice the mirror. <strong>Batch inserts into a mirrored table were 20–30 % slower</strong> than into a non-mirrored table on the same instance.</li><li><strong>Operations:</strong> SUSPEND_MIRROR took <strong>2.5–4.7 s</strong> under load, RESTART_MIRROR re-copies every table despite saying <em>&quot;soft&quot;</em>, adding an 8M-row table to a running mirror took <strong>under a minute</strong>, and the replication slot survived every suspend/resume of the instance.</li><li><strong>Gotchas:</strong> numeric without precision becomes FLOAT, json/jsonb become TEXT, 2-D arrays become NULL, PostgreSQL refuses ADD COLUMN … DEFAULT &lt;volatile&gt; on mirrored tables, any ADD, DROP or RENAME COLUMN wipes that table&#39;s $changes history, and tables without a primary key turn insert-only.</li></ul><h3>Why this matters</h3><p>Every team that runs an application on Postgres and analytics in Snowflake owns a pipeline between the two. It is usually one of these:</p><ul><li><strong>a managed ELT tool </strong>(Fivetran, Airbyte, Azure Data Factory, Matillion, etc.), priced by changed rows, data volume or run time depending on the vendor,</li><li><strong>Snowflake Openflow</strong>, a NiFi-based runtime you pay for even when nothing changes,</li><li><strong>something the team built and runs itself:</strong> open-source CDC such as Debezium (it reads the Postgres WAL and publishes every change to Kafka), Kafka Connect or Snowpipe Streaming on the Snowflake side, or a scheduled export script.</li></ul><p>All of them work, and each is one more system that someone pays for, upgrades and gets woken up by, and most of them break the first time someone adds a column.</p><p>Mirroring replaces that pipeline with one procedure call: selected tables stay continuously replicated into Snowflake, including updates, deletes and schema changes, with no external tool.</p><h3>How mirroring works under the hood</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eHy3yXBmcestX7My3S_VZw.png" /><figcaption><em>How mirroring works: capture inside Postgres, apply in Snowflake.</em></figcaption></figure><p>Mirroring is push-based, and most of the work happens inside Postgres:</p><ol><li><strong>Logical decoding.</strong> Postgres writes every change to its <strong>write-ahead log (WAL)</strong> before it touches the table files. Each mirror owns a logical replication slot, and the walsender process decodes the WAL into row-level inserts, updates and deletes.</li><li><strong>Capture.</strong> The snowflake_cdc extension runs a background worker per mirror. Roughly every <strong>15 seconds</strong> it writes the decoded batch into <strong>per-table Iceberg change logs</strong> (Parquet, via pg_lake) and records it in a <strong>meta log</strong>.</li><li><strong>Apply.</strong> On the Snowflake side, a <strong>serverless task</strong> runs every refresh_interval (30 seconds to 1 day, default 10 minutes) and applies the pending batches. Inserts are appended; updates and deletes become <strong>delete + append</strong>. Each run is one transaction aligned to the Postgres transaction boundaries, so foreign keys and joins stay consistent.</li></ol><p>Every mirrored table gives you three objects:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hY7GaboR0ESSXHgjvMqAQA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hlH16HpAwinhDkTjckSR9w.png" /><figcaption><em>One order, three views of it ~30 seconds after the commit.</em></figcaption></figure><p>The picture follows one order from the demo app. At 12:00:00 the customer changes order #500 from 2 to 3 snowboards. About 15 s later the CDC worker pushes the change to the change log, and at the measured median of ~26 s it is visible in ORDERS$live and ORDERS$changes. ORDERS keeps showing 2 until the next apply run, which with a 10-minute interval can be up to 10 minutes later.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eRPp66E7-zk0nXzI6cicTw.gif" /><figcaption><em>What the animation shows: a small Streamlit in Snowflake app with its own mirror on the benchmark instance (</em><em>refresh_interval 30 s). One click inserts 1,000 orders into Postgres in a single transaction, and the app then polls Snowflake once a second until all 1,000 rows are in </em><em>ORDERS$live (23.8 s) and in the </em><em>ORDERS table (25.1 s). The counters below then show the same row count on both sides. The waiting is sped up 4× in the animation; the stopwatch shows real seconds.</em></figcaption></figure><p>Two details are worth knowing before the results. The UPDATE shows up in $changes as a <strong>D + I pair</strong> (a delete of the old row and an insert of the new one) with _IS_UPDATE = true, and $live is <strong>not transactional</strong>, so for a moment it can show half of a large transaction.</p><h3>Setup in six steps</h3><p>You need a Snowflake Postgres instance of the STANDARD or HIGHMEM family on AWS or Azure. The instance accepts no connections until it has a network policy, and the policy is attached when the instance is created, so it comes first.</p><p><strong>Before you start</strong>, ACCOUNTADMIN grants three privileges, once, to the role that will own the instance and manage the mirror:</p><pre>GRANT CREATE POSTGRES INSTANCE ON ACCOUNT TO ROLE PG_ADMIN;<br>GRANT CREATE NETWORK POLICY ON ACCOUNT TO ROLE PG_ADMIN;<br>GRANT APPLICATION ROLE SNOWFLAKE.POSTGRES_MIRROR_ADMIN TO ROLE PG_ADMIN;</pre><p><strong>The application role isn’t granted to anyone by default, not even ACCOUNTADMIN.</strong> Run all six steps as that role: every mirror procedure checks that the caller owns the source instance, so the role that creates the instance in step 2 is the one that manages the mirror. (If you create the mirror in Snowsight, under Postgres → Manage → Mirroring, the grants are set up for you.)</p><p><strong>1. Create the network policy.</strong> Allow Snowflake’s own egress ranges (mirroring and Snowflake apps connect from there, SELECT SYSTEM$GET_SNOWFLAKE_EGRESS_IP_RANGES() lists them) plus the IP of your client. Avoid 0.0.0.0/0.</p><pre>CREATE NETWORK RULE MY_DB.MY_SCHEMA.PG_INGRESS   -- network rules live in a schema<br>  TYPE = IPV4 MODE = POSTGRES_INGRESS<br>  VALUE_LIST = (&#39;192.0.2.0/24&#39;, &#39;198.51.100.0/24&#39;,   -- sample values: Snowflake egress ranges for your region<br>                &#39;203.0.113.10/32&#39;);                 -- sample value: your client IP<br><br>CREATE NETWORK POLICY MY_PG_POLICY<br>  ALLOWED_NETWORK_RULE_LIST = (&#39;MY_DB.MY_SCHEMA.PG_INGRESS&#39;);</pre><p>The addresses above are documentation placeholders. Take the real ranges for your region from the function, and check them again from time to time, because each one comes with an expiry date.</p><p><strong>2. Create the instance</strong> (it returns the passwords exactly once, so store them):</p><pre>CREATE POSTGRES INSTANCE KS_PG_BENCH<br>  COMPUTE_FAMILY = &#39;STANDARD_M&#39;<br>  STORAGE_SIZE_GB = 50<br>  AUTHENTICATION_AUTHORITY = POSTGRES<br>  POSTGRES_VERSION = 18<br>  NETWORK_POLICY = &#39;MY_PG_POLICY&#39;;</pre><p><strong>3. Let mirroring use the instance</strong>, once it is READY:</p><pre>GRANT USAGE ON POSTGRES INSTANCE KS_PG_BENCH TO APPLICATION SNOWFLAKE;</pre><p><strong>4. Install the extension in Postgres</strong> (it is not there by default):</p><pre>CREATE EXTENSION IF NOT EXISTS snowflake_cdc CASCADE;</pre><p><strong>5. Create the mirror:</strong></p><pre>CALL SNOWFLAKE.POSTGRES.CREATE_MIRROR(<br>  mirror_name       =&gt; &#39;ks_bench_mirror&#39;,<br>  postgres_instance =&gt; &#39;KS_PG_BENCH&#39;,<br>  postgres_database =&gt; &#39;postgres&#39;,<br>  target_database   =&gt; &#39;KS_BENCH_MIRROR&#39;,<br>  postgres_schemas  =&gt; [&#39;bench&#39;],<br>  refresh_interval  =&gt; &#39;30 seconds&#39;);</pre><p><strong>6. Watch the snapshot finish:</strong></p><pre>CALL SNOWFLAKE.POSTGRES.LIST_MIRRORED_TABLES(&#39;ks_bench_mirror&#39;);<br>-- SNAPSHOTTING -&gt; REPLICATING</pre><p>Most of the gotchas I met while preparing the demo for an internal talk:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lh58VwWpIUQyu7oTnwf_ZA.png" /></figure><h3>Benchmark methodology</h3><p>The environment was AWS eu-central-1, Enterprise edition, Postgres 18 on <strong>STANDARD_M (1 vCPU, 4 GB RAM, 50 GB disk)</strong>. It is the smallest instance that supports mirroring, so for source impact it is the worst case. One mirror covered schema bench (4 tables at first, 19 by the end, after the seed tables and the type and DDL tests were added), with logical_decoding_work_mem = 64MB and max_slot_wal_keep_size = 5GB.</p><p>The test runner is a Python script on my laptop: psycopg 3 for Postgres, the Snowflake connector with key-pair auth, an XS warehouse for polling, results as JSON lines. Write load runs <strong>server-side</strong> (PL/pgSQL procedures, pg_cron), so network jitter stays out of the TPS numbers. After every test a verification step compares row counts, sums and a content hash between Postgres, the table and $live. <strong>All 7 load tables matched after every test</strong>, including the runs that crashed halfway (more on those in the operational notes).</p><p>For latency, the script commits a batch and then polls $live, the table and $changes every second with separate queries until the batch is complete. Commits come at random gaps of 15–60 s, so they land at random phases of the push and apply cycles. Resolution is about ±1.3 s (query time + poll). I also checked the laptop clock against _COMMIT_TIME; the offset was under 0.1 s.</p><p>I report latency as percentiles, because the average hides the slow tail. <strong>p50</strong> is the median: half of the commits arrived faster than this, half slower. <strong>p90</strong> means 9 out of 10 commits arrived within that time, and <strong>max</strong> is the single slowest one.</p><p>The limitations are real: one region, one instance size and one run per configuration (N = 40–60 samples for latency, 4 repetitions per profile for source impact, up to three runs per operation). Costs come from ACCOUNT_USAGE (up to 3 h delay) in 2-hour windows. The 10-minute latency run also ended early. Polling once a second keeps an XS warehouse busy all day, and near the end of that run the 8-credit resource monitor on it ran out and stopped everything, so that run has 49 samples instead of 60. Use the numbers as an order of magnitude for your own planning; they are not an SLA. The whole benchmark cost about <strong>13 credits (~51 USD)</strong>, and two thirds of that was my own once-per-second polling warehouse, not the mirroring.</p><h3>Result 1: latency, and what refresh_interval really changes</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d3mAjkPP9Dy2TPr7BlEQQQ.png" /><figcaption><em>Latency from commit, every dot is one 100-row commit.</em></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cv9FQJxEbPwpjHBsv6NK1A.png" /><figcaption><em>Batches of 100 rows. N = number of measured commits.</em></figcaption></figure><p>The $live median stays in the 21–26 s band at all three settings, which fits the ~15 s push cycle plus the ~30 s $live refresh from the docs. The table is a different story: it lags $live by up to one interval and by half of it on average, so at 10 minutes half the samples needed more than 5.5 minutes to reach the table.</p><p>Batch size barely matters. At 30 seconds, a single-row commit reached $live in 30 s (p50), a 100-row batch in 21 s and a 100 000-row batch in 18 s, and $changes tracked $live within a second in every run.</p><p>The first single-row run looked broken: the table median came out at 280 s. One apply execution had taken <strong>415 s</strong> instead of the usual 3–15 s, and for those 7 minutes the table stood still while $live and $changes stayed fresh. A rerun gave normal numbers, and I have no explanation for the slow run. I kept it in the results, because it is the best argument for pointing any dashboard that has to be fresh at $live.</p><blockquote><em>In practice I decide per consumer which object it reads, and only then pick the interval. Reports and dbt models can read the table at 2–10 minutes. Operational dashboards, Streamlit apps, alerts and AI agents should read </em><em>$live, which stays at ~30 s whatever the interval. To business users I promise &quot;about half a minute&quot; for </em><em>$live. If something needs guaranteed seconds, it needs a different tool, and the table can stall for minutes now and then (the 415 s run).</em></blockquote><h3>Result 2: cost per refresh interval</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Gmyy4C607vXh81vbSlYOPQ.png" /><figcaption><em>Apply cost per month by refresh interval, idle vs. steady load.</em></figcaption></figure><p>I ran six 2-hour windows overnight. “Load” means pg_cron every 5 s inserting 100 rows and updating 10 random rows: about 72 000 inserts and 7 000 updates per hour, spread over 4 tables.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/950/1*67qlE8n-LZcGwI36KzsWBA.png" /><figcaption><em>*730 h × 3.90 USD/credit (Enterprise on-demand). Apply task compute only, instance not included.</em></figcaption></figure><p>Under load the interval is the main cost lever: going from 30 s to 2 min saved 73 %, going to 10 min saved 88 %.</p><p>Idle surprised me. I had expected 10 minutes to be at least 5× cheaper than 30 s when nothing changes, and it came out 3.4×. Two minutes and ten minutes cost the same, about 0.02 credits/h. The docs say the apply task checks for work at least every 2 minutes (that is also how DDL gets through quickly), which would explain the floor. At the other end, 0.063 credits/h at 30 seconds is about <strong>45 credits a month</strong> for a mirror that is just waiting. That is the number I’d show a client before they pick 30 seconds just because it is the fastest option.</p><p>The load itself was a worst case on purpose. Updating random rows makes delete + append rewrite micro-partitions all over the table, so insert-only workloads should come out cheaper. I haven’t measured that.</p><p>You can check your own numbers with a single query:</p><pre>SELECT DATE_TRUNC(&#39;hour&#39;, start_time) AS hour,<br>       ROUND(SUM(credits_used), 4)    AS credits<br>FROM SNOWFLAKE.ACCOUNT_USAGE.SERVERLESS_TASK_HISTORY<br>WHERE task_name ILIKE &#39;APPLY_MIRROR%&#39;<br>GROUP BY 1 ORDER BY 1 DESC;</pre><p>Storage is the part that doesn’t show up in that query. Delete + append rewrites micro-partitions, so frequently updated tables accumulate <strong>Time Travel of 2–2.5× their active size</strong>, and $changes keeps 7 days of history on top. Our most-changed table had 1.6 GB of active data, 4.0 GB in Time Travel and 3.5 GB in $changes. At ~23 USD/TB/month the money is small, but plan for 4–5× the raw table size.</p><p>I wanted to shorten Time Travel on the mirror, and you can’t. The target database and all its tables are owned by the SNOWFLAKE application, retention is set to 1 day on the tables and 5 days on the internal change log tables, and both ALTER TABLE … SET DATA_RETENTION_TIME_IN_DAYS and ALTER DATABASE … failed with <em>&quot;Insufficient privileges&quot;</em> for the role that created the mirror. (I did not try it as ACCOUNTADMIN.) So the 4.0 GB above is what a single day of Time Travel looks like on a table that gets updated all the time.</p><p>For the instance itself, STANDARD_M runs at about 111 USD/month on Enterprise and 76 USD on Standard edition. An Azure Flexible Server B2s is around 62 USD, but then you still pay for the pipeline.</p><blockquote><em>For a new mirror I’d start at 2 minutes. Under my test load that was ~192 USD/month against ~712 USD at 30 seconds (idle 55 against 180), with </em><em>$live just as fresh. Thirty seconds pays off only when a consumer has to read the table itself every minute and can&#39;t switch to </em><em>$live. The cost query above belongs in a monthly check, and tables that are updated often need 4–5× their source size in storage.</em></blockquote><h3>Result 3: impact on the source database</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vrreWPQBWPuF4cklikRs-Q.png" /><figcaption><em>Throughput of the source database relative to a run without a mirror.</em></figcaption></figure><p>The docs warn that CDC shares CPU with your application, but give no numbers. So I wrote the same load into Postgres in six <strong>write profiles</strong>, each one a different kind of work an application does:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sCT3BkGw4RPSjtHnuy3RYw.png" /></figure><p>Each profile ran for 120 s as fast as it could, and I counted committed <strong>transactions per second (TPS)</strong>. Every profile ran in three situations:</p><ul><li><strong>no mirror:</strong> the day before, on the same instance, before the slot existed;</li><li><strong>other table:</strong> mirror exists, writes go to a non-mirrored table (the walsender still decodes all WAL);</li><li><strong>mirrored table:</strong> writes go to a table in the mirror.</li></ul><p>The last two ran 4 times each, alternating in the order other, mirrored, mirrored, other (and again), so that anything drifting during the evening, like a noisy neighbour on the host, hits both sides equally.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MLapep4iRIgo48Zq7J3Vow.png" /></figure><p>The clean comparison is between the “other table” and “mirrored table” columns, which ran the same evening, interleaved. Small transactions show no meaningful difference there. <strong>Batch inserts lose 20–30 %</strong> when they hit a mirrored table, because decoding and pushing those rows competes with the application for the single vCPU.</p><p>I would be careful with the “no mirror” column. It comes from the previous day on a fresh instance, and part of the −7 to −17 % is plain day-to-day variance, so please don’t quote it as “mirroring costs 17 %”.</p><p>The CDC worker caught up quickly: the slot drained to under 1 MB within <strong>6–13 s</strong> after small-transaction runs and <strong>8–138 s</strong> after P4, and Snowflake matched Postgres <strong>10–85 s</strong> after the load stopped.</p><p>The 138 s came from the run that worried me most. While I was watching the second P4 repetition, the mirror held <strong>4.95 GB of WAL after 120 s</strong> of writes, against a max_slot_wal_keep_size of 5 GB. Had the worker fallen a little further behind, Postgres would have invalidated the slot and the mirror would have needed a full re-snapshot. It didn&#39;t happen, but on a 1-vCPU instance I&#39;d size this setting with headroom for the largest burst, or split the load.</p><blockquote><em>If your application writes small transactions, like a new order or a status change, you can size the instance as if the mirror weren’t there. Imports and batch jobs into mirrored tables are what to plan for: expect 20–30 % less throughput on them, give </em><em>max_slot_wal_keep_size headroom above the biggest burst, and move off the smallest instance before a heavy import lands on it.</em></blockquote><h3>Result 4: bulk loads, one big transaction vs batches</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ME2rErzcn1c2Y0bkD_Kynw.png" /><figcaption><em>Write = how long the statement ran in Postgres. </em><em>$live and Table = latency measured from the commit (for the 50 batches, from the last commit). Spill = how much the decoder had written to disk so far, cumulative over the steps.</em></figcaption></figure><p>Everything went through: no RESTART_REQUIRED, wal_status always reserved, verify green after every step.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rkxxqyZtIPVPpX06fdw3FA.png" /><figcaption><em>5 million rows: one transaction vs. 50 batches, on a timeline from the start of the load.</em></figcaption></figure><p>The split load is the row I looked at twice. <strong>Fifty transactions of 100k took 40 % longer to write, but after the last commit the data was in Snowflake twice as fast (28 vs. 54 s)</strong> and caused <strong>no spill to disk</strong>. Measured from the start of the write, both variants landed at about the same time (~105 s). Batching therefore doesn’t speed up the load end to end; what changes is that memory and disk stay flat. A single 5M-row transaction exceeds logical_decoding_work_mem (64 MB), so the decoder spills it to disk and can only push it after the commit, which matches the docs&#39; advice to split bulk loads.</p><p>If the import already lands in a staging table, a procedure that commits after every chunk is enough (the benchmark did the same with COMMIT after each 100k-row batch):</p><pre>CREATE OR REPLACE PROCEDURE load_in_chunks(chunk int DEFAULT 100000)<br>LANGUAGE plpgsql AS $$<br>DECLARE lo bigint; hi bigint;<br>BEGIN<br>  SELECT min(id), max(id) INTO lo, hi FROM staging.orders_import;<br>  WHILE lo &lt;= hi LOOP<br>    INSERT INTO app.orders<br>    SELECT * FROM staging.orders_import WHERE id &gt;= lo AND id &lt; lo + chunk;<br>    COMMIT;                      -- each chunk is its own transaction<br>    lo := lo + chunk;<br>  END LOOP;<br>END $$;<br><br>CALL load_in_chunks();           -- outside an explicit transaction, or COMMIT failsThe cheapest fix in the whole benchmark is a conversation with whoever owns the imports: commit in chunks of up to 100k rows. Total load time stays the same, the data reaches Snowflake just as soon, and the 1.2 GB spill goes away, which was the closest the 1-vCPU instance came to trouble here.</pre><h3>Result 5: adding a table to a running mirror</h3><pre>CALL SNOWFLAKE.POSTGRES.ALTER_MIRROR(<br>  mirror_name =&gt; &#39;ks_bench_mirror&#39;,<br>  add_tables  =&gt; [&#39;bench_seed.big_l&#39;]);</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Rkjtz-KdbKOxRYdQnh7_lA.png" /></figure><p>The mirror kept running while the tables were added. A regression over the three points gives about <strong>36 s of fixed overhead plus ~50 MB/s</strong>, close to <a href="https://floredata.com/blog/snowflake-postgres-mirroring/">Floredata’s</a> ~40 MB/s on a 70 GB production snapshot. Snowflake stored the data about 4× smaller than Postgres.</p><blockquote><em>Because a new table takes about a minute per gigabyte and needs no downtime, I’d start with the few tables one concrete use case needs and add more with the next one. Every mirrored table adds apply work, Time Travel and </em><em>$changes storage, and mirroring the whole database &quot;just in case&quot; pays for all of that from day one.</em></blockquote><h3>Five things to check before you mirror a real schema</h3><p>I also pushed every Postgres type group and 14 kinds of schema change through the mirror. Most of it just works (including ALTER COLUMN TYPE and large TOAST values). These are the five findings that change what you do before you start:</p><ol><li><strong>Primary keys on every table.</strong> Postgres rejects UPDATE and DELETE on a published table without a replica identity, so a table without a PK becomes insert-only. Add the PK <em>before</em> you mirror (<a href="https://floredata.com/blog/snowflake-postgres-mirroring/">Floredata</a> had to delete 7 million duplicates to get there).</li><li><strong>No volatile defaults in migrations.</strong> ALTER TABLE … ADD COLUMN created_at timestamptz DEFAULT clock_timestamp() fails with <em>&quot;cannot add column … with volatile DEFAULT&quot;</em>. Add the column without a default, backfill in batches and set the default afterwards. Constant defaults are fine.</li><li><strong>Types that change on the way.</strong> numeric without precision (or above 38 digits) arrives as FLOAT and loses precision, json/jsonb arrive as TEXT and need PARSE_JSON downstream, and timetz is normalised to UTC with the offset dropped.</li><li><strong>2-D arrays arrive as NULL.</strong> The column exists in Snowflake, the values don’t. This is documented, but easy to miss.</li><li><strong>Sentinel dates.</strong> infinity and -infinity become 9999-12-31 and 0001-01-01 (−4712-01-01 for date). Plain SQL reads them, but the Snowflake Python connector failed to fetch that table (<em>&quot;ordinal must be &gt;= 1&quot;</em>), and the same values had already crashed my own psycopg script on the Postgres side. If you use them, test your downstream tools.</li></ol><p>One more thing turned up only after the benchmark, while I was recording the demo animation. I updated, cancelled and deleted an order and then added a column to the same table. When I went back to $changes for that order&#39;s history, it was gone: the change log held only the commit that came after the ADD COLUMN.</p><p>So I tested it properly on four fresh tables with the same history (10 inserts, 3 updates, 1 delete, 17 rows in $changes). On three of them I ran one schema change each, ADD COLUMN, DROP COLUMN and RENAME COLUMN, followed by one more update; the fourth got only the update. Within about 30 seconds the three changed tables had lost their whole history, and $changes held just the D + I pair of the update that came after the DDL. The control table kept all 19 rows. The data itself was fine: the table and $live had all 9 rows in every case, and the new, dropped or renamed column showed up correctly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bTFJUdCXy4wPlFdcIJlb-A.png" /></figure><p>The docs say that schema changes on mirrored tables rebuild the feed, and this is what that means in practice: $changes keeps 7 days of history <em>or</em> everything since the last schema change on that table, whichever is shorter. Application teams ship migrations all the time, so if $changes is meant to be an audit trail, copy it into a table of your own on a schedule (a task with INSERT … SELECT from $changes since the last copied _COMMIT_LSN is enough).</p><h3>Operational notes</h3><p>At first I ran each operation once. Some of the conclusions I drew from that turned out to be too strong, so I repeated the ones that matter for a runbook. The table includes all runs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VJKLKmRBdmBEC7lFmX_pyg.png" /></figure><p>Pausing is cheap. The CDC worker keeps decoding WAL and pushing change logs while the mirror is suspended, and only the apply step stops, so the slot held under half a megabyte during 5- and 10-minute pauses under load. On resume, Snowflake applies the queued batches within a minute. I haven’t tested pauses longer than 10 minutes.</p><p>The restart is where my first conclusion was wrong. The first time, I checked $changes a few seconds after the tables showed REPLICATING again, found no snapshot rows and wrote down &quot;soft restart, no re-snapshot&quot;. When I repeated it and kept watching, the tables went back to SNAPSHOTTING a little later, and every restart (two under load, one idle) ended with a full copy: 505 700 S rows in $changes for events_load alone, and about 9 minutes until all 19 tables were replicating. The docs describe RESTART_MIRROR as a new snapshot, and that is what it does, whatever the return message says.</p><p>Restarts also break queries for a while. Snowflake recreates the $live and $changes views, and until it does, every query fails with <em>&quot;does not exist&quot;</em>. My verification script crashed on that twice: 35–40 s after the first restart, and around two minutes after one of the three repeated ones. Both times it did what it was built to do and suspended the mirror and the instance, the second time in the middle of the re-snapshot. Resuming was enough to let the snapshot finish.</p><p>The error message after an instance resume turned out to be rare. Once, the mirror came back ACTIVE while error_message said <em>&quot;Apply suspended: the source postgres instance is suspended … call RESTART_MIRROR&quot;</em>, and the text disappeared after ~5 minutes. I restarted the mirror that time. In the other resumes I logged (at least four more, one of them with 25 pending operations) there was no message, and the mirror caught up on its own within ~3 minutes. Knowing now that a restart means a full re-snapshot, I&#39;d give apply_lag_operations a few minutes before following that advice. This is the one thing I&#39;d like Snowflake to change. A message that recommends RESTART_MIRROR to a mirror that is catching up on its own sends you to the most expensive fix there is, a full re-snapshot that also wipes $changes. And the <em>&quot;soft&quot;</em> in the return value made me believe, the first time, that nothing would be copied again.</p><p>The status alone tells you little, because the mirror was ACTIVE during all of this. An alert should look at the lag. DESCRIBE_MIRROR is a procedure, and an alert condition can only hold one statement, so I wrapped the check in a procedure that returns rows only when something is wrong:</p><pre>CREATE OR REPLACE PROCEDURE MIRROR_LAGGING(MIRROR STRING, MAX_LAG_S NUMBER)<br>RETURNS TABLE (MIRROR_NAME STRING, STATUS STRING, APPLY_LAG_SECONDS NUMBER,<br>               APPLY_LAG_OPERATIONS NUMBER, SLOT_LAG_BYTES NUMBER)<br>LANGUAGE SQL AS<br>DECLARE res RESULTSET;<br>BEGIN<br>  CALL SNOWFLAKE.POSTGRES.DESCRIBE_MIRROR(:MIRROR);<br>  res := (SELECT MIRROR_NAME, STATUS, APPLY_LAG_SECONDS, APPLY_LAG_OPERATIONS,<br>                 POSTGRES_STATUS:slot_lag_bytes::NUMBER<br>          FROM TABLE(RESULT_SCAN(LAST_QUERY_ID()))<br>          WHERE STATUS &lt;&gt; &#39;SUSPENDED&#39; AND APPLY_LAG_SECONDS &gt; :MAX_LAG_S);<br>  RETURN TABLE(res);<br>END;</pre><pre>CREATE ALERT MIRROR_LAG_ALERT<br>  WAREHOUSE = MY_WH  SCHEDULE = &#39;5 MINUTE&#39;<br>  IF (EXISTS (CALL MIRROR_LAGGING(&#39;ks_bench_mirror&#39;, 90)))  -- 3 × refresh_interval<br>  THEN CALL SYSTEM$SEND_EMAIL(&#39;my_email_int&#39;, &#39;oncall@example.com&#39;,<br>                              &#39;Mirror lag&#39;, &#39;ks_bench_mirror is falling behind&#39;);<br><br>ALTER ALERT MIRROR_LAG_ALERT RESUME;   -- a new alert starts suspended; needs EXECUTE ALERTThe procedure returned the right rows on my mirror and Snowflake accepted the alert with a CALL in the condition. To run or resume an alert, the owning role also needs the account-level EXECUTE ALERT privilege, which SYSADMIN doesn&#39;t have by default. If you also want to catch a slot that is filling up, add a condition on SLOT_LAG_BYTES against your max_slot_wal_keep_size.</pre><p>The procedure returned the right rows on my mirror and Snowflake accepted the alert with a CALL in the condition. Create the procedure with the role that manages the mirror, because DESCRIBE_MIRROR inside it runs with the owner&#39;s privileges. To run or resume an alert, the owning role also needs the account-level EXECUTE ALERT privilege, which SYSADMIN doesn&#39;t have by default. If you also want to catch a slot that is filling up, add a condition on SLOT_LAG_BYTES against your max_slot_wal_keep_size.</p><blockquote><em>Three things from this go straight into a runbook. Pausing is cheap enough to be part of a maintenance plan: suspend the mirror before a big migration and resume it afterwards (I measured up to 10 minutes under load). </em><em>RESTART_MIRROR is not: it copies every table again, so plan it like the initial load and keep queries on </em><em>$live and </em><em>$changes away from it for a few minutes. And whoever is on call should alert on apply lag and slot lag, and give </em><em>apply_lag_operations a few minutes after an instance resume before restarting anything.</em></blockquote><h3>How it compares</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hcGckxfLtoNEOyNZGnZLZQ.png" /></figure><p>Airbyte OSS itself is free. It runs on 2 vCPU / 8 GB in low-resource mode and 4 vCPU is recommended (t3.large–t3.xlarge in eu-central-1 ≈ 70–140 USD/month on demand, + disk), which makes it the cheapest path on paper as long as you don’t count the people who upgrade and watch it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r_UuBWgqi96uG5koI4zTIw.png" /><figcaption><em>What the whole path from Postgres to Snowflake costs per month (STANDARD_M vs. the closest Azure instance, B2s).</em></figcaption></figure><h3>When I would (and would not) use it</h3><p><strong>Yes, if:</strong></p><ul><li>the application can run on Snowflake Postgres (AWS/Azure, same account),</li><li>you want operational dashboards, apps or AI agents on data that is seconds-to-minutes old,</li><li>you want to retire a pipeline and a vendor,</li><li>you need a 7-day change history without building CDC yourself (and copy it away if it has to survive schema changes).</li></ul><p><strong>Not yet, if:</strong></p><ul><li>the source must stay on RDS, Azure Flexible Server or on-prem,</li><li>you have tables without primary keys you cannot fix,</li><li>you rely on 2-D arrays, unconstrained numeric or timetz offsets being exact,</li><li>a 1-vCPU instance has to absorb heavy bulk loads (size up; the docs recommend STANDARD_XL as a starting point).</li></ul><p><strong>Settings I’d start with:</strong> refresh_interval of 2–10 minutes for the table, $live for anything operational, PKs on everything, bulk loads in ≤ 100k-row transactions, and max_slot_wal_keep_size with headroom above your biggest burst.</p><p><strong>What has your experience been?</strong> If you already run Postgres in Snowflake, with or without mirroring, I’d like to hear how it has gone so far: what worked, what surprised you, and whether your numbers look like mine. Leave a comment, especially if you’ve seen different latencies or costs, or a restart or schema change that behaved differently than described here.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b50eba22b8de" width="1" height="1" alt=""><hr><p><a href="https://medium.com/snowflake/i-benchmarked-snowflake-postgres-mirroring-for-two-days-here-are-the-numbers-b50eba22b8de">I benchmarked Snowflake Postgres mirroring for two days. Here are the numbers.</a> was originally published in <a href="https://medium.com/snowflake">Snowflake Builders Blog: Data Engineers, App Developers, AI, &amp; Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Treat LLM Output as Evidence, Not Truth: A Production Pattern for Sentiment and NER]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/snowflake/treat-llm-output-as-evidence-not-truth-a-production-pattern-for-sentiment-and-ner-43bdbd4bed6e?source=rss----34b6daafc07---4"><img src="https://cdn-images-1.medium.com/max/2600/1*ubwnuDJtIfFO-4ujMf7W6Q.jpeg" width="2752"></a></p><p class="medium-feed-snippet">How to turn customer comments into validated, graph-ready data with Snowflake Cortex, and what the demos leave out</p><p class="medium-feed-link"><a href="https://medium.com/snowflake/treat-llm-output-as-evidence-not-truth-a-production-pattern-for-sentiment-and-ner-43bdbd4bed6e?source=rss----34b6daafc07---4">Continue reading on Snowflake Builders Blog: Data Engineers, App Developers, AI, &amp; Data Science »</a></p></div>]]></description>
            <link>https://medium.com/snowflake/treat-llm-output-as-evidence-not-truth-a-production-pattern-for-sentiment-and-ner-43bdbd4bed6e?source=rss----34b6daafc07---4</link>
            <guid isPermaLink="false">https://medium.com/p/43bdbd4bed6e</guid>
            <category><![CDATA[snowflake-cortex]]></category>
            <category><![CDATA[sentiment-analysis]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[snowflake]]></category>
            <dc:creator><![CDATA[Akshay Sharma]]></dc:creator>
            <pubDate>Tue, 17 Nov 2026 00:58:11 GMT</pubDate>
            <atom:updated>2026-11-17T00:58:09.766Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Operational Ontology on Snowflake: From Business Meaning to Governed AI Action]]></title>
            <link>https://medium.com/snowflake/operational-ontology-on-snowflake-from-business-meaning-to-governed-ai-action-6f3ef270d3f2?source=rss----34b6daafc07---4</link>
            <guid isPermaLink="false">https://medium.com/p/6f3ef270d3f2</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[ontology]]></category>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Tianxia Jia]]></dc:creator>
            <pubDate>Thu, 12 Nov 2026 21:36:38 GMT</pubDate>
            <atom:updated>2026-11-12T21:36:37.601Z</atom:updated>
            <content:encoded><![CDATA[<p><em>A reference architecture for AI agents to reason over operational state, simulate decisions, execute governed actions, and verify real-world outcomes.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6MPy796wB_A_qCE4vHE46A.png" /></figure><h3>From Enterprise Knowledge to Operational Decisions</h3><p>At 1:30 PM, packing station PB-04 goes offline in a distribution center. Orders begin accumulating, and a carrier scheduled to depart at 5:00 PM may leave before its shipments are ready.</p><p>An AI agent connected to warehouse data can identify the failed station, trace affected orders, and explain the shipping risk. But the warehouse manager needs to decide what to do.</p><p>Can qualified workers move to another station? Can orders be rerouted? Which recovery plan has the best chance of meeting the deadline without creating another bottleneck?</p><p>Answering these questions requires current operational state, resource capabilities, business constraints, and a way to evaluate proposed changes. Acting on a decision also requires authorization and confirmation that the intended changes occurred.</p><p>In the earlier <a href="https://medium.com/snowflake/ontology-on-snowflake-part-1-overview-and-data-model-9e8eeaac7363">Ontology on Snowflake series</a>, I explored how ontology definitions, enterprise data, semantic models, and AI reasoning establish a shared understanding of business information. Snowflake now has a product of <a href="https://www.snowflake.com/en/blog/business-ontology-context/">Business Ontology</a> for governed business definiton (Public Preview soon).</p><p>This article builds on that foundation by introducing a <strong>Snowflake Operational Ontology Reference Architecture</strong>. It connects governed business meaning to operational state, constraints, simulation, AI reasoning, and controlled actions.</p><p>The goal is to establish a reusable pattern. A warehouse, supply chain network, and data center require different operational models, but they can share architectural contracts and a common decision lifecycle.</p><h3>1. What Makes an Ontology Operational?</h3><p>An ontology defines the concepts, relationships, and semantics of a domain. In our warehouse example, the ontology defines Packing Station, Worker, Order, and Carrier Commitment, along with relationships such as workers operating stations and stations processing orders.</p><p>That model explains why PB-04’s failure matters. It does not, by itself, tell us whether moving a worker to another station will improve the shipping outcome.</p><p>Operational ontology extends the domain model with the semantics needed to evaluate and govern changes. Here, <em>operational semantics</em> means the business rules that govern how real resources can change: what they can do, which limits apply, and which interventions are allowed. (This is not the programming-language term of the same name.)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I0Ywpxd3YDELwS7zqYakQA.png" /></figure><p>These elements connect business meaning to operational behavior.</p><p>The ontology defines what an action means and which rules apply. Decision services evaluate its consequences. Execution services carry out authorized changes.</p><p><strong>The key architectural principle is to separate meaning from behavior while connecting them through explicit contracts.</strong></p><h3>2. The Operational Ontology Reference Architecture</h3><p>The proposed framework has six capabilities.</p><ol><li><strong>Governed meaning</strong> provides consistent definitions of business entities, metrics, and relationships.</li><li><strong>Operational state</strong> captures current and historical conditions, including timestamps, versions, and provenance.</li><li><strong>Operational semantics</strong> defines capabilities, constraints, policies, and action types.</li><li><strong>Decision services</strong> validate proposed changes and evaluate alternatives through simulation, forecasting, or optimization.</li><li><strong>AI reasoning</strong> uses Cortex Agents to investigate problems, invoke tools, and explain recommendations.</li><li><strong>Governed execution and verification</strong> handle approvals, action dispatch, observations, and outcome evaluation.</li></ol><p>Together, these capabilities support a common decision lifecycle:</p><p><strong>Observe → Understand → Evaluate → Decide → Execute → Verify</strong></p><p>The architecture remains consistent across applications. Each domain supplies its own rules, behavioral models, and execution adapters.</p><p>Figure 1 shows how the six capabilities fit together on Snowflake. Meaning, state, and semantics feed decision services and Cortex Agents. Only governed procedures dispatch actions to external systems, and observed outcomes flow back into operational state.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7-rp9nj1fRsogWmT3ZjwuQ.png" /><figcaption><strong>Figure 1. Snowflake Operational Ontology Reference Architecture.</strong> The six numbered capabilities inside the Snowflake platform boundary, the four shared contracts, the operations manager in the loop, and external systems of execution. The dashed path returns observed outcomes to operational state, and the ribbon shows the six-step decision lifecycle.</figcaption></figure><h3>3. Implementing the Framework on Snowflake</h3><p>Snowflake provides the data, metadata, computation, AI, and governance capabilities needed to implement this architecture.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JkSa90_5DM8foyfvdgAzGg.png" /></figure><p>Streams and tasks provide asynchronous processing across these capabilities on supported sources.</p><p>The subsections below focus on the design choices specific to each capability.</p><h4>Governed meaning: Business Ontology</h4><p><em>(*Business Ontology Public Preview Soon.)</em></p><p>Snowflake’s new <strong>Business Ontology</strong>, part of Horizon Catalog, provides a steward-governed layer for business terms, metrics, entities, and relationships.</p><p>Concepts can be bound to the Snowflake tables, views, columns, and Semantic Views that represent them.</p><p>For example, <em>Packing Station</em>, <em>Worker</em>, <em>Order</em>, and <em>Carrier Commitment</em> can have consistent definitions across warehouse management, labor, and shipping systems.</p><p>Business Ontology complements Semantic Views, which define executable analytical metrics, and Cortex Sense (in preview), which helps derive context for data that has not been fully modeled.</p><p>Business Ontology governs meaning. Operational semantics and services provide the rules and behavior needed for decisions and actions.</p><p>It models concepts rather than instances, and it does not execute rules or actions. That suggests a clear division of labor:</p><ul><li><strong>In Business Ontology:</strong> object types such as <em>Worker</em> and <em>Station</em> (Entity nodes), the typed relationships between them, and named constraints and action types (Terms linked through relationships such as applies to).</li><li><strong>In Snowflake tables:</strong> the objects themselves, their current links, constraint parameters, and action-type specifications.</li></ul><p>Representations bind each concept to the table or view that implements it, so an agent can move from the concept <em>Reassign worker</em> to the record that defines its parameters and approval policy.</p><h4>Operational state: Standard and hybrid tables</h4><p>Operational applications need historical context and a reliable view of current conditions.</p><p>For PB-04, that means station status, worker assignments, order queues, and carrier commitments.</p><p>Standard tables store events, transactions, and history and are a sound default. Hybrid tables can support selected transactional state workloads that benefit from indexed point lookups and enforced keys; <em>(note that they are available only in AWS and Azure commercial regions)</em> and carry per-database storage and throughput quotas.</p><p>One application structure might be:</p><pre>ONT   Operational Ontology  Constraint parameters and action-type specifications,<br>                            keyed to Business Ontology concepts (plus object types<br>                            and relationships where Business Ontology is not used)<br>LIVE  Live State            Current state, assignments, active commitments<br>SIM   Simulation            Scenarios, proposed plans, projected outcomes<br>DEC   Decision              Proposals, approvals, commands, observations</pre><p>These are application-defined schemas.</p><p>Separating LIVE, SIM, and DEC prevents hypothetical changes from being confused with actual operations.</p><p>Each operational snapshot should carry a state version, observation time, and source information. Simulation results should also identify the snapshot and model version used to produce them.</p><h4>Operational semantics: Capabilities, constraints, and actions</h4><p>The operational model defines what resources can do and which rules apply.</p><p>In our warehouse, these include worker qualifications, station capacities, order-handling requirements, and shipping deadlines.</p><p>Action types such as REASSIGN_WORKER and REROUTE_ORDERS describe parameters, preconditions, and expected state transitions.</p><p>These definitions can live in governed Snowflake tables and views, linked to canonical business concepts.</p><p>Execution handlers should be explicitly registered and tested. Metadata should not become arbitrary executable code.</p><p>The same rules should inform simulation and execution validation, so an action considered feasible during planning is subject to consistent checks before dispatch.</p><h4>Decision services: Simulation with Snowpark</h4><p>Simulation answers a question that an ontology alone cannot: <em>What happens if we change something?</em></p><p>For PB-04, a resource-scheduling engine can compare staffing and routing plans using processing rates, capacity, queue sizes, and carrier deadlines.</p><p>Snowpark Python stored procedures can implement these calculations close to Snowflake data. More demanding simulations can run in specialized services hosted through Snowpark Container Services.</p><p>A common application interface might look like:</p><pre>result = simulate(<br>    snapshot=current_state,<br>    actions=proposed_plan,<br>    constraints=operational_rules<br>)</pre><p>This is a conceptual interface, not a built-in Snowflake function. In practice it might be the signature of a Snowpark Python stored procedure that reads a LIVE snapshot and writes results to SIM.</p><p>A simulation returns feasibility, projected outcomes, constraint violations, assumptions, and model version information.</p><p>Results belong in an isolated scenario context and must not modify authoritative operational state.</p><p>Not every action requires simulation. Straightforward changes may need only validation. Simulation is most useful when decisions involve competing objectives, uncertainty, or downstream effects.</p><h4>AI reasoning: Cortex Agents</h4><p><strong>Cortex Agents provide the primary AI orchestration layer</strong> in this architecture.</p><p>Structured questions are grounded in Snowflake Semantic Views. Cortex Search provides access to operating procedures, policies, and incident reports. Custom tools connect the agent to ontology relationships, validation services, and simulations.</p><p>The warehouse manager asks:</p><p><em>“How can we protect the 5:00 PM carrier departure?”</em></p><p>The agent retrieves operational metrics, identifies affected orders, consults relevant procedures, and requests simulations of feasible recovery plans.</p><p>If the manager adds a constraint such as <em>“Keep PB-03 fully staffed”</em>, the agent can request another evaluation.</p><p>The agent explains the options. Operational services remain responsible for calculations, feasibility checks, and authorization.</p><h4>Governed execution: Write-back and verification</h4><p>Simulation evaluates possible changes. <strong>Governed write-back connects an approved decision to an operational system.</strong></p><p>An application can expose typed Snowflake stored procedures for proposing, approving, dispatching, and verifying actions.</p><p>For example:</p><pre>SP_PROPOSE_DECISION(...)<br>SP_APPROVE_DECISION(...)<br>SP_DISPATCH_ACTIONS(...)<br>SP_VERIFY_OUTCOME(...)</pre><p>These are illustrative application procedures, not built-in Snowflake functions.</p><p>Between proposal and dispatch sits a <strong>human authorization gate</strong>: the agent can propose, but only an authorized role can approve.</p><p>The execution path should enforce authorization, action parameters, current-state versions, and idempotency.</p><p>An adapter then sends the approved request to the responsible system, such as a WMS. Snowflake retains the decision record, command history, acknowledgements, and observations.</p><p>Streams and tasks can support asynchronous processing on supported sources. Hybrid tables do not support streams, and triggered tasks cannot monitor them, so state changes held in hybrid tables need a supported event source (for example, an event or outbox table on a standard table) or an application-managed dispatcher. For low-latency external commands, an application service or SPCS-hosted adapter may be more appropriate.</p><p>The distinction between command acceptance and operational completion matters. A WMS accepting a rerouting request does not prove that the orders were reassigned.</p><p>Verification requires observed evidence.</p><h4>Closing the loop on PB-04</h4><p>Here is how the lifecycle plays out for the 1:30 PM failure. The plans and steps below are illustrative.</p><ol><li><strong>Observe.</strong> LIVE shows PB-04 as unavailable, with its order queue growing against the 5:00 PM carrier commitment.</li><li><strong>Understand.</strong> The ontology links PB-04 to the queued orders, the qualified workers, and the stations that can process the same order types.</li><li><strong>Evaluate.</strong> The agent requests simulations of candidate plans, for example:</li></ol><ul><li><em>Plan A:</em> reassign two qualified workers from PB-03 to PB-02.</li><li><em>Plan B:</em> reroute PB-04’s queue across PB-01 and PB-02 without moving workers.</li><li><em>Plan C:</em> combine a partial reroute with one worker reassignment.</li><li>Each result in SIM records its feasibility, projected readiness at 5:00 PM, any constraint violations, and the snapshot and model version it used.</li></ul><p><strong>4. Decide.</strong> The manager adds <em>“Keep PB-03 fully staffed”</em>, which makes Plan A infeasible. The agent explains the trade-off between B and C, and the manager approves Plan C through SP_APPROVE_DECISION.</p><p>5. <strong>Execute.</strong> SP_DISPATCH_ACTIONS checks that the state version still matches the simulation, then sends the REROUTE_ORDERS and REASSIGN_WORKER commands to the WMS with idempotency keys. The WMS acknowledges both.</p><p>6.<strong> Verify.</strong> Acknowledgement is not completion. SP_VERIFY_OUTCOME compares the observed assignments and queue depth against the simulated forecast, records the variance, and marks the decision verified only when the orders are confirmed ready for the carrier.</p><p>Every step leaves a record: the question, the simulations considered, the constraint that ruled out Plan A, the approver, the commands, and the observed result.</p><p>Figure 2 summarizes this loop, with the PB-04 example at its center.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A1B0oI-Or4BTe0WUY__CXw.png" /><figcaption><strong>Figure 2. From Operational State to Verified Outcome.</strong> The six-step decision loop applied to the PB-04 example. A human authorization gate separates Decide from Execute, and observed feedback from Verify returns to Observe. Proposals and forecasts never overwrite observed state.</figcaption></figure><h3>4. Four Contracts That Make the Architecture Reusable</h3><p>A reference architecture becomes useful when applications can adopt it without rebuilding the same infrastructure.</p><p>We propose four logical contracts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ayEN_LXNNjegDi_laqH7jA.png" /></figure><p>These contracts define how components communicate without prescribing domain-specific algorithms.</p><p>The simulation contract can support warehouse scheduling, supply chain allocation, or data center thermal modeling. The action contract can support a WMS command, ERP transaction, or infrastructure-management request.</p><p>The verification contract distinguishes accepted instructions from confirmed changes.</p><p>To make this concrete, an Action record for the PB-04 plan might carry fields like these (illustrative):</p><pre>action_type      REASSIGN_WORKER<br>parameters       { worker_id: W-117, from_station: PB-04, to_station: PB-02 }<br>preconditions    worker qualified for PB-02; PB-02 below capacity<br>state_version    LIVE snapshot used by the approved simulation<br>policy           requires OPS_MANAGER approval<br>approval         approved_by, approved_at<br>idempotency_key  decision_id + action sequence</pre><p>Its Verification record shows why acceptance is not the same as completion:</p><pre>action           REASSIGN_WORKER (W-117 → PB-02)<br>command_status   ACKNOWLEDGED by WMS at 1:42 PM<br>expected_effect  W-117 at PB-02; 42 orders ready by 5:00 PM<br>observed_state   W-117 at PB-02; 41 orders ready at 5:00 PM<br>outcome          VERIFIED_WITH_VARIANCE<br>forecast_variance    −1 order (held for an inventory shortage)</pre><p>The WMS acknowledged the command at 1:42 PM, but the decision is verified only at 5:00 PM, from observed state. The variance feeds back into the simulation model.</p><p><strong>Share the contracts and decision lifecycle. Specialize the operational models and execution logic.</strong></p><h3>5. Applying the Framework Across Three Domains</h3><p>We are exploring this architecture through three reference applications on Snowflake.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QM8zefjcbTN__k5tB0IA_w.png" /></figure><p>The domain models are different, and they should remain different.</p><p>A generic simulation engine should not attempt to model supply chain material flow, warehouse resource scheduling, and data center thermal behavior using the same algorithms.</p><p>What can be shared are the operational contracts, state-versioning conventions, decision lifecycle, governance patterns, and Cortex Agent tool interfaces.</p><p>The applications are synthetic reference implementations under development. They illustrate the architecture rather than production integration with physical enterprise systems.</p><p>Figure 3 shows what is shared and what is specialized.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*25eUDLdyiFtZS2pjGjenLQ.png" /><figcaption><strong>Figure 3. Shared Contracts, Specialized Operational Behavior.</strong> The top band holds the four shared contracts and their fields, together with the common decision lifecycle, versioning, governance, and agent tool interfaces. The bottom band shows how each ATLAS application specializes its objects, constraints, simulation engine, actions, execution systems, and outcomes.</figcaption></figure><h3>6. What This Means for Enterprise AI Agents</h3><p>As enterprise agents gain access to tools that can change operational state, they need reliable data, explicit constraints, trusted evaluation services, and controlled action interfaces.</p><p>The framework assigns these responsibilities to separate components.</p><p>The agent investigates and recommends. The decision service evaluates feasibility and consequences. The action service enforces authorization and policies. The execution system performs the operation. Observations establish the outcome.</p><p>This separation makes decisions easier to inspect.</p><p>A rejected plan can identify the violated constraint. A simulation can expose its assumptions. An executed action can be traced to its requester, approval, command, and observed result.</p><p>It also limits the authority of the agent without limiting its ability to help people make decisions.</p><h3>7. From Operational Ontology to Digital Twins</h3><p>Operational ontology provides the meaning, constraints, and action contracts needed to support decisions.</p><p>A digital twin builds on that foundation with a representation of operational state and behavior over time.</p><p>In our warehouse, the ontology determines which staffing and routing changes are valid. A digital twin can project how queues evolve, visualize the facility, and reconcile predictions with observed outcomes, across supply chain and data center domains as well.</p><p>The common architecture remains:</p><p><strong>Governed Meaning → Operational State → Feasible Decisions → Governed Actions → Verified Outcomes</strong></p><p>Snowflake provides the data, semantic, AI, computation, and governance capabilities. Domain-specific services supply the operational behavior.</p><p>In the next article, we’ll explore how this framework supports AI-powered digital twins on Snowflake to examine temporal state, simulation, visualization, and verified outcomes.</p><h3>Reference</h3><p><a href="https://github.com/snowflake-Labs/ontology-on-snowflake"><em>Ontology on Snowflake</em></a></p><p><a href="https://docs.snowflake.com/en/LIMITEDACCESS/snowflake-horizon/business-ontology"><em>Snowflake Business Ontology</em></a><em> (Public preview soon)</em></p><p><a href="https://docs.snowflake.com/en/LIMITEDACCESS/cortex-sense"><em>Cortex Sense</em></a><em> (Public preview soon)</em></p><p><a href="https://aisystemslabhq.substack.com/p/enterprise-world-model-giving-agents"><em>Enterprise World Model: Giving Agents a World to Operate In</em></a></p><p><strong>About the Author</strong>: <a href="https://www.linkedin.com/in/tianxiajia/"><em>Tianxia Jia</em></a><em> is an AI and Cloud expert, specializing in architecting cutting-edge AI/ML solutions on Snowflake and AWS.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6f3ef270d3f2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/snowflake/operational-ontology-on-snowflake-from-business-meaning-to-governed-ai-action-6f3ef270d3f2">Operational Ontology on Snowflake: From Business Meaning to Governed AI Action</a> was originally published in <a href="https://medium.com/snowflake">Snowflake Builders Blog: Data Engineers, App Developers, AI, &amp; Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Streamlit’s Floor, React’s Ceiling, Inside Snowflake]]></title>
            <link>https://medium.com/snowflake/streamlits-floor-react-s-ceiling-inside-snowflake-c78f90e39dc8?source=rss----34b6daafc07---4</link>
            <guid isPermaLink="false">https://medium.com/p/c78f90e39dc8</guid>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[dashboard]]></category>
            <category><![CDATA[react]]></category>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[streamlit]]></category>
            <dc:creator><![CDATA[Vinod S]]></dc:creator>
            <pubDate>Thu, 12 Nov 2026 18:01:04 GMT</pubDate>
            <atom:updated>2026-11-12T18:01:03.960Z</atom:updated>
            <content:encoded><![CDATA[<h4>Keep the simplicity of Streamlit in Snowflake, and move the interactions that a rerun cannot do to the front end</h4><p>Picture a revenue review. An analyst opens the pipeline dashboard and points at EMEA on the regional bar chart. A tooltip appears. The other charts and the KPIs do not change.</p><p>She wants more. She wants every chart and KPI to show the EMEA numbers while she points. Then she wants to drag across two quarters on the trend line and watch bookings and the commit share change during the drag. That is how she finds the quarter where the commit share fell. Without these interactions, she must click, wait for the rerun, read the result and click again, for every guess.</p><p>I build many Streamlit in Snowflake (SiS) apps. I use SiS because <strong>it is the simplest way that I know to show data to people, not to mention that you can have an app up and running with a </strong><a href="https://www.snowflake.com/en/developers/guides/build-streamlit-apps-with-agent-skills/"><strong>SINGLE PROMPT</strong></a><strong>. </strong>I did not want to move my apps out of SiS. I wanted SiS to support these interactions. I call the result <strong>RiSiS</strong> (React in SiS). This post shows how RiSiS works, what it costs, and one open question.</p><p><strong>TL;DR:</strong> react-in-streamlit-in-snowflake is an open example. It adds a React island to a SiS page with st.html. An island is one React component at one position on a Streamlit page. Python and SiS continue to control the session, the queries, the permissions and the deploy. The island adds three interactions that a rerun cannot do:</p><ul><li>hover previews across all charts</li><li>KPIs that change during a drag</li><li>linked charts that change together</li></ul><p>The example is free under Apache-2.0 and runs in your own account.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/1*jSXtIsMfeSToE-0AgbaDfg.gif" /></figure><h3>Why I use SiS</h3><p>Streamlit has a low floor. A chart is three lines of Python. SiS adds the parts that I would otherwise build myself:</p><ul><li><strong>One language for the data.</strong> Snowpark, pandas and the queries stay in Python.</li><li><strong>No hosting work.</strong> Snowflake runs the app. Users open the app in Snowsight with their usual sign-in.</li><li><strong>Known permissions.</strong> The app queries with the privileges of its owner (<a href="https://docs.snowflake.com/en/developer-guide/streamlit/object-management/security">security overview</a>). A role controls what the app can see.</li></ul><p>The teams that I work with run hundreds of these apps. To rewrite each app for a better hover is not a realistic plan.</p><h3>What a rerun cannot do</h3><p>By default, when a user changes a widget, <strong>your script reruns from top to bottom</strong>. This model makes Streamlit simple. It also means that the page cannot respond to an event that Python does not receive.</p><p><strong>“Can a hover on one chart change another chart?”</strong> Not across separate elements. In one chart, Altair, Plotly and ECharts show hover and brush effects in the browser. Altair can also link views in one combined spec. Between separate Streamlit elements, Python is the only connection. Selections are the only chart events that Python receives (<a href="https://docs.streamlit.io/develop/api-reference/charts/st.altair_chart">st.altair_chart</a>).</p><p><strong>“Can the page change during a drag?”</strong> No. Python receives a brush selection when the pointer pauses or when the user releases the mouse button. In one test, I dragged for three seconds, and Streamlit did not rerun.</p><p>This is not a fault in Streamlit. It is also not Python against React, because Streamlit’s own front end uses React. The difference is where the work runs. A rerun is back-end data processing: queries, permissions and results. A hover or a drag is front-end data processing on data that the browser already has. The open question is where the front-end work should live.</p><h3>Try these first</h3><p>You may not need this example.</p><ul><li><a href="https://docs.streamlit.io/develop/concepts/architecture/fragments"><strong>Fragments</strong></a> rerun only part of the script. <a href="https://docs.streamlit.io/develop/concepts/architecture/fragments"><strong>Keyed fragments</strong></a> let a callback rerun only the fragment that you name.</li><li><a href="https://docs.streamlit.io/develop/quick-reference/release-notes/2026">on_change=&quot;ignore&quot;</a> lets a widget change in the browser without a rerun (Streamlit 1.63).</li><li><a href="https://docs.streamlit.io/develop/api-reference/custom-components/st.components.v2.component"><strong>Components v2</strong></a> adds your own HTML, CSS and JavaScript inline in the page, and sends values back to Python. For one complex widget, start with Components v2.</li></ul><p>One limit applies. The warehouse runtime supports Streamlit up to 1.52.2 and does not support Components v2 (<a href="https://docs.snowflake.com/en/developer-guide/streamlit/limitations">limitations</a>). Thus, the newer features need the container runtime. The repo uses <a href="https://docs.streamlit.io/develop/api-reference/text/st.html">st.html</a> with unsafe_allow_javascript=True, which works on both runtimes. On the container runtime, the same pattern could use a Components v2 mount.</p><h3>What the island does</h3><p>The demo dashboard, Instant BI, shows a sales pipeline. The data is synthetic. It comes from the TPC-H sample data that every Snowflake account has. The queries map that data in SQL. Orders become deals, order priority becomes deal stage, and the five TPC-H regions map to AMER, EMEA and APAC. The demo uses no customer data. I planted one pattern: in 2025-Q2, half of the Commit deals sit at Best case. The commit share falls from about 20% to about 10% in that quarter, and the sample data has no such dip. The dashboard has four KPIs, five charts and two drill-down tables. One React tree draws the charts and the KPIs. The tree receives every pointer movement:</p><ul><li><strong>Hover preview.</strong> Point at EMEA. All other charts and all KPIs show the EMEA values until the pointer leaves the bar. No click and no rerun occur.</li><li><strong>Drag preview.</strong> Drag across the time series. The KPIs and the breakdowns change on every frame. You find the useful range when you watch the values, without many attempts.</li><li><strong>Linked charts.</strong> Click a bar on any chart. All other charts apply the filter. Bars and numbers move to their new values with a short animation. If you turn on reduced motion in your operating system, they change immediately.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wsOc7Locy6axi_68uZ432g.png" /><figcaption>Hover on EMEA. The industry and deal-stage charts show the EMEA values in blue. Every KPI shows the EMEA value before a click (synthetic data).</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OVTNI-4Cvkx8lPTKJc0ROQ.png" /><figcaption>Mouse button still down during a drag across 2025–01 to 2025–06. The commit share reads 14.9%, against 19.6% for all deals, and the Deal stage chart shows fewer Commit deals (synthetic data, with the planted dip).</figcaption></figure><p>To make this possible, the browser receives the deal cube (49,000 rows, about 1 MB of JSON) one time per session. The browser filters the cube. As a result, clicks also respond fast. <a href="https://github.com/sfc-gh-vshiv/react-in-streamlit-in-snowflake/blob/main/docs/measurements.md">docs/measurements.md</a> has my measurements of both apps. The measurements cover one dashboard at one data size. Thus, speed is a secondary benefit. The interactions are the primary goal.</p><p>The repo also has a <strong>native twin</strong>. The twin is the same dashboard in plain Streamlit, with cache_data, a fragment and native selections. It gives the same answers. If the twin is sufficient for your users, use the twin.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VbUg0qthz0PyVTi7FjDihQ.png" /><figcaption>Same click, same answers: Instant BI (left) and the native twin (right) after a click on EMEA (synthetic data).</figcaption></figure><h3>No rewrite necessary</h3><p>You add an island to one position on a Streamlit page. All other parts of the page stay plain Streamlit: widgets, st.dataframe, layout and sign-in. Start with the chart that causes the most problems. Stop when the page meets your needs.</p><p><strong>Where App Runtime fits.</strong> For a new app with a full web stack, use <a href="https://docs.snowflake.com/en/developer-guide/snowflake-app-runtime/about-snowflake-app-runtime">Snowflake App Runtime</a>. App Runtime builds and hosts the app for you. As a test, I moved this React UI to App Runtime as a Next.js app. The result works well. The trade-off is that you write the server side yourself: API routes for the data, query code and input checks.</p><p>RiSiS is for a different case. You have Streamlit apps, and you want better interactions without a migration.</p><h3>How a request goes to Python</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cJp6p3v6xJap359jgbIpOQ.png" /><figcaption>The path of a request: React writes JSON to a hidden input. The rerun runs an allow-listed handler, and slow SQL goes to a background query. The answer goes back as props in the same rerun.</figcaption></figure><p>Python continues to control the app:</p><ol><li>load_bundle() adds the code to the page one time per session.</li><li>bridge_request() reads the hidden input.</li><li>handle() runs only the actions that you list.</li><li>Slow SQL goes through query_job(). This function starts the query with <a href="https://docs.snowflake.com/en/developer-guide/snowpark/reference/python/latest/snowpark/api/snowflake.snowpark.DataFrame.collect_nowait">collect_nowait()</a>. It returns &quot;pending&quot; while the browser polls for the result.</li><li>render_island() sends all results back to the island as props.</li></ol><h3>Who can see what</h3><ul><li><strong>Owner’s rights.</strong> setup.sql creates a role that can see only the sample data. The app queries as that role.</li><li><strong>The bridge is an RPC surface.</strong> Only allow-listed actions run. Inputs go through validators. The bridge binds SQL values. It does not paste values into the SQL text.</li><li><strong>Errors stay on the server.</strong> The browser receives a reference number. The details go to the log.</li><li><strong>Uploads are explicit.</strong> deploy.sh uploads only the files that deploy.manifest lists. It refuses files that look like credentials.</li></ul><h3>What you lose</h3><p>An island is outside the Streamlit model. Thus, some tasks that Streamlit does for you become your tasks:</p><ul><li><strong>Theme and consistency.</strong> The island does not use your Streamlit theme. It uses the styles that you write.</li><li><strong>Accessibility.</strong> You must make keyboard access, focus and screen-reader labels work inside the island.</li><li><strong>Tools that read the page.</strong> A tool that reads a Streamlit app through its native elements sees much less inside a custom island.</li><li><strong>More code.</strong> A UI change needs Node and a bundler. Each user downloads a 790 KB bundle one time per session. A deploy does not need Node, because the repo includes the built bundles.</li><li><strong>A dependency on the sanitiser</strong> in Streamlit 1.52 (<a href="https://docs.snowflake.com/en/developer-guide/streamlit/app-development/dependency-management">dependency management</a>). The repo pins the Streamlit version. A test checks the sanitiser rule.</li></ul><p>If a fragment and a good chart library meet your needs, use them. Plain Streamlit needs less code.</p><h3>An open question</h3><p>Streamlit’s own front end uses React. Thus, this pattern does not have to stay a workaround. More front-end processing in Streamlit could give these interactions without a custom island. A community proposal asks for <a href="https://github.com/streamlit/streamlit/issues/17242">client-side state and events in Streamlit</a>. In that proposal, a hover or a drag changes the page, and Python runs only when necessary.</p><p>I want to know what you need:</p><ul><li>Which interactions do your users ask for?</li><li>Do you want them as native Streamlit features, or as a simpler way to add your own front end?</li></ul><p><strong>The Streamlit team at Snowflake is open to feedback! </strong>While working with them to review this article, I was thoroughly impressed at how responsive the team is and how deeply they are looking at more native solutions. Hit up or mention <a href="https://medium.com/u/859434caf1b6">Yavor Georgiev</a> in the comments here if you want to continue the conversation.</p><h3>Try it</h3><p><strong>The fast method.</strong> Copy the sis-to-react-in-sis skill into Snowflake CoCo. Then ask it to convert your app. The skill tests the app locally first. It deploys only when you tell it to deploy.</p><p><strong>The manual method.</strong> Clone <a href="https://github.com/sfc-gh-vshiv/react-in-streamlit-in-snowflake">the repo</a>. Install and configure the <a href="https://docs.snowflake.com/en/developer-guide/snowflake-cli/index">Snowflake CLI</a>. Then run these commands:</p><pre># once, as ACCOUNTADMIN, after editing the settings at the top of the file<br>snow sql -c &lt;connection&gt; -f setup/setup.sql</pre><pre># the bundles are committed, so no Node is needed to deploy<br>bash starter/deploy.sh app REACT_SIS_DEMO.APPS.INSTANT_BI --warehouse COMPUTE_WH \<br>  -r REACT_SIS_DEMO -c &lt;connection&gt; --title &quot;Instant BI&quot;<br>bash starter/deploy.sh native_twin REACT_SIS_DEMO.APPS.INSTANT_BI_NATIVE --warehouse COMPUTE_WH \<br>  -r REACT_SIS_DEMO -c &lt;connection&gt; --title &quot;Instant BI (native)&quot;</pre><p>Open Snowsight. Go to <strong>Projects</strong>, then <strong>Streamlit</strong>. Hover, drag and click in each app. To use the container runtime, add --container --pool &lt;POOL&gt;. setup/uninstall.sql removes all objects.</p><p>This repo is an example, provided as is under Apache-2.0. Review it before you depend on it.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c78f90e39dc8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/snowflake/streamlits-floor-react-s-ceiling-inside-snowflake-c78f90e39dc8">Streamlit’s Floor, React’s Ceiling, Inside Snowflake</a> was originally published in <a href="https://medium.com/snowflake">Snowflake Builders Blog: Data Engineers, App Developers, AI, &amp; Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Same Table, Different Revenue: Apache Ossie and Snowflake to the Rescue]]></title>
            <link>https://medium.com/snowflake/same-table-different-revenue-apache-ossie-and-snowflake-to-the-rescue-a26e5c99f5c5?source=rss----34b6daafc07---4</link>
            <guid isPermaLink="false">https://medium.com/p/a26e5c99f5c5</guid>
            <category><![CDATA[data-governance]]></category>
            <category><![CDATA[semantic-layer]]></category>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[daanalytics]]></category>
            <category><![CDATA[apache-ossie]]></category>
            <dc:creator><![CDATA[Daan Bakboord]]></dc:creator>
            <pubDate>Wed, 11 Nov 2026 22:01:02 GMT</pubDate>
            <atom:updated>2026-11-11T22:01:01.942Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8Yp1cp7CTPE1DON7.png" /></figure><p>Ask two tools for revenue from the same orders table and both can return a different, defensible number. Here is what that looks like in the demo.</p><p>The difference is <strong>12,350</strong>. Both queries read the same orders and do the arithmetic correctly. What changes is which orders qualify as revenue. In this demo, revenue counts only when an order is completed. Apache Ossie and a Snowflake semantic view make that rule explicit.</p><p>That definition is context. It is the contract that tells a person, a dashboard, or an AI agent which reading of the table is allowed to be called revenue.</p><h3>Context is the contract</h3><p>A table stores values. Context says what those values mean when someone asks a business question. For a metric, the contract has a few parts:</p><ul><li>the expression, such as a sum or a count</li><li>the rows that qualify, such as a status filter</li><li>the entities it can be sliced by, and how those entities join</li><li>the other metrics it is built from</li><li>a description that a human and a model can both read</li></ul><p>Leave any of those out and the column name does the rest of the work. total_amount looks like revenue. It also looks like revenue before discounts, and like revenue that includes orders that never completed. An analyst picks one reading in a worksheet. A dashboard picks another. An agent, given the same column list, picks a third and sounds sure about it.</p><p>The sample model shows the distinction with three metrics on the same orders table.</p><p><strong>total_revenue</strong> sums total_amount where order_status = &#39;completed&#39;. On this data that is 103,200.</p><p><strong>gross_revenue</strong> uses the same completed orders and adds discount_amount back. Same orders, different question: value before the discount rather than after it.</p><p><strong>active_customers</strong> counts distinct customers with an order in completed, shipped, or processing. It is a wider rule than revenue. The two metrics can disagree. In this sample, the four open orders belong to customers who also have a completed order, so the customer count does not move. The next customer who has only a shipped order would show up in one metric and not the other.</p><p>average_order_value is not a fresh formula. It is total_revenue / order_count. Because it is built from the completed-order metrics, it keeps their rule. A hand-written average over every row in ORDERS would not.</p><h3>Joins are context too</h3><p>Metrics say what to aggregate. Relationships say which path is safe.</p><p>An order belongs to one customer. A line item belongs to one order and one product. Those are many-to-one links, and they matter as soon as someone asks for revenue by region or by category.</p><p>Region lives on the customer, not on the order. The path is order to customer, which is many orders to one customer. That is a safe way to group order-level revenue. Category lives on the product, and reaching it means going through line items. If you sum orders.total_amount after joining line items, every multi-line order gets counted once per line. The relationship tells the query engine, and an agent, to use line-level revenue for a category cut and order-level revenue for a region cut.</p><p>The model keeps those as separate metrics on purpose. total_revenue is completed order amounts. line_revenue and items_sold are sums on line items, and they carry no status filter. If category revenue should also exclude open orders, that rule has to be added to line_revenue itself. Putting it only in one dashboard query means the next consumer will not see it.</p><h3>Apache Ossie writes the contract once</h3><p><a href="https://ossie.apache.org/?ref=daanalytics.nl">Apache Ossie</a> (Incubating) is an open specification for a semantic layer. It was known as Open Semantic Interchange (OSI) until the project was renamed to avoid confusion with other projects that share the OSI acronym. The project is incubating at the Apache Software Foundation, which keeps the format open and free of a single controlling vendor.</p><p>An Ossie model is a YAML file that a person and a tool can both read. The e-commerce model in this demo names its sources ( orders, customers, products, order_items), the relationships between them, the dimensions used to slice results, and the metrics above. total_revenue in that file carries a sum, a completed-order filter, a description, and tags. customer_segment lists the only values it may take: Enterprise, Mid-Market, and SMB. order_month is defined once as DATE_TRUNC(&#39;month&#39;, order_date), so two dashboards do not invent two truncations.</p><p>The project mascot is a kangaroo carrying semantic metadata from one system to the next. The part worth keeping from that image is the carrying. A definition should survive the move from a YAML file to a warehouse, a BI tool, and an agent.</p><p>Ossie is the interchange format. It is not the engine that runs the query. Something still has to execute the contract. In this demo, that something is a Snowflake semantic view.</p><h3>Snowflake runs the same contract</h3><p>A <a href="https://docs.snowflake.com/en/sql-reference/sql/create-semantic-view?ref=daanalytics.nl">Snowflake semantic view</a> stores the model next to the tables. Logical tables point at the physical ones. Relationships use primary keys. Dimensions and metrics become objects a query can name. Comments keep the business description on each object.</p><p>A filter label is not enough to hold a metric rule. A WHERE clause the caller might forget does not protect the definition. The rule belongs inside the expression:</p><pre>orders.total_revenue AS SUM( IFF(orders.order_status = &#39;completed&#39;, orders.total_amount, NULL) ) COMMENT = &#39;Sum of total_amount for completed orders. Shipped, processing, and cancelled orders are excluded.&#39;</pre><p>average_order_value then refers to that metric, and to order_count, which carries the same predicate:</p><pre>orders.average_order_value AS orders.total_revenue / NULLIF(orders.order_count, 0)</pre><p>revenue_per_customer divides completed revenue by active customers. Each input keeps its own status rule, so the ratio does not quietly widen revenue to include shipped orders. The disagreement between the two definitions stays visible in the model.</p><p>Mapping the Ossie file onto the view is fairly direct:</p><h3>The query names the metric</h3><p>Once the view exists, a question becomes a request for a metric and a dimension. Revenue by region does not restate the sum, the status test, or the join:</p><pre>SELECT * FROM SEMANTIC_VIEW( OSSIE_DEMO.ECOMMERCE.ecommerce_semantic_model DIMENSIONS customers.region METRICS orders.total_revenue, orders.order_count ) ORDER BY total_revenue DESC;</pre><p>The view applies total_revenue as it was defined. Another worksheet, a Streamlit app, or Cortex Analyst asks for the same name and gets the same contract. The SQL in the demo app is the request. The meaning stays in ecommerce_semantic_model.</p><p>Active customers by month is the query that shows two contracts at once. active_customers includes shipped and processing orders, and total_revenue on the same result does not. One shared WHERE clause would have forced both onto a single rule and hidden the difference.</p><h3>Agents need this more than another copy of the SQL</h3><p>A person who knows the business can argue about a metric and settle it. An agent will not. Give it table and column names, and it writes a plausible query: sum the amount, join whatever foreign keys it can see, keep every status. Give it a semantic model, and the legal moves are the metrics, dimensions, and relationships in that model.</p><p>That is why this gets urgent as soon as natural-language questions reach the warehouse. An agent is only as consistent as the context it is given. Ossie is a way to write that context so it is not trapped in one product. A Snowflake semantic view is a way to make the warehouse enforce it, with comments and synonyms a model can read and expressions the engine will actually run.</p><p>The same YAML can be converted toward other consumers. Define the contract once, then let each engine run it. A semantic layer that exists only inside one tool is just another copy of the problem.</p><h3>Try the demo</h3><p>The demo is a small Streamlit in Snowflake app over this e-commerce model.</p><ol><li>01_setup.sql creates OSSIE_DEMO.ECOMMERCE and the sample tables.</li><li>ossie_semantic_model.yaml is the Ossie contract.</li><li>02_semantic_view.sql creates ecommerce_semantic_model, with the status rules inside the metrics.</li><li>The app’s live-query tab selects from that view.</li></ol><p><a href="https://www.daanalytics.nl/content/files/library/ossie-demo-walkthrough.mp4?v=1793712152029330852">Open the demo video separately</a></p><p><strong>Code:</strong> <a href="https://github.com/daanalytics/apache-ossie-demo?ref=daanalytics.nl">github.com/daanalytics/apache-ossie-demo</a></p><p>Apache Ossie is incubating at the Apache Software Foundation. The specification, mailing lists, and community are at <a href="https://ossie.apache.org/?ref=daanalytics.nl">ossie.apache.org</a>. Snowflake’s write-up of the project is <a href="https://www.snowflake.com/en/blog/apache-ossie-open-semantic-interchange-incubator/?ref=daanalytics.nl">on the Snowflake blog</a>.</p><p>The warehouse already knows the amounts. Context is what makes one of those amounts revenue, in every tool that asks.</p><p><em>Originally published at </em><a href="https://www.daanalytics.nl/apache-ossie-semantic-layer/"><em>https://www.daanalytics.nl</em></a><em> on November 3, 2026.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a26e5c99f5c5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/snowflake/same-table-different-revenue-apache-ossie-and-snowflake-to-the-rescue-a26e5c99f5c5">Same Table, Different Revenue: Apache Ossie and Snowflake to the Rescue</a> was originally published in <a href="https://medium.com/snowflake">Snowflake Builders Blog: Data Engineers, App Developers, AI, &amp; Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Clean-Looking Addresses Still Fail Delivery. Here Is How to Catch Them in Snowflake]]></title>
            <link>https://medium.com/snowflake/clean-looking-addresses-still-fail-delivery-here-is-how-to-catch-them-in-snowflake-32cd51d5f4c4?source=rss----34b6daafc07---4</link>
            <guid isPermaLink="false">https://medium.com/p/32cd51d5f4c4</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[address-validation]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[data-quality]]></category>
            <dc:creator><![CDATA[Jan Rehorik]]></dc:creator>
            <pubDate>Wed, 11 Nov 2026 20:01:01 GMT</pubDate>
            <atom:updated>2026-11-11T20:01:01.664Z</atom:updated>
            <content:encoded><![CDATA[<h3>Clean-Looking Addresses Still Fail Delivery. Here Is How to Catch Them</h3><h4>How to validate and standardize US addresses in Snowflake using SQL</h4><p>One source writes “26 S Chestnut St.” Another writes “26 s chstnt, vntura, californa” with the wrong ZIP code. Both are the same building in Ventura, and both will break a join, a dedupe rule, or a mailing run.</p><p>Formatting rules in SQL can make strings look alike. They cannot tell you whether an address is a real delivery point, or whether it is missing the suite number the carrier needs. That takes validation against postal reference data, and in most stacks it means an export, an API client, and a re-import to maintain.</p><p>In this tutorial, you’ll use SQL to submit addresses for validation, review the results, and decide what happens to each record. You will:</p><ol><li>Install Service Objects’ <strong>US Address Validation and Verification for Snowflake</strong> app from the Marketplace.</li><li>Shape your address columns into the input the app expects, without changing your source table.</li><li>Validate a batch of records with one CALL.</li><li>Read the results and route each record with SQL.</li></ol><p>Every screenshot and result below comes from a real run in a Snowsight worksheet.</p><h4><strong>Architecture Diagram</strong></h4><figure><img alt="The app runs in your account. Only the address fields you select cross the boundary, through one integration you approve." src="https://cdn-images-1.medium.com/max/1024/1*Qm63FhlhD-3DI1PIkDI-Pw.png" /><figcaption><em>The app runs in your account. Only the address fields you select cross the boundary, through one integration you approve.</em></figcaption></figure><p>The app is installed into your Snowflake account and exposes a stored procedure. You pass it a SELECT statement and an output table. It reads those rows, sends the address fields to the Service Objects API for validation, and writes the results into the table you named. From there, the results are ordinary Snowflake data.</p><p>The external access integration is the one point where data leaves your account. The app sends address data to the Service Objects API through a single external access integration you grant and revoke, and retains nothing. Data is encrypted in transit over HTTPS. That boundary is worth showing your security reviewer up front.</p><h3>Why formatting rules are not enough</h3><p>REPLACE(address, ‘Street’, ‘St’) handles predictable differences. It cannot answer the questions that decide what happens to a record next:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YLKE4Yv3B881aga_oStXzg.png" /></figure><p>Delivery Point Validation (DPV) is the signal that answers them. It tells you whether an address is a recognized delivery point, and separates “confirmed” from “valid building, bad unit” and “valid building, missing unit.” Those are different actions in a pipeline, so the routing logic later in this tutorial is built around the DPV code.</p><h3>Step 1: Install the app and activate the trial</h3><ol><li>Open the <strong>US Address Validation and Verification for Snowflake</strong> listing on the Snowflake Marketplace and select <strong>Try Now</strong>.</li><li>Snowflake shows the permission the app needs: CREATE EXTERNAL ACCESS INTEGRATION, for the call to the Service Objects API. Select <strong>Agree &amp; continue</strong>.</li><li>Under <strong>Options</strong>, name the application. The examples below use AV3. Then select <strong>Start trial</strong>.</li><li>When the install finishes, review the external access integration on the <strong>External connections</strong> screen and select <strong>Proceed to app</strong>.</li><li>In the app, open the <strong>Licensing</strong> tab, enter your contact details, and select <strong>Activate License</strong>. The status changes to a trial plan showing your transaction limit and expiry date.</li></ol><p>The free trial runs for 90 days or 1,000 records, whichever comes first, with no credit card and no automatic charge.</p><p><strong>One catch on Snowflake trial accounts.</strong> External access is restricted on Snowflake’s own trial accounts, and approving the permission alone does not lift that. Use a paid account, or ask Snowflake to enable external access on your trial account first.</p><h3>Step 2: Shape the input</h3><p>The procedure expects seven columns, in this order. Every column must appear in your SELECT, even when it holds no value for a record.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-hJXOxF0kL-vdewXONpUdQ.png" /></figure><p>Your CRM will not use these names. Rather than altering the source table, create a view that maps your columns:</p><pre>CREATE OR REPLACE VIEW MY_DB.PUBLIC.CUSTOMERS_FOR_VALIDATION AS<br>SELECT<br>  CUSTOMER_ID     AS CONSUMER_ROW_ID,<br>  COMPANY_NAME    AS BUSINESSNAME,<br>  STREET_ADDRESS  AS ADDRESS,<br>  UNIT            AS ADDRESS2,<br>  CITY,<br>  STATE,<br>  ZIP_CODE        AS POSTALCODE<br>FROM MY_DB.PUBLIC.CUSTOMERS;</pre><p>To follow along without real data, create a test table with the four records used in this post:</p><pre>CREATE OR REPLACE TABLE MY_DB.PUBLIC.TEST_ADDRESSES (<br>  CONSUMER_ROW_ID NUMBER,  BUSINESSNAME VARCHAR,<br>  ADDRESS VARCHAR,         ADDRESS2 VARCHAR,<br>  CITY VARCHAR,            STATE VARCHAR,<br>  POSTALCODE VARCHAR<br>);<br> <br>INSERT INTO MY_DB.PUBLIC.TEST_ADDRESSES VALUES<br>  (1, NULL, &#39;26 S Chestnut&#39;, NULL, &#39;Ventura&#39;, &#39;CA&#39;, &#39;93033&#39;),<br>  (2, NULL, &#39;26 s chstnt&#39;, NULL, &#39;vntura&#39;, &#39;californa&#39;, &#39;93101&#39;),<br>  (3, NULL, &#39;908 N 11th St&#39;, NULL, NULL, NULL, &#39;58501&#39;),<br>  (4, NULL, &#39;10000 E Cota St&#39;, NULL, NULL, NULL, &#39;93101&#39;);</pre><h3>Step 3: Validate with one call</h3><p>The procedure takes two arguments: the SELECT that picks your records, and the fully qualified name of the results table. Use fully qualified table names inside the SELECT as well, as in the example. Dollar-quoting the first argument saves you from escaping quotes inside it:</p><pre>CALL AV3.APP_CODE.AV3_GET_BEST_MATCHES(<br>  $$SELECT CONSUMER_ROW_ID, BUSINESSNAME, ADDRESS, ADDRESS2,<br>           CITY, STATE, POSTALCODE<br>    FROM MY_DB.PUBLIC.TEST_ADDRESSES$$,<br>  &#39;MY_DB.PUBLIC.AV3_RESULTS&#39;<br>);</pre><p>The call returns a one-line summary of the run. Here it is from our 36-record test:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*3f3nAKTTqNOoMDmR3iCYOA.png" /></figure><p>The results land in exactly the table you named. Because the input is a query, you control scope and consumption: validate a handful of records first, then a sample of production data, then the full set.</p><h3>Step 4: Read the results</h3><pre>SELECT CONSUMER_ROW_ID, ADDRESS1, ADDRESS2, CITY, STATE, ZIP,<br>       DPV, DPV_DESC, CORRECTIONS_DESC, IS_RESIDENTIAL,<br>       ERROR_TYPE, ERROR_DESC<br>FROM MY_DB.PUBLIC.AV3_RESULTS<br>ORDER BY CONSUMER_ROW_ID;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZQLoF6k_S6653WjV5qjtXA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZrYnX0Eg0iUNdyFNhV6mIQ.png" /><figcaption><em>Results from the test run in Snowsight.</em></figcaption></figure><p>Here is what came back for the four records from Step 2:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DsChhHNgi502u7zvE61MjA.png" /></figure><p>Two things are worth noticing. The misspelled second record resolves to the same delivery point as the first, so the two now match. And the third record, which arrived with only a street and a ZIP code, came back with its city, state, and ZIP+4 filled in.</p><p>The results table has more columns than this. These are the typical ones you can build rules on:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yTDCMl5Qxv8DTy_7fBMS3A.png" /></figure><p>DPV is a text column. Its documented values are:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GM3m9Kk4ngVeVNCWXxlBPw.png" /></figure><h3>Step 5: Route records with SQL</h3><p>The app produces the signals. Your team decides what each one means. Here is one way to encode that as a view, with errors handled first because they carry no DPV value:</p><pre>CREATE OR REPLACE VIEW MY_DB.PUBLIC.ADDRESS_ROUTING AS<br>SELECT<br>  r.CONSUMER_ROW_ID,<br>  r.ADDRESS1, r.ADDRESS2, r.CITY, r.STATE, r.ZIP,<br>  r.DPV, r.DPV_DESC, r.CORRECTIONS_DESC, r.ERROR_DESC,<br>  CASE<br>    WHEN r.ERROR_TYPE IS NOT NULL THEN &#39;FIX_AT_SOURCE&#39;<br>    WHEN r.DPV = &#39;1&#39;              THEN &#39;ACCEPT&#39;<br>    WHEN r.DPV IN (&#39;3&#39;, &#39;4&#39;)      THEN &#39;NEEDS_UNIT&#39;<br>    ELSE                               &#39;INVESTIGATE&#39;<br>  END AS ROUTE<br>FROM MY_DB.PUBLIC.AV3_RESULTS r;</pre><p>A quick count tells you how much work each queue holds:</p><pre>SELECT ROUTE, COUNT(*) AS RECORDS<br>FROM MY_DB.PUBLIC.ADDRESS_ROUTING<br>GROUP BY ROUTE<br>ORDER BY RECORDS DESC;</pre><p>For review, join back to the source so a reviewer sees what the customer entered next to what validation returned. The original value is never overwritten:</p><pre>SELECT<br>  c.CUSTOMER_ID,<br>  c.STREET_ADDRESS  AS ORIGINAL_ADDRESS,<br>  r.ADDRESS1        AS VALIDATED_ADDRESS,<br>  COALESCE(r.ERROR_DESC, r.DPV_DESC) AS REASON,<br>  r.ROUTE<br>FROM MY_DB.PUBLIC.CUSTOMERS c<br>JOIN MY_DB.PUBLIC.ADDRESS_ROUTING r<br>  ON r.CONSUMER_ROW_ID = c.CUSTOMER_ID<br>WHERE r.ROUTE &lt;&gt; &#39;ACCEPT&#39;;</pre><p>Where you draw the lines depends on the workload. A mailing run might hold back everything except ACCEPT. A customer master job might accept NEEDS_UNIT records for matching while flagging them for outreach.</p><h3>Running it in production</h3><ul><li><strong>Validate only what changed.</strong> The input is a query, so filter it. This run picks up only records created or edited in the last day:</li></ul><pre>CALL AV3.APP_CODE.AV3_GET_BEST_MATCHES(<br>  $$SELECT v.*<br>    FROM MY_DB.PUBLIC.CUSTOMERS_FOR_VALIDATION v<br>    JOIN MY_DB.PUBLIC.CUSTOMERS c<br>      ON c.CUSTOMER_ID = v.CONSUMER_ROW_ID<br>    WHERE c.UPDATED_AT &gt;= DATEADD(day, -1, CURRENT_TIMESTAMP())$$,<br>  &#39;MY_DB.PUBLIC.AV3_RESULTS_DAILY&#39;<br>);</pre><ul><li><strong>Keep a stable row identifier.</strong> Every result joins back to its source on CONSUMER_ROW_ID, so use a key that does not change between runs.</li><li><strong>Store results beside the source, not over it.</strong> Keeping the original value makes it possible to audit a change, explain it to a customer, and roll it back.</li><li><strong>Give every outcome a destination.</strong> Errors, missing units, and invalid units are normal in real data. Decide where each one goes before you go live.</li><li><strong>Know what validation does not tell you.</strong> A valid delivery point confirms that the address exists and can receive mail. It does not confirm who lives there, or that a customer has not moved.</li><li><strong>Review the boundary with security.</strong> Walk your reviewer through the external access integration, the address fields that cross it, and how to revoke it.</li></ul><h3>Next steps</h3><p>Run the four test records above, then a few hundred of your own that cover clean addresses, missing units, and known problem rows. Check the spread of DPV values and errors before you decide how the routing view should treat each outcome.</p><p>The app is listed on the Snowflake Marketplace as <a href="https://app.snowflake.com/marketplace/listing/GZ1MJZ4VGDFVX/service-objects-inc-us-address-validation-and-verification-for-snowflake"><strong>US Address Validation and Verification for Snowflake</strong>.</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=32cd51d5f4c4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/snowflake/clean-looking-addresses-still-fail-delivery-here-is-how-to-catch-them-in-snowflake-32cd51d5f4c4">Clean-Looking Addresses Still Fail Delivery. Here Is How to Catch Them in Snowflake</a> was originally published in <a href="https://medium.com/snowflake">Snowflake Builders Blog: Data Engineers, App Developers, AI, &amp; Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Agents Are Code Too: What I Learned Shipping a Cortex Agent Through Azure DevOps]]></title>
            <link>https://medium.com/snowflake/agents-are-code-too-what-i-learned-shipping-a-cortex-agent-through-azure-devops-fab3adae29e8?source=rss----34b6daafc07---4</link>
            <guid isPermaLink="false">https://medium.com/p/fab3adae29e8</guid>
            <category><![CDATA[ci-cd-pipeline]]></category>
            <category><![CDATA[snowflake-cortex]]></category>
            <category><![CDATA[azure-devops]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[ai-agent]]></category>
            <dc:creator><![CDATA[Sanket Prabhu]]></dc:creator>
            <pubDate>Wed, 11 Nov 2026 15:01:07 GMT</pubDate>
            <atom:updated>2026-11-11T15:01:06.096Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>Building the agent took a week. Getting it safely from sandbox to production took longer, and taught me more.</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IoiTayWx6bS1MZhlCQbokA.png" /></figure><p>Most Cortex Agent content I read ends at the same place: a working demo in Snowsight, a clever question, a correct answer.</p><p>That is where my real work started.</p><p>In my recent project, I built a Cortex Agent for the finance domain. The agent itself came together quickly. What took real thought was the part nobody demos: moving it through <strong>sbx, dev, uat and prod</strong> with the same discipline we already applied to every other Snowflake object on the platform.</p><p>After roughly a decade of building data platforms, I’ve learned to be suspicious of anything that only exists in one environment. So here is what I learned treating an agent like code, and what I’d do the same way again.</p><blockquote>An agent you can’t rebuild from Git is a prototype, no matter how good its answers are.</blockquote><h3>1. The myth: “An agent is just a config in the UI”</h3><p>It’s easy to see a Cortex Agent as a few settings in a form: pick a semantic view, add a search service, write some instructions, done.</p><p>But look at what actually decides whether the agent gives a good answer:</p><ul><li><strong>Tool descriptions</strong> that tell the orchestrator when to use which tool</li><li><strong>Orchestration and response instructions</strong></li><li><strong>The semantic view</strong> behind Cortex Analyst, including its verified queries</li><li><strong>The Cortex Search service</strong> and the columns it exposes</li><li><strong>Grants</strong> that decide who can call the agent and what it can reach</li></ul><p>Every one of these is a behavior change. If someone edits a tool description in prod by hand, the agent can start routing questions differently, and you have no diff, no review and no way back.</p><p>On our platform, warehouses, roles, schemas and grants were already managed through Terraform and Azure DevOps. It didn’t make sense for the agent to be the one thing people clicked together.</p><p>So where does an agent actually live, if not in the UI?</p><h3>2. Everything the agent needs goes in the repo</h3><p>The good news is that Snowflake gives you a clean way to express an agent as code. CREATE AGENT ... FROM SPECIFICATION takes a YAML spec covering models, orchestration, instructions, tools and tool resources.</p><p>A trimmed version of the pattern I used looks like this:</p><pre>CREATE OR REPLACE AGENT &lt;% db %&gt;.AGENTS.FINANCE_AGENT<br>  COPY GRANTS<br>  COMMENT = &#39;Finance domain agent. Managed by CI/CD, do not edit by hand.&#39;<br>  FROM SPECIFICATION<br>  $$<br>  models:<br>    orchestration: auto</pre><pre>  orchestration:<br>    budget:<br>      seconds: 30<br>      tokens: 16000</pre><pre>  instructions:<br>    orchestration: &gt;<br>      Use FinanceAnalyst for questions about balances, flows and<br>      period-over-period numbers. Use PolicySearch for questions about<br>      definitions, methodology or policy documents.<br>    response: &quot;Answer concisely. State the period and source for every number.&quot;<br>    sample_questions:<br>      - question: &quot;What were net contributions last quarter?&quot;</pre><pre>  tools:<br>    - tool_spec:<br>        type: &quot;cortex_analyst_text_to_sql&quot;<br>        name: &quot;FinanceAnalyst&quot;<br>        description: &quot;Structured finance metrics from the curated finance semantic view.&quot;<br>    - tool_spec:<br>        type: &quot;cortex_search&quot;<br>        name: &quot;PolicySearch&quot;<br>        description: &quot;Finance policy and methodology documents.&quot;</pre><pre>  tool_resources:<br>    FinanceAnalyst:<br>      semantic_view: &quot;&lt;% db %&gt;.SEMANTIC.FINANCE_SV&quot;<br>    PolicySearch:<br>      search_service: &quot;&lt;% db %&gt;.SEARCH.FINANCE_POLICY_SEARCH&quot;<br>      max_results: &quot;5&quot;<br>  $$;</pre><p>A few choices here mattered more than I expected:</p><ul><li><strong>The comment says “do not edit by hand.”</strong> Small thing, but it stops well-meaning fixes in prod.</li><li><strong>COPY GRANTS with </strong><strong>OR REPLACE.</strong> Without it, every redeploy wipes the grants and someone&#39;s agent &quot;stops working&quot; the next morning.</li><li><strong>Environment is a variable, not a copy.</strong> One spec file, with the database name injected at deploy time. No finance_agent_dev.sql and finance_agent_prod.sql drifting apart.</li></ul><p>The repo ended up with a simple layout: agent spec, semantic view definition, search service DDL, grants, and an evaluation question set.</p><h3>3. The agent is the easy part. The semantic view is the dependency.</h3><p>This is the lesson I’d underline twice.</p><p>When the agent gave a bad answer in uat, the agent spec was rarely the cause. Usually it was the <strong>semantic view</strong>: a missing synonym, a metric defined slightly differently from how finance actually talks about it, or a verified query that didn’t cover a common question.</p><p>That changes the pipeline design. The semantic view has to deploy <strong>before</strong> the agent, in the same release, and it needs its own checks. If the agent and its semantic view ship on different schedules, you’re testing a combination that will never exist in prod.</p><p>It also changes ownership. The agent spec was mine. The semantic view needed review from people who knew the finance definitions. Having both in a pull request made that review natural instead of a meeting.</p><h3>4. The pipeline: same promotion path as everything else</h3><p>We didn’t invent a new release process for the agent. It rode the same Azure DevOps path as the rest of the platform: branch-based deployment, gated approvals before uat and prod.</p><p>Stripped down, the deploy stage looked roughly like this:</p><pre>- stage: deploy_uat<br>  dependsOn: deploy_dev<br>  jobs:<br>    - deployment: agent<br>      environment: snowflake-uat   # approval gate lives on this environment<br>      strategy:<br>        runOnce:<br>          deploy:<br>            steps:<br>              - checkout: self<br>              - script: |<br>                  snow sql -f semantic/finance_sv.sql   -D &quot;db=FIN_UAT&quot; -c uat<br>                  snow sql -f search/policy_search.sql  -D &quot;db=FIN_UAT&quot; -c uat<br>                  snow sql -f agents/finance_agent.sql  -D &quot;db=FIN_UAT&quot; -c uat<br>                  snow sql -f grants/finance_agent.sql  -D &quot;db=FIN_UAT&quot; -c uat<br>                displayName: Deploy semantic view, search, agent, grants<br>              - script: python evals/run_questions.py --env uat<br>                displayName: Smoke test with known questions</pre><p>Two things worth calling out:</p><ul><li><strong>Order is explicit.</strong> Semantic view, then search, then agent, then grants. Deploying them in one script avoids the “agent points at a view that doesn’t exist yet” failure.</li><li><strong>The service account deploying the agent is not a human.</strong> It has CREATE AGENT on the target schema and nothing more than it needs. Same least-privilege thinking we applied across the platform.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IufalCmy1ez4uP-8rb3GMQ.png" /></figure><h3>5. “It deployed” is not “it works”</h3><p>Here’s the trap. CREATE AGENT can succeed even when a tool configuration won&#39;t actually work at run time. Snowflake&#39;s own docs say so: test the agent after creating it.</p><p>So what does a test even look like for something non-deterministic?</p><p>What helped me was keeping it boring:</p><ul><li><strong>A fixed question set per domain.</strong> Questions finance actually asks, with the expected number or the expected source.</li><li><strong>Check routing, not just answers.</strong> Did a policy question go to search and a metrics question go to Analyst? A wrong route is often the first sign a tool description regressed.</li><li><strong>Run it in every environment after deploy.</strong> Same questions, same expectations.</li></ul><p>Snowflake has since published a developer guide that wires this into CI properly: evaluating the semantic view’s SQL correctness against verified queries, then evaluating the agent on answer correctness and tool selection accuracy, and only promoting when both pass. It’s worth a look if you’re starting today. Check the current release status of agent evaluations and versioning for your account before you depend on them.</p><h3>6. Grants are part of the release</h3><p>The last thing that bit people (me included) was access.</p><p>An agent needs USAGE on the search service and access to every object behind the semantic view, and the people calling it need USAGE on the agent. If any link in that chain is missing, the user just sees an agent that can’t answer.</p><p>What worked for me was treating agent grants exactly like our other role automation: defined in code, deployed with the agent, and mapped to the same domain roles we already used for the finance data. If you can see the finance data, you can use the finance agent. If you can’t, the agent shouldn’t be able to show it to you either.</p><h3>What I’d do the same way again</h3><ul><li>Put the agent, semantic view, search service and grants in one repo and one release</li><li>Inject environment names; never keep per-environment copies of the spec</li><li>Gate uat and prod with approvals, like any other production change</li><li>Run a fixed question set after every deploy and check tool routing, not just answers</li><li>Mark the agent as CI-managed so nobody “quick fixes” it in prod</li></ul><h3>TL;DR</h3><ol><li><strong>An agent is not a UI config.</strong> Its behavior lives in tool descriptions, instructions, semantic views and grants, and all of them need version control.</li><li><strong>CREATE AGENT ... FROM SPECIFICATION makes the agent code.</strong> Keep one spec and inject the environment at deploy time.</li><li><strong>The semantic view is the real dependency.</strong> Deploy it first, in the same release, and review it with the people who own the definitions.</li><li><strong>Use the promotion path you already have.</strong> Same branches, same gates, same service accounts.</li><li><strong>Deployed is not working.</strong> Test with a fixed question set and check routing.</li><li><strong>Grants ship with the agent.</strong> Map agent access to the same roles that govern the data.</li></ol><p>If you’re running Cortex Agents across environments, I’d love to hear how you’re testing them. That’s the piece I’m still refining.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fab3adae29e8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/snowflake/agents-are-code-too-what-i-learned-shipping-a-cortex-agent-through-azure-devops-fab3adae29e8">Agents Are Code Too: What I Learned Shipping a Cortex Agent Through Azure DevOps</a> was originally published in <a href="https://medium.com/snowflake">Snowflake Builders Blog: Data Engineers, App Developers, AI, &amp; Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Is your Iceberg catalog healthy?]]></title>
            <link>https://medium.com/snowflake/is-your-iceberg-catalog-healthy-939dc7817154?source=rss----34b6daafc07---4</link>
            <guid isPermaLink="false">https://medium.com/p/939dc7817154</guid>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[apache-iceberg]]></category>
            <category><![CDATA[apache-polaris]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[Daniel Hunt]]></dc:creator>
            <pubDate>Tue, 10 Nov 2026 22:38:46 GMT</pubDate>
            <atom:updated>2026-11-10T22:38:45.189Z</atom:updated>
            <content:encoded><![CDATA[<p><em>A comparison of five open-source Iceberg catalogs on functionality and OSS project health.</em></p><p>Apache Iceberg™ allows an organisation to own their data layer in an open format in their own storage, and has become the de facto open table format. Still, every engine must reach those Iceberg tables through an Iceberg catalog.</p><p>A natural extension is an open-source Apache Iceberg catalog to complete the open lakehouse stack, acting as an open governance layer, so that access to your tables isn’t dependent on any one vendor’s managed service.</p><p>In this blog, I aim to simplify the decision process when it comes to choosing an Iceberg catalog and therefore building an open lakehouse on Iceberg.</p><p>I focus on the functional completeness of each catalog for Iceberg data and dive deep on OSS project health. After all, an OSS project with one vendor driving development carries similar lock-in risk to a proprietary product. My goal here is to provide transparency on the relative health of common OSS catalogs, specifically Apache Polaris™, Apache Gravitino™, Lakekeeper, Unity Catalog OSS and Project Nessie.</p><p>The heart of the blog is a collation of public information from each project (which you can review in the <a href="https://gitlab.com/hunt.dan/iceberg-state-of-catalogs/-/blob/main/docs/oss-health.md?ref_type=heads">public Git repo</a>), but the criteria and analysis are informed by my experience as a specialist architect working with 50+ organisations implementing Apache Iceberg over the past couple of years.</p><p>This is a companion post to <a href="https://medium.com/snowflake/iceberg-catalogs-below-the-surface-2d646645f4ad">Iceberg Catalogs: Below the Surface</a> (updated November 2026), which scores six vendor-hosted catalogs and shares the same underlying repo. I split the two posts since the first one was becoming a dissertation!</p><p><em>Disclosure: This is my personal opinion, not representative of the views of my employer. I’m a specialist architect at Snowflake, one of the vendors contributing towards the Apache Polaris project. Every claim below is sourced and reproducible. Don’t take my word for it: test the claims, read the </em><a href="https://gitlab.com/hunt.dan/iceberg-state-of-catalogs/-/blob/main/docs/oss-health.md?ref_type=heads"><em>linked project docs</em></a><em>, and please comment if you spot any inconsistencies.</em></p><h3>Project health in one image</h3><p>After much iteration, this is the best single view of project health I could come up with: commit velocity against the number of contributing organisations.</p><p>If you’re skimming, this is the tl;dr version.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r4TR0v8ToR5qd9HNpGQ17A.png" /><figcaption>FIGURE 1: Velocity against organisational breadth over the past 18 months (based on attributed commits). Bubble size is the share of commits <em>not</em> from the primary organisation, i.e. a proxy for lower concentration risk.</figcaption></figure><p>Here, we see <strong>Apache Gravitino and Apache Polaris pull ahead </strong>of the other assessed catalogs. Both are shipping at <strong>~4x the velocity </strong>and with <strong>&gt;7x the number of contributing organisations</strong> compared to the two relatively inactive projects, Unity OSS and Nessie.</p><p>Between Gravitino and Polaris, a subtle difference is reflected in single-vendor concentration: only 15% of all commits for Gravitino came from organisations other than Datastrato, its lead contributor, while Polaris sees 32% of all commits from organisations other than Dremio.</p><p>Lakekeeper sits in the middle ground, with 8 contributing organisations and &gt;50% higher velocity than Unity OSS, while still clocking in below 50% of the leading catalogs on both metrics.</p><p>For a deep dive on the methodology and the code to reproduce it yourself, please take a look at the <a href="https://gitlab.com/hunt.dan/iceberg-state-of-catalogs/-/blob/main/docs/oss-health.md?ref_type=heads#methodological-caveats-read-before-quoting-these-numbers">underlying repo</a>.</p><p>In order to directly compare the catalogs, I also created a tabular version of the project health data, split into 5 criteria for a total of 10 possible points per catalog. This helps to highlight the differences between the various catalogs across various health criteria that don’t all fit on one chart.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PIlwwJFOkJQZZjOBLqTRzg.png" /><figcaption>FIGURE 2: Open-source project health, five dimensions graded 0 to 2, commits from May 2025 to October 2026. Change is relative to my previous assessment in July 2026. Full methodology in <a href="https://gitlab.com/hunt.dan/iceberg-state-of-catalogs/-/blob/main/docs/oss-health.md?ref_type=heads#methodological-caveats-read-before-quoting-these-numbers">the repo</a>.</figcaption></figure><p>We see <strong>Apache Polaris (10/10) and Apache Gravitino (9/10) take the lead again</strong>. Both demonstrate not only project velocity and diversity, but <strong>strong production references</strong>: Fivetran / dbt, Cloudera, and Snowflake in the case of Polaris (outside of lead contributor Dremio / SAP), and Pinterest and Bilibili in the case of Gravitino. The strength of Apache project governance also provides a strong foundation for future roadmap openness.</p><p><strong>Lakekeeper (5/10) has a strong IRC surface and two public production references</strong> (Fresha and Databaas), but remains single-vendor with no foundation backstop.</p><p><strong>Unity Catalog OSS (3/10) has improved on vendor diversity </strong>since July, though at 8.9% of commits from outside Databricks it sits under the 10% line, and it still has <strong>no public production reference </strong>(at least, none I could find).</p><p><strong>Nessie (0/10) grades zero on all five</strong>, and as such I may exclude it from future analyses (anyone have a recommendation for another catalog to take its place?).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5HM657vStb8_VJxasoVgWg.png" /><figcaption>FIGURE 3: Released capability as of 9 November 2026. A P marks a capability the project labels experimental or in preview, which is not counted (but could influence the score if released).</figcaption></figure><p>The field is closer on capability. Apache Polaris and Lakekeeper implement the full Iceberg REST spec, including multi-table commits, with authorisation and scoped credential vending. Polaris also federates remote IRC catalogs, though federation is off by default. Apache Gravitino federates through its connectors but currently lacks support for multi-table transactions. Unity Catalog OSS serves native Iceberg tables read and write but has no named roles, and Nessie still labels its IRC support experimental.</p><h3>Who writes the code?</h3><p>A single company can spread work across many employees and still own nearly all the commits, so headcount is a poor measure of community. The cleanest visual is the share of commits from different organisations, which shows the shared investment and balance in an open-source project. For projects largely driven by a single vendor, this is expected to influence the roadmap and transparency of how it is built.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wqd8ZZYdpHW4qR0kgD5BZA.png" /><figcaption>FIGURE 4: Share of identified commits by organisation, past 18 months. Shares are of identified commits, so they run higher than the shares of all commits quoted in the text (i.e. unattributed commits are excluded here).</figcaption></figure><p>Apache Polaris stands out on vendor neutrality, with Apache Gravitino an honourable mention. Unity Catalog OSS shows a visible outside share, and Lakekeeper and Nessie remain single-vendor.</p><p>Beyond the lead vendor, we see <strong>notably more large enterprises contributing</strong> towards each of the <strong>Apache Polaris</strong> (Zscaler, Snowflake, JPMorgan Chase, Bloomberg, Uber, Cloudera, Google, etc.) and <strong>Apache Gravitino</strong> (Alibaba, Roku, Uber, Cisco, ByteDance, many more) projects compared to others, providing another signal of enterprise readiness:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j6-QnADtRzzeZEvRHRAT2w.png" /><figcaption>FIGURE 5: Apache Polaris and Apache Gravitino show a significantly higher proportion of contributions attributed to large enterprises compared to the other assessed projects, providing a signal of enterprise buy-in. Apache Polaris shows the lowest single-vendor risk, with 60% of identified commits from Dremio.</figcaption></figure><h3>Open source as an exit path</h3><p>An open-source catalog provides a clear exit path from a managed service only if the managed service runs the same code as an open-core code, much like we see with other industry favourites such as dbt (hosted via dbt Labs, or self-hostable).</p><p>Of the assessed catalogs, Apache Polaris is integrated as the open core Iceberg catalog in multiple managed services: Snowflake Horizon, Dremio Cloud (now part of SAP), dbt / Fivetran, and Cloudera. As the OSS project evolves, these catalogs stand to benefit, and likewise the OSS project is more likely to receive continued contributions from these organisations given their dependency, leading to a mutual rising tide.</p><p>Unity Catalog OSS shares a name, a REST API and an Apache Spark™ connector with Databricks Unity Catalog, but the OSS server lacks production-readiness features such as named-role RBAC or federation, and I found no documented migration path from the managed service to the OSS edition. It is not clear whether the OSS project shares code with the proprietary, hosted product that shares its name.</p><p>If you’d like to dig deeper on managed catalogs, my companion blog covers this topic in detail.</p><h3>In summary</h3><p>Apache Polaris leads the open-source catalogs on both capability and project health, with Apache Gravitino close behind and Lakekeeper one to watch if you can accept a single vendor. Unity Catalog OSS is improving, but is not yet a production choice or an exit from managed Unity. Nessie is converging into Apache Polaris (by their <a href="https://projectnessie.org/blog/2024/08/02/open-source-polaris-announcement/">own admission</a>), so start a new project on Apache Polaris instead.</p><p>The release notes and contributor data are all public, and the repo has the links and the scripts to check them.</p><h3>Methodology</h3><ul><li><strong>Capability:</strong> rated from released versions and project documentation as of 9 November 2026, checked against the source code of each release.</li><li><strong>Project health:</strong> public git history (commits, excluding merges and bots) and merged pull requests (GitHub API) from 1 May 2025 to 31 October 2026, for the main repository of each project. A trailing 12-month view is kept in the repo as a sensitivity check.</li><li><strong>Attribution:</strong> a contributor counts towards an organisation through a public employer signal, such as a corporate email domain or the employer on their public GitHub profile, or through a curated identity mapping. Other contributors are left unattributed, so outside shares and each leader’s share of all commits are lower bounds. The curated mappings link individuals to employers, gitignored to respect contributors’ privacy.</li><li><strong>Privacy:</strong> results are published as organisation-level counts only; no individual contributor is named.</li></ul><h3>How to reproduce</h3><pre>git clone https://gitlab.com/hunt.dan/iceberg-state-of-catalogs<br>cd iceberg-state-of-catalogs<br><br># Commit history for the five open-source catalogs, 18 months (clones the repos on first run)<br>python3 scripts/fetch_catalog_health.py --since 2025–05–01 --until 2026–11–01 --out data/oss-health-2026–11.json<br><br># Merged pull requests, public signals only (needs the GitHub CLI: gh auth login)<br>python3 scripts/fetch_pr_attribution.py --since 2025–05–01 --until 2026–11–01 --extra-window 2025–11–01 2026–11–01 --out data/oss-pr-health-2026–11.json<br><br># Charts, then the health score, capability and other tables (the tables need Google Chrome)<br>CATALOG_HEALTH_DATA=data/oss-health-2026–11.json CATALOG_HEALTH_OUT=docs/assets/oss-2026–11 python3 scripts/plot_catalog_health.py<br>python3 scripts/render_oss_tables.py data/oss-health-scorecard-2026–11.yaml docs/assets/oss-2026–11/</pre><p>All figures land in docs/assets/oss-2026–11/; the per-cell evidence is in data/oss-catalogs-2026–11.yaml and data/oss-health-scorecard-2026–11.yaml. There are bound to be gaps in such a fast-moving and complex domain. If you spot something that’s missing or misrepresented, please drop a comment below. Keen to hear your thoughts!</p><h3>Sources</h3><p><strong>Open-source catalogs:</strong> <a href="http://polaris.apache.org, github.com/apache/polaris">Apache Polaris</a> · <a href="github.com/unitycatalog/unitycatalog">Unity Catalog OSS</a> · <a href="http://gravitino.apache.org">Apache Gravitino</a> · <a href="http://docs.lakekeeper.io">Lakekeeper</a> · <a href="http://projectnessie.org">Project Nessie</a>.</p><p><strong>Production references (Polaris):</strong> Fivetran, “<a href="fivetran.com/blog/getting-started-with-apache-polaris-catalog-in-fivetrans-managed-data-lake-service">Getting Started with Apache Polaris Catalog in Fivetran’s Managed Data Lake Service</a>” · Dremio, “<a href="dremio.com/blog/why-dremio-co-created-apache-polaris-and-where-its-headed">Why Dremio Co-Created Apache Polaris and Where It’s Headed</a>” · Cloudera, “<a href="cloudera.com/about/news-and-blogs/press-releases/2026–06–04-cloudera-adopts-apache-polaris-to-advance-open-governed-data-access-for-enterprise-ai-anywhere.html">Cloudera Adopts Apache Polaris to Advance Open, Governed Data Access</a>” · SAP, “<a href="https://news.sap.com/2026/05/sap-to-acquire-dremio-unify-sap-and-non-sap-data-power-agentic-ai/">SAP to Acquire Dremio</a>” — “<em>With Dremio, SAP will deliver a universal, open catalog built on Apache Polaris and the open Apache Iceberg REST Catalog API. It serves as both the discovery and semantic layer of SAP Business Data Cloud”.</em></p><p><em>Apache®, Apache Iceberg™, Apache Polaris™, Apache Gravitino™ and Apache Spark™ are trademarks of The Apache Software Foundation. Unity Catalog, Lakekeeper and Project Nessie are trademarks of their respective owners. Use here is nominative and does not imply endorsement.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=939dc7817154" width="1" height="1" alt=""><hr><p><a href="https://medium.com/snowflake/is-your-iceberg-catalog-healthy-939dc7817154">Is your Iceberg catalog healthy?</a> was originally published in <a href="https://medium.com/snowflake">Snowflake Builders Blog: Data Engineers, App Developers, AI, &amp; Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Support Transcripts to Trusted AI Agents: Building a Knowledge Architecture with Snowflake…]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/snowflake/from-support-transcripts-to-trusted-ai-agents-building-a-knowledge-architecture-with-snowflake-4569fe0ebd5a?source=rss----34b6daafc07---4"><img src="https://cdn-images-1.medium.com/max/2600/1*p5_Z3pEnLIZX-R5lMLwUPA.jpeg" width="3168"></a></p><p class="medium-feed-snippet">Enterprise support systems already contain years of troubleshooting knowledge. The challenge is transforming that history into structured&#x2026;</p><p class="medium-feed-link"><a href="https://medium.com/snowflake/from-support-transcripts-to-trusted-ai-agents-building-a-knowledge-architecture-with-snowflake-4569fe0ebd5a?source=rss----34b6daafc07---4">Continue reading on Snowflake Builders Blog: Data Engineers, App Developers, AI, &amp; Data Science »</a></p></div>]]></description>
            <link>https://medium.com/snowflake/from-support-transcripts-to-trusted-ai-agents-building-a-knowledge-architecture-with-snowflake-4569fe0ebd5a?source=rss----34b6daafc07---4</link>
            <guid isPermaLink="false">https://medium.com/p/4569fe0ebd5a</guid>
            <category><![CDATA[apache-iceberg]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[knowledge-graph]]></category>
            <category><![CDATA[snowflake]]></category>
            <dc:creator><![CDATA[Akshay Sharma]]></dc:creator>
            <pubDate>Tue, 10 Nov 2026 17:44:52 GMT</pubDate>
            <atom:updated>2026-11-10T17:44:51.196Z</atom:updated>
        </item>
    </channel>
</rss>