<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Databricks Community Articles - Medium]]></title>
        <description><![CDATA[Practical Databricks tutorials, architecture guides, AI insights, product updates, and best practices for modern data professionals + Databricks MVPs (Most Valuable Professionals) - Medium]]></description>
        <link>https://medium.com/databrickscommunity?source=rss----8dd843bb8b39---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Databricks Community Articles - Medium</title>
            <link>https://medium.com/databrickscommunity?source=rss----8dd843bb8b39---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 02:04:49 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/databrickscommunity" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com" rel="hub"/>
        <item>
            <title><![CDATA[Databricks new office in Amsterdam: a photo report 2026]]></title>
            <link>https://medium.com/databrickscommunity/databricks-new-office-in-amsterdam-a-photo-report-2026-adf1dfb0430d?source=rss----8dd843bb8b39---4</link>
            <guid isPermaLink="false">https://medium.com/p/adf1dfb0430d</guid>
            <category><![CDATA[databricks-events]]></category>
            <category><![CDATA[databricks-office]]></category>
            <category><![CDATA[amsterdam]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[databricks-news]]></category>
            <dc:creator><![CDATA[Maksim Pachkouski Databricks MVP]]></dc:creator>
            <pubDate>Sun, 15 Nov 2026 17:03:25 GMT</pubDate>
            <atom:updated>2026-11-15T17:03:25.225Z</atom:updated>
            <content:encoded><![CDATA[<p>9 november 2026 I attended the opening of Databricks new office in Amsterdam, learned about its interesting and unusual features, and learned about the benefits of being a Databricks MVP. Here&#39;s a short photo report.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MSLDCYRoc6_cWz1LM53bsQ.jpeg" /></figure><p>Databricks new office is located in Amsterdam&#39;s Zuidas business district, in The Rock building. It&#39;s conveniently accessible, just a couple of minutes&#39; walk from the metro station. A large bicycle rack is also located nearby.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*drK9AggOaEMmTZH28FdTdA.jpeg" /></figure><h3>New office and global plans</h3><p>Databricks plans to invest <strong>over $500 million in the Netherlands over the next three years</strong> .</p><p>Amsterdam is already Databricks largest European center. Here, teams are working on key components of the platform, including Genie, Lakewatch, and, of course, Lakehouse RT (Reyden).</p><p>As part of its expansion, Databricks plans to double its headcount in the Netherlands and develop partnerships with local companies, universities, and startups.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pt7IHKJawT60SY6PUBYfiw.jpeg" /></figure><p>The new office itself is also quite large: approximately <strong>13,000 m² across 11 floors</strong> . It is currently the largest Databricks hub in Europe.</p><h3>What does the office look like from the inside?</h3><p>At the entrance, of course, there is a reception, and I am a barista nearby, so that I don’t waste time at the nearest coffee shops and can concentrate on work.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J1EQFbqT9BYFMe6U6hFYGg.jpeg" /></figure><p>Hot meals and snacks are also provided for employees and guests.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ILLUmcAf89oMrWfRKfc2BQ.jpeg" /></figure><p>For relaxation and socializing, there&#39;s billiards, table football, and even a piano. I don&#39;t know what that&#39;s for.</p><p>There&#39;s also a racing simulator that runs on the Databricks App and collects metrics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NyMh2uD8EaHSndSc3XcdAQ.jpeg" /></figure><p>There&#39;s also a separate games room with board games and a PlayStation 5.</p><p>I think we might have to set up a reservation system after GTA VI comes out.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y4Hx3F0rnj_9YaiZYKmFDg.jpeg" /></figure><p>Databricks MVP organized a tour of the upper floors, which are still undergoing renovation. The modern building features heated ceilings, which is unusual.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7SHXQY3olsd4eUcSvfpcmw.jpeg" /></figure><p>Of course, the view from the window is simply amazing, I must definitely share:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9aUKfnaJE1zS39tsHgGVRA.jpeg" /></figure><h3>Chatting with the Databricks Team</h3><p>For me, the main value of such events is the opportunity to personally interact with the people developing the platform. A special meeting was organized for the MVP, where we discussed various topics and asked our questions. Many thanks to Jeremy van Doorn for organizing this meeting.</p><p>The good news is that if you created a Databricks App using Genie, Scale to Zero is already enabled for it, but for other Apps, you&#39;ll have to wait until next year.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-LimreC5wEs-TKRmE2IrmQ.jpeg" /></figure><p>I also had the opportunity to speak with Ari Kaplan and David Meyer, and we discussed the pet projects <a href="https://rebricked.org/"><strong>Rebricked</strong></a><strong> and </strong><a href="https://finopsway.com/"><strong>FinOpsWay</strong></a>. It was nice to receive feedback.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gJw7iN55b2L9cgKKaOq0Yg.jpeg" /></figure><h3>An autographed book to end the day</h3><p>At the end of the day, we were invited to see Maria Vechtomova, also a Databricks MVP. Maria&#39;s company rents office space on the ground floor of the building that now houses Databricks.</p><p>Maria recently released her book, <strong>MLOps with Databricks</strong> , published by O&#39;Reilly. We were lucky enough to receive our own signed copies!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WEp-7HLgh1hS_MJMzDItQg.jpeg" /></figure><p>Thanks to the Databricks team for the invitation, to all the MVPs for the engaging conversation, and to Maria for the hospitality and the book!</p><p>I hope these photos help at least somewhat convey the atmosphere of that day. In my next report, I&#39;ll tell you about the Databricks AI World Tour in Amsterdam and London and how it differs from DAIS.</p><p><em>Author:</em> <strong>Maksim Pachkouski</strong> [Databricks MVP &amp; Databricks Solutions Architect Champion]. Follow my posts on <a href="https://medium.com/@protmaks">Medium</a> and <a href="https://www.linkedin.com/in/protmaks/">LinkedIn</a> to stay up-to-date on the latest news in Data Engineering and learn about best practices.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=adf1dfb0430d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/databrickscommunity/databricks-new-office-in-amsterdam-a-photo-report-2026-adf1dfb0430d">Databricks new office in Amsterdam: a photo report 2026</a> was originally published in <a href="https://medium.com/databrickscommunity">Databricks Community Articles</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Ways to Add Business Knowledge (Context) on Databricks -UC Pages & UC Wikis:]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/databrickscommunity/ways-to-add-business-knowledge-context-on-databricks-uc-pages-uc-wikis-318cfd8ca01f?source=rss----8dd843bb8b39---4"><img src="https://cdn-images-1.medium.com/max/1022/1*3aUGkvd8R7z4jDFXy9tsUQ.png" width="1022"></a></p><p class="medium-feed-snippet">Most data platforms do a reasonably good job of telling us what data exists.</p><p class="medium-feed-link"><a href="https://medium.com/databrickscommunity/ways-to-add-business-knowledge-context-on-databricks-uc-pages-uc-wikis-318cfd8ca01f?source=rss----8dd843bb8b39---4">Continue reading on Databricks Community Articles »</a></p></div>]]></description>
            <link>https://medium.com/databrickscommunity/ways-to-add-business-knowledge-context-on-databricks-uc-pages-uc-wikis-318cfd8ca01f?source=rss----8dd843bb8b39---4</link>
            <guid isPermaLink="false">https://medium.com/p/318cfd8ca01f</guid>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[context-engineering]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[context-layer]]></category>
            <dc:creator><![CDATA[Hitesh Sahni]]></dc:creator>
            <pubDate>Sun, 15 Nov 2026 10:55:17 GMT</pubDate>
            <atom:updated>2026-11-15T10:55:15.739Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Spark 4.3 AUTO CDC: SCD2 Made Simpler]]></title>
            <link>https://medium.com/databrickscommunity/spark-4-3-auto-cdc-scd2-made-simpler-5e45e08748ff?source=rss----8dd843bb8b39---4</link>
            <guid isPermaLink="false">https://medium.com/p/5e45e08748ff</guid>
            <category><![CDATA[delta-lake]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[change-data-capture]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[Venugopal Dabbara]]></dc:creator>
            <pubDate>Sat, 14 Nov 2026 23:37:37 GMT</pubDate>
            <atom:updated>2026-11-14T23:37:35.881Z</atom:updated>
            <content:encoded><![CDATA[<p><em>What’s new in upstream Spark SQL, how selective history tracking works, and why Databricks users may already be familiar with it.</em></p><h3>SCD Type 2: Simple in Theory, Not Always in SQL</h3><p>If you’ve implemented Slowly Changing Dimension Type 2 (SCD2), you’ve probably thought:</p><p><em>“It’s just a MERGE statement. How complicated can it be?”</em> 😄</p><p>Then come the change-detection conditions, expiring old records, creating new versions, sequencing events, and maintaining the logic as business requirements evolve.</p><p>Custom SCD2 implementations are sometimes necessary, but they can also become surprisingly expensive to maintain.</p><p><strong>Apache Spark 4.3 introduces native declarative AUTO CDC SQL support, including selective SCD Type 2 history tracking.</strong></p><p>The particularly interesting addition is TRACK HISTORY ON, which lets you decide which attribute changes should create new historical versions.</p><h3>1. What’s New in Apache Spark 4.3?</h3><p>The key distinction is between <em>implementing</em> SCD2 and <em>declaring</em> how CDC events should be applied.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/717/1*-9RSn9kKcTVWChqhjzB1UA.png" /></figure><p>Spark 4.2 can implement SCD2 through custom logic. Spark 4.3 introduces SQL constructs that express CDC behavior directly within Spark Declarative Pipelines.</p><p>This doesn’t make every SCD2 problem disappear. It does reduce the amount of custom history-management logic needed for supported CDC patterns.</p><p><strong>The most useful improvement? Not every changed column needs to create another historical record.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/497/1*w_zj6rfiVNnID8jjoPQUmw.png" /><figcaption>Apache Spark 4.2 vs. 4.3 AUTO CDC capabilities, selective SCD2 history tracking, and existing Databricks support.</figcaption></figure><h3>2. Understanding Selective History Tracking</h3><p>Imagine a customer dimension containing:</p><ul><li>customer_id: Business key</li><li>name: Customer name</li><li>city: Customer location</li><li>phone: Contact number</li></ul><p>Suppose the business wants to preserve historical versions whenever a customer’s name or city changes, but doesn’t need a new version for every phone number update.</p><p>Spark 4.3 supports an explicit inclusion list:</p><pre>TRACK HISTORY ON (name, city)</pre><p>It also supports an exclusion-based form:</p><pre>TRACK HISTORY ON * EXCEPT (phone)</pre><p>The first form explicitly identifies tracked attributes. The second tracks all applicable attributes except phone.</p><p>These forms are useful for different schema-management preferences: an explicit list keeps the tracked attributes tightly controlled, while an exclusion list is convenient when most attributes should trigger history.</p><p>The examples below use the explicit inclusion list.</p><h3>3. A Customer CDC Example</h3><p>Consider the following CDC events, ordered by sequence_num.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/692/1*WmXTNgHQ1RGYYIea08hwOQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/502/1*NJ73nJej-mMznnmNcKxOuw.png" /></figure><p>Two different kinds of updates appear here:</p><p><strong>Phone-only updates:</strong> Alice’s phone changes at sequence 2, and Bob’s at sequence 5. Neither changes the attributes selected for history tracking.</p><p><strong>Business-relevant update:</strong> Alice moves from London to Manchester at sequence 3. Because city is tracked, this creates a new historical version.</p><p>With TRACK HISTORY ON (name, city), the resulting SCD2 records are:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/722/1*MSCAugo6_uWt9TSE94LzqA.png" /></figure><p>Notice the subtle but important behavior: <strong>phone-only changes update the applicable record without creating another historical version.</strong></p><p>Alice’s London record therefore contains phone 222, even though the original insertion had phone 111.</p><p>The sequence-based boundaries identify the historical versions, and a NULL end value indicates the current version.</p><h3>Full history vs. selective history</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/741/1*xMfK8OLWvpyS6dISUhuraA.png" /></figure><p>Selective tracking creates fewer historical records for this example while retaining the city change that matters to the business.</p><p>That is a reduction in version count, not a measured claim about storage consumption or query performance.</p><h3>4. Declarative AUTO CDC SQL in Spark 4.3</h3><p>A representative upstream Spark 4.3 flow definition looks like this:</p><pre>CREATE FLOW customer_cdc_flow AS<br>AUTO CDC INTO customers<br>FROM stream(customers_cdc)<br>KEYS (customer_id)<br>APPLY AS DELETE WHEN operation = &#39;DELETE&#39;<br>SEQUENCE BY sequence_num<br>COLUMNS * EXCEPT (operation, sequence_num)<br>STORED AS SCD TYPE 2<br>TRACK HISTORY ON (name, city)</pre><p>The important clauses are straightforward:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/632/1*A6qBIyZj3HEt2naiK41h2g.png" /></figure><p>AUTO CDC flows are defined within Spark Declarative Pipelines. The source and target must be configured in the appropriate pipeline context, this is not a standalone batch SQL replacement for MERGE.</p><h3>Inclusion versus exclusion syntax</h3><p>If the requirement is to track only a known set of columns:</p><pre>TRACK HISTORY ON (name, city)</pre><p>If the requirement is to track all columns except selected ones:</p><pre>TRACK HISTORY ON * EXCEPT (phone)</pre><p>One small syntax detail matters: in upstream Spark 4.3, the exclusion form is written as * EXCEPT (..), not wrapped inside an additional pair of parentheses.</p><p>Also, TRACK HISTORY ON belongs with STORED AS SCD TYPE 2 it isn&#39;t a history-tracking option for SCD Type 1.</p><h3>4.1 Hands-On: Validating AUTO CDC SCD2 in Spark 4.3 OSS</h3><p>The SQL looks simpler. But does it actually work? 😄</p><p>I tested <strong>Apache Spark 4.3.0 OSS</strong> on Windows 11 (Python 3.12.10, Java 21) using its native Declarative Pipelines CLI.</p><p><strong>Setup note:</strong> AUTO CDC requires a target supporting row-level operations. Plain Parquet fails with AUTOCDC_TARGET_DOES_NOT_SUPPORT_MERGE, so this lab used an <strong>experimental, locally source-built Iceberg adapter</strong> for Spark 4.3 / Scala 2.13 — not an officially published connector.</p><h4>Step 1 — Create and populate the CDC source</h4><p>Using the five Alice/Bob events from Section 3:</p><pre>CREATE TABLE cdc_source (<br>    customer_id INT,<br>    name STRING,<br>    city STRING,<br>    phone STRING,<br>    operation STRING,<br>    sequence_num BIGINT<br>)<br>USING iceberg;</pre><pre>INSERT INTO cdc_source VALUES<br>(101, &#39;Alice&#39;, &#39;London&#39;,     &#39;111&#39;, &#39;INSERT&#39;, 1),<br>(101, &#39;Alice&#39;, &#39;London&#39;,     &#39;222&#39;, &#39;UPDATE&#39;, 2),<br>(101, &#39;Alice&#39;, &#39;Manchester&#39;, &#39;222&#39;, &#39;UPDATE&#39;, 3),<br>(102, &#39;Bob&#39;,   &#39;Leeds&#39;,      &#39;333&#39;, &#39;INSERT&#39;, 4),<br>(102, &#39;Bob&#39;,   &#39;Leeds&#39;,      &#39;444&#39;, &#39;UPDATE&#39;, 5);</pre><h4>Step 2 — Define the native AUTO CDC pipeline</h4><pre>CREATE STREAMING TABLE history<br>USING iceberg<br>TBLPROPERTIES (&#39;format-version&#39;=&#39;2&#39;);</pre><pre>CREATE FLOW history_flow AS AUTO CDC INTO history<br>FROM STREAM(cdc_source)<br>KEYS (customer_id)<br>SEQUENCE BY sequence_num<br>COLUMNS * EXCEPT (operation, sequence_num)<br>STORED AS SCD TYPE 2<br>TRACK HISTORY ON (name, city);</pre><p>I tested two additional policies in separate pipelines by changing the final clause:</p><p><strong>All-column SCD2</strong></p><pre>STORED AS SCD TYPE 2;</pre><p><strong>Exclude phone from history tracking</strong></p><pre>STORED AS SCD TYPE 2<br>TRACK HISTORY ON * EXCEPT (phone);</pre><h4>Step 3 — Execute and inspect</h4><p>With the connector and catalog configured, each pipeline ran through Spark’s packaged CLI:</p><pre>spark-pipelines run --spec spark-pipeline.yaml</pre><p><em>The Windows execution also required connector classpath and catalog settings.</em></p><p>The resulting history tables were queried in separate Spark processes:</p><pre>SELECT customer_id, name, city, phone,<br>       __START_AT, __END_AT<br>FROM history<br>ORDER BY customer_id, __START_AT;</pre><p><strong>Actual results — all three history policies</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*51ccid181wgka9nJQVfejw.png" /></figure><p>All three configurations completed successfully, producing <strong>5, 3, and 3 historical versions</strong>, respectively.</p><p>Notice how the phone-only updates create additional versions under all-column SCD2, but not when tracking only name/city or excluding phone. The latest phone values are still reflected, while Alice’s move to Manchester creates a new historical version.</p><h4>Step 4 — Validate incremental processing</h4><p>A rerun without new events left the selective-history result unchanged. I then appended Bob’s move to York:</p><pre>INSERT INTO cdc_source VALUES<br>(102, &#39;Bob&#39;, &#39;York&#39;, &#39;444&#39;, &#39;UPDATE&#39;, 6);</pre><p>Rerunning the same pipeline with its existing checkpoint produced the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*taI29bmAZqpyrhfhkArIZA.png" /></figure><p>Bob’s Leeds version now ends at sequence <strong>6</strong>, and his York version begins at <strong>6</strong>. The selective-history table increased from <strong>3 to 4 versions</strong>, confirming successful incremental processing.</p><p><strong>The takeaway:</strong> Spark 4.3 OSS successfully executed native AUTO CDC with selective SCD2 history tracking in this lab. The feature simplifies history management, provided the target supports row-level operations.</p><p><em>Lab scope: The results were obtained using an experimental Iceberg adapter. Official Spark 4.3 connector compatibility, production readiness, and performance/storage improvements were not evaluated.</em></p><p>Here’s an important distinction.</p><p><strong>Although native AUTO CDC SQL and selective SCD2 tracking are new to upstream Apache Spark 4.3, this functionality is already available in existing Databricks Lakeflow capabilities.</strong></p><p>Databricks supports declarative AUTO CDC through Lakeflow pipelines and standalone streaming tables.</p><p>For example, a Lakeflow SQL flow can express the same selective-history requirement:</p><pre>CREATE FLOW customer_cdc_flow AS<br>AUTO CDC INTO customers<br>FROM stream(customers_cdc)<br>..</pre><p>The flow is defined within the appropriate Lakeflow pipeline context, with the source and target streaming tables configured accordingly.</p><p>Databricks also provides an embedded FLOW AUTO CDC form for standalone streaming tables, using its own streaming-table DDL conventions.</p><p>This distinction matters when comparing platforms:</p><ul><li><strong>Spark 4.2 OSS:</strong> SCD2 requires custom implementation rather than native AUTO CDC SQL.</li><li><strong>Spark 4.3 OSS:</strong> Introduces declarative AUTO CDC and selective history tracking upstream.</li><li><strong>Existing Databricks:</strong> Already supports the equivalent selective-history behavior through Lakeflow and standalone streaming tables.</li></ul><p>So, if you’re already using Databricks, you don’t need to wait for a Spark 4.3-based runtime to explore selective SCD2 history tracking.</p><p>The upstream Spark and Databricks SQL surfaces aren’t completely interchangeable, so use the syntax documented for your platform.</p><h3>6. When Should You Use TRACK HISTORY ON?</h3><p>Selective history tracking makes sense when different attributes have different historical significance.</p><p>For example, a customer dimension might contain attributes such as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/572/1*U_3fNsrvj5KbUhtr27Z7tg.png" /></figure><p>The correct choices depend on the business.</p><p>A phone number might be irrelevant to historical reporting in one system but legally or operationally significant in another.</p><p>That’s why TRACK HISTORY ON is useful: it allows the SCD2 strategy to reflect actual reporting requirements rather than treating every column equally.</p><p>There is also an important modelling consideration. <strong>If you choose not to track an attribute, don’t expect to reconstruct every historical value of that attribute from the SCD2 table.</strong></p><p>Selective history tracking is a deliberate trade-off between the history you preserve and the history you don’t need.</p><h3>7. Does This Replace Custom SCD2 MERGE Logic?</h3><p>Not universally.</p><p>Custom implementations may still be appropriate for specialized business rules, unusual source-event behaviour, or transformations that don’t fit the supported declarative CDC model.</p><p>But for compatible CDC workloads, declarative AUTO CDC can remove much of the repetitive work around identifying changes and managing historical versions.</p><p>And selective tracking adds another benefit: history is created based on the attributes that matter, not simply because <em>something</em> changed.</p><h3>Final Thoughts</h3><p>Apache Spark 4.3 brings <strong>declarative AUTO CDC and selective SCD Type 2 history tracking into upstream Spark SQL</strong>.</p><p>The practical value is straightforward:</p><p><strong>Less custom history-management logic, more control over what gets historized, and a clearer connection between business requirements and pipeline behavior.</strong></p><p>For Databricks users, this is also a reminder that the capability is already available through existing Lakeflow functionality.</p><p>Sometimes the best way to simplify a complicated SCD2 MERGE isn&#39;t to add another condition.</p><p>It’s to stop writing all those conditions yourself. 😄</p><h3>References</h3><ul><li><a href="https://issues.apache.org/jira/browse/SPARK-58261">Apache Spark — SPARK-58261</a></li><li><a href="https://spark.apache.org/docs/latest/declarative-pipelines-programming-guide.html#change-data-capture-cdc-with-auto-cdc">Apache Spark Declarative Pipelines — SCD Type 2 (AUTO CDC) Documentation</a></li><li><a href="https://docs.databricks.com/aws/en/ldp/cdc">Databricks AUTO CDC documentation</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5e45e08748ff" width="1" height="1" alt=""><hr><p><a href="https://medium.com/databrickscommunity/spark-4-3-auto-cdc-scd2-made-simpler-5e45e08748ff">Spark 4.3 AUTO CDC: SCD2 Made Simpler</a> was originally published in <a href="https://medium.com/databrickscommunity">Databricks Community Articles</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Lakeflow Jobs Triggers]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/databrickscommunity/lakeflow-jobs-triggers-06af6feae80c?source=rss----8dd843bb8b39---4"><img src="https://cdn-images-1.medium.com/max/1974/1*EsjoUrXSj34STPQSJ0aQDg.png" width="1974"></a></p><p class="medium-feed-snippet">Not so long ago, a schedule was the only starting point for Databricks orchestration. Run ingestion at midnight, wait an hour, and then&#x2026;</p><p class="medium-feed-link"><a href="https://medium.com/databrickscommunity/lakeflow-jobs-triggers-06af6feae80c?source=rss----8dd843bb8b39---4">Continue reading on Databricks Community Articles »</a></p></div>]]></description>
            <link>https://medium.com/databrickscommunity/lakeflow-jobs-triggers-06af6feae80c?source=rss----8dd843bb8b39---4</link>
            <guid isPermaLink="false">https://medium.com/p/06af6feae80c</guid>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[etl]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[Hubert Dudek Databricks MVP]]></dc:creator>
            <pubDate>Fri, 13 Nov 2026 16:36:01 GMT</pubDate>
            <atom:updated>2026-11-13T18:04:40.166Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[How Spark Processes Datasets Larger Than Cluster Memory: The Kitchen Architecture]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/databrickscommunity/how-spark-processes-datasets-larger-than-cluster-memory-the-kitchen-architecture-9db71669e8ce?source=rss----8dd843bb8b39---4"><img src="https://cdn-images-1.medium.com/max/1920/1*IW-yVgUKoFoYkmOZy-y7Kw.png" width="1920"></a></p><p class="medium-feed-snippet">When I first started out, I always wondered: how exactly does Apache Spark process terabytes of data on very modest hardware?</p><p class="medium-feed-link"><a href="https://medium.com/databrickscommunity/how-spark-processes-datasets-larger-than-cluster-memory-the-kitchen-architecture-9db71669e8ce?source=rss----8dd843bb8b39---4">Continue reading on Databricks Community Articles »</a></p></div>]]></description>
            <link>https://medium.com/databrickscommunity/how-spark-processes-datasets-larger-than-cluster-memory-the-kitchen-architecture-9db71669e8ce?source=rss----8dd843bb8b39---4</link>
            <guid isPermaLink="false">https://medium.com/p/9db71669e8ce</guid>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[pyspark]]></category>
            <category><![CDATA[apache-spark]]></category>
            <dc:creator><![CDATA[Likith Raj Gollu]]></dc:creator>
            <pubDate>Tue, 03 Nov 2026 23:06:36 GMT</pubDate>
            <atom:updated>2026-11-03T23:06:34.990Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Databricks Real-time mode(RTM) in Spark Declarative Pipelines]]></title>
            <link>https://medium.com/databrickscommunity/databricks-real-time-mode-rtm-in-spark-declarative-pipelines-c5f661412c2b?source=rss----8dd843bb8b39---4</link>
            <guid isPermaLink="false">https://medium.com/p/c5f661412c2b</guid>
            <category><![CDATA[apache-flink]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[apache-spark]]></category>
            <dc:creator><![CDATA[G e o r g i a n]]></dc:creator>
            <pubDate>Fri, 30 Oct 2026 16:07:38 GMT</pubDate>
            <atom:updated>2026-10-30T16:14:25.242Z</atom:updated>
            <content:encoded><![CDATA[<h3>Databricks Real-time mode (RTM) in Spark Declarative Pipelines</h3><h4>What went GA and how it fits to the platform</h4><h3>Introduction</h3><p>A card authorization usually has a few hundred milliseconds to come back with a fraud decision. The model making that decision needs features such as “total amount this card spent in the last 10 minutes,” and that number has to include the transaction that happened two seconds ago. A feature table refreshed every minute is already stale for this purpose.</p><p>Until now we solved this in two parts. Spark ran the ETL, the lakehouse tables and the training data. <strong>Apache Flink</strong> or a hand-written consumer service ran the operational path from <strong>Kafka to an online store</strong>. That worked but with great extra effort:</p><ul><li>The same business logic implemented twice, often in different languages.</li><li>Different deployment and operational models for each path.</li><li>Possible inconsistencies between the offline features used for training and the online features used for inference.</li><li>Separate monitoring, failure recovery and governance for each environment.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ccab1uES9fOhIwRk0zO5eQ.png" /><figcaption>Offline vs Online feature computation</figcaption></figure><p>On October 29, 2026, Databricks announced GA of <strong>real-time mode (RTM)</strong> <strong>in Spark Declarative Pipelines (SDP)</strong>. RTM itself is not new: it shipped in Spark Structured Streaming earlier. What is new is that it now runs inside the declarative pipeline framework, with SDP’s orchestration, failure recovery and serverless compute.</p><h3>What real-time mode changes inside the engine</h3><p>The <a href="https://docs.databricks.com/aws/en/ldp/real-time">Databricks documentation</a> describes three changes in RTM:</p><ol><li><strong>Long-running batches.</strong> A batch stays open for a long time (five minutes by default), and records are processed as they arrive in the source instead of being collected first.</li><li><strong>Simultaneous stage scheduling.</strong> All stages of the query are scheduled at the same time. A stateful query with a source stage and a shuffle stage has both sets of tasks running at once.</li><li><strong>Streaming shuffle.</strong> Data moves from one stage to the next as soon as it is produced, instead of waiting for the upstream stage to finish.</li></ol><p>The result is that <strong>a record entering Kafka flows through the whole query without waiting for a batch boundary</strong>. Databricks quotes end-to-end latency as low as five milliseconds.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*40vBBvH8BlH5Or4TdPlp7w.png" /><figcaption>Example of use with RTM</figcaption></figure><h3>The problem it solves: one engine for the operational path</h3><p>The case for RTM is less about raw speed and more about removing the second engine. When fraud features are computed in Flink and training features in Spark the same aggregation exists twice in two languages and two runtimes. Sooner or later the two versions drift and the model sees different numbers in production than it saw in training.</p><p>RTM lets the low-latency path use the same DataFrame API, the same Unity Catalog permissions, the same pipeline event log and the same deployment process as the rest of the data platform. In SDP, it also inherits dependency handling, automatic failure recovery and serverless compute.</p><blockquote>On serverless, you don’t choose instance types and Databricks says you pay for the cores the query actually uses.</blockquote><blockquote>The announcement names fraud detection, live personalization, real-time recommendations and real-time feature computation as target workloads.</blockquote><p>Databricks also publishes performance numbers, and they should be read as vendor claims:</p><ul><li>RTM is described as up to 100 times faster than the micro-batch architecture.</li><li>On customer feature-computation workloads that Databricks selected, RTM is described as up to 92% faster than Apache Flink.</li><li>The Databricks Feature Store, which uses this architecture, reports a p99 latency of 200 ms from Kafka to the online feature store.</li></ul><h3>Where RTM fits in the Databricks platform</h3><p><strong>RTM flows run beside the medallion architecture, not inside it. </strong>This is the most important thing to understand before designing with it and the announcement does not say it.</p><p>The supported sources and sinks table in the docs lists Delta as unsupported, both as a source and as a sink. An RTM flow reads from <strong>Apache Kafka, Azure Event Hubs through its Kafka interface, Amazon MSK or Amazon Kinesis</strong>.</p><p>It writes to <strong>Kafka, Event Hubs or MSK or to a ForEach sink that runs your own Python code for each row.</strong> Kinesis works only as a source.</p><blockquote>So an RTM flow cannot read a silver table and cannot write a gold one.</blockquote><p>It sits on the event bus and writes to operational systems: another topic, a key-value store, a Postgres database such as Lakebase or an API.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z-i2T0tLbRoaVTiJ6N7mXA.png" /><figcaption>RTM beside medallion path</figcaption></figure><p>The components around RTM complete the picture. Lakebase, the managed Postgres service, is the natural target for features or decisions that applications query by key. The Feature Store online tables are built on this pattern. Model Serving endpoints read those features at inference time. Unity Catalog governs the lookup tables an RTM flow joins against and the connections it uses for Kinesis.</p><h3>Building with it: card transactions to a fraud feature store</h3><p>(1) The payment service publishes each transaction to the event bus.</p><p>(2) The RTM flow reads it and enriches it with the merchant lookup table.</p><p>(3) The flow upserts that card’s features into Lakebase.</p><p>(4) When the next authorization arrives, the payment service sends a scoring request to Model Serving.</p><p>(5) The fraud model reads the card’s features from Lakebase by key.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v7Ny-Y3f9Qix7USdv9Q9jA.png" /><figcaption>RTM keeps the online feature table current for the fraud model</figcaption></figure><h4>Configure the pipeline</h4><p>RTM needs two settings at the pipeline level: continuous mode and the RTM Spark flag.</p><pre>{<br>  &quot;continuous&quot;: true,<br>  &quot;spark_conf&quot;: {<br>    &quot;spark.databricks.streaming.realTimeMode.enabled&quot;: &quot;true&quot;<br>  }<br>}</pre><h4>Example 1: the smallest real-time flow</h4><p>An RTM flow must be an update flow, its trigger must be RealTime, and its target must be a sink.</p><p>This flow reads the raw topic and republishes it, which is enough to verify connectivity and measure baseline latency.</p><pre>from pyspark import pipelines as dp<br><br>dp.create_sink(&quot;txn_passthrough_sink&quot;, &quot;kafka&quot;, {<br>    &quot;kafka.bootstrap.servers&quot;: broker_address,<br>    &quot;topic&quot;: output_topic,<br>})<br><br>@dp.update_flow(<br>    name=&quot;txn_passthrough&quot;,<br>    target=&quot;txn_passthrough_sink&quot;,<br>    spark_conf={<br>        &quot;pipelines.trigger&quot;: &quot;RealTime&quot;,<br>        &quot;pipelines.trigger.interval&quot;: &quot;5 minutes&quot;,  # checkpoint interval, optional<br>    },<br>)<br>def txn_passthrough():<br>    return (<br>        spark.readStream<br>            .format(&quot;kafka&quot;)<br>            .option(&quot;kafka.bootstrap.servers&quot;, broker_address)<br>            .option(&quot;subscribe&quot;, input_topic)<br>            .option(&quot;startingOffsets&quot;, &quot;latest&quot;)<br>            .load()<br>            .selectExpr(&quot;CAST(key AS STRING)&quot;, &quot;CAST(value AS STRING)&quot;, &quot;timestamp&quot;)<br>    )</pre><h4>Example 2: write features to Lakebase with a ForEach sink</h4><p>A Kafka topic is not something a fraud model can query by card ID. The features need to land in a store that supports point lookups. Inside SDP, the documented way to do this today is a ForEach sink.</p><p>Databricks does have a native Lakebase sink with built-in batching, retries and managed authentication, but its documentation lists it for Structured Streaming on classic compute (<strong>Databricks Runtime 18.3 and above</strong>) and it states that serverless and SDP are not supported.</p><blockquote>The announcement lists a native Lakebase sink for SDP as upcoming work. Until it ships, the writer is yours to build.</blockquote><p>Here is one example with upserts into a Postgres table with psycopg.</p><pre>mport json<br>from pyspark import pipelines as dp<br><br>class CardFeatureWriter:<br>    def __init__(self, conninfo: str):<br>        # Plain strings are serialized to the executors; never pass a live connection.<br>        self.conninfo = conninfo<br><br>    def open(self, partition_id, epoch_id):<br>        import psycopg  # import on the executor<br>        self.conn = psycopg.connect(self.conninfo, autocommit=True, connect_timeout=5)<br>        return True<br><br>    def process(self, row):<br>        f = json.loads(row.value)<br>        self.conn.execute(<br>            &quot;&quot;&quot;<br>            INSERT INTO fraud.card_features (card_id, window_end, spend_10m, txn_count_10m)<br>            VALUES (%s, %s, %s, %s)<br>            ON CONFLICT (card_id) DO UPDATE<br>              SET window_end = EXCLUDED.window_end,<br>                  spend_10m = EXCLUDED.spend_10m,<br>                  txn_count_10m = EXCLUDED.txn_count_10m<br>              WHERE EXCLUDED.window_end &gt;= fraud.card_features.window_end<br>            &quot;&quot;&quot;,<br>            (f[&quot;card_id&quot;], f[&quot;window_end&quot;], f[&quot;spend_10m&quot;], f[&quot;txn_count_10m&quot;]),<br>        )<br><br>    def close(self, error):<br>        if getattr(self, &quot;conn&quot;, None) is not None:<br>            self.conn.close()<br><br>conninfo = dbutils.secrets.get(&quot;fraud&quot;, &quot;lakebase_conninfo&quot;)  # resolved on the driver<br>dp.foreach_sink(name=&quot;card_features_sink&quot;, writer=CardFeatureWriter(conninfo))</pre><h3>What to check before production</h3><p>The operator list has gaps that matter for fraud and personalization logic. According to the <a href="https://docs.databricks.com/aws/en/ldp/real-time">operator table</a> and the <a href="https://docs.databricks.com/aws/en/structured-streaming/real-time/limitations">limitations page</a>:</p><ul><li>Stream-to-stream joins and session windows are not supported.</li><li>dropDuplicates works but keeps unbounded state; dropDuplicatesWithinWatermark is not supported.</li><li>foreachBatch, mapPartitions, mapInPandas, mapInArrow and flatMapGroupsWithState are not supported or not recommended.</li><li>transformWithState works with differences: handleInputRows is called once per row, event-time timers are not supported, and transformWithStateInPandas is not supported.</li><li>union cannot follow a stateful operator and cannot read the same source DataFrame twice.</li></ul><h3>My honest view: adopt it for features, keep Flink for event logic</h3><p>My recommendation for teams on Databricks today: use RTM for single-flow feature pipelines and alerting where the source is already Kafka, Event Hubs, MSK or Kinesis and the target is a topic or a key-value store. Keep complex event processing where it runs now. Wait for multi-flow support to reach GA before consolidating many operational streams into one pipeline.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c5f661412c2b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/databrickscommunity/databricks-real-time-mode-rtm-in-spark-declarative-pipelines-c5f661412c2b">Databricks Real-time mode(RTM) in Spark Declarative Pipelines</a> was originally published in <a href="https://medium.com/databrickscommunity">Databricks Community Articles</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Naming Your Data Sources in Databricks Unity Catalog: A Convention Tested on a Real Multi-Source…]]></title>
            <link>https://medium.com/databrickscommunity/naming-your-data-sources-in-databricks-unity-catalog-a-convention-tested-on-a-real-multi-source-432a661f406e?source=rss----8dd843bb8b39---4</link>
            <guid isPermaLink="false">https://medium.com/p/432a661f406e</guid>
            <category><![CDATA[jdbc-connection]]></category>
            <category><![CDATA[data-governance]]></category>
            <category><![CDATA[data-virtualization]]></category>
            <category><![CDATA[federation]]></category>
            <category><![CDATA[databricks-unity-catalog]]></category>
            <dc:creator><![CDATA[Ennio Noventa]]></dc:creator>
            <pubDate>Mon, 26 Oct 2026 19:08:42 GMT</pubDate>
            <atom:updated>2026-10-26T19:08:41.207Z</atom:updated>
            <content:encoded><![CDATA[<h3>Naming Your Data Sources in Databricks Unity Catalog: A Convention Tested on a Real Multi-Source Lab</h3><h3>Connections, foreign catalogs, JDBC sources, secrets and bundles: what I learned federating PostgreSQL, Oracle, MariaDB, YugabyteDB, Snowflake and Denodo into one metastore</h3><figure><img alt="medium-cover.png" src="https://cdn-images-1.medium.com/max/1024/6edf938cb559ae60a7a922b9085973d9b8cb0a6b299a7dee4571d690650b5fdb" /><figcaption>Physical connections, logical catalogs: every name derived from one source ID.</figcaption></figure><p>It started with a simple question I couldn&#39;t answer: <em>which foreign catalog uses which connection?</em></p><p>I had created a handful of connections and catalogs in Unity Catalog, and a few days later I no longer remembered how they were linked. A Python snippet gave me the answer in seconds, but the real problem was the names. They said nothing.</p><p>Databricks doesn&#39;t impose a naming standard. If you don&#39;t set one, you end up with a metastore where postgres, pgsql and pg all mean the same thing, and where renaming a server breaks every downstream query.</p><p>So I built a convention, and then I stress-tested it on a lab with very different sources: public demo databases, an Oracle Free instance, a MariaDB server without TLS, a YugabyteDB cloud cluster requiring certificates and IP allow-lists, a vendor-managed Snowflake account, and two Denodo instances. Every rule below exists because something in that lab forced it.</p><p>The full convention, a deployable bundle, and the test queries are on GitHub: <strong>unity-catalog-federation-blueprint</strong> <em>(link at the end)</em>.</p><h3>Rule zero: logical names, not physical names</h3><p>Server names, hostnames, cloud providers, and database products all change during a system&#39;s life. The business system they serve does not.</p><p>A catalog called fed_mssql_srvdb01_billing_prod looks precise until the billing database migrates to Azure SQL. Then you have two bad options: rename it and break every query, dashboard, and grant, or keep a name that now lies.</p><p>The fix is to separate two objects that are easy to confuse:</p><ul><li><strong>The connection is physical.</strong> It stores <em>how</em> to connect: technology, host, credentials. Only admins see it, so it may carry technical details.</li><li><strong>The foreign catalog is logical.</strong> It is <em>what</em> users query, and its name appears in every query and lineage graph. It should speak the language of the business.</li></ul><h3>The source ID: one identifier to derive everything</h3><p>The core of the convention is the <strong>source ID</strong>, a short, stable, logical identifier for each data source, assigned and recorded centrally.</p><p>For a SaaS application, it is the product name: salesforce, servicenow. For an application database, it is the business system: billing, crm. For a sandbox it is the lab name: oralab, yugalab. For a dataset replicated on several technologies, it is the dataset: pkdd_financial.</p><p>What to avoid: physical database names that recur everywhere (main, appdb, postgres), names that will clash with future systems (financial), and anything that describes <em>where</em> the data lives rather than <em>what</em> it is.</p><p>Every other name is derived from the source ID:</p><pre>Connection        conn_&lt;tech&gt;_&lt;source_id&gt;[_&lt;qualifier&gt;]_&lt;env&gt;<br>Foreign catalog   fed_&lt;source_id&gt;[_&lt;variant&gt;]_&lt;env&gt;<br>JDBC view catalog jdbc_&lt;source_id&gt;[_&lt;variant&gt;]_&lt;env&gt;.&lt;schema&gt;.&lt;table&gt;<br>Secret            _platform_&lt;env&gt;.secrets.&lt;connection without env&gt;_&lt;key&gt;</pre><p>So conn_pg_billing_prod feeds fed_billing_prod, and the link between them is obvious just from reading the names.</p><h3>Why the environment suffix, if dev and prod are separate workspaces?</h3><p>This was my first objection. Databricks recommends separate workspaces per environment, so why put dev or prod in names at all?</p><p>Because <strong>workspaces are separate, but the metastore is not</strong>. All workspaces in the same region attach to one Unity Catalog metastore, so catalog and connection names must be unique across environments. Workspace-catalog binding controls <em>visibility</em>, not <em>naming</em>.</p><p>The rule follows from object scope:</p><ul><li><strong>Metastore-level objects</strong> (catalogs, connections, external locations, storage credentials) get the _&lt;env&gt; suffix.</li><li><strong>Objects inside a catalog</strong> (schemas, tables, views, volumes, secrets) don&#39;t: the catalog already carries it.</li><li><strong>Workspace objects</strong> (jobs, pipelines, clusters, warehouses) don&#39;t either: the workspace already carries it.</li></ul><p>The environment always goes <strong>at the end</strong>. Listings then group by object type and system, with dev and prod side by side: fed_billing_dev, fed_billing_prod.</p><h3>One connection per catalog</h3><p>My first draft used one connection per server. The lab broke it immediately: one server hosted an Oracle instance and several PostgreSQL databases, each a different source. What source ID goes in the name of a connection serving five sources?</p><p>The answer is to stop sharing: <strong>one connection per foreign catalog, always</strong>, even when two connections point to the same server with the same credentials. It costs a few extra objects, and in exchange:</p><ul><li>the connection name always derives from the source ID, with no exceptions;</li><li>a database can move to another server without touching its neighbours;</li><li>the original question (&quot;which catalog uses which connection?&quot;) answers itself.</li></ul><p>Two refinements came later. When the source ID already starts with the technology, as with SaaS products or a Denodo instance, write the technology once: conn_denodo_29090_dev, not conn_denodo_denodo_29090_dev. And use a closed, documented list of technology codes (pg, ora, mssql, mariadb, yb, snowflake, denodo, sfdc…), or you&#39;ll soon have postgres, pgsql and pg side by side.</p><h3>When the technology is the point</h3><p>&quot;No technology in catalog names&quot; has one legitimate exception: when the technology <em>is</em> the subject.</p><p>In the lab I loaded the same banking dataset (PKDD&#39;99 financial) on MariaDB and PostgreSQL to compare behaviour. Querying &quot;the dataset&quot; is meaningless here: you need to know which copy you&#39;re hitting. The convention adds a <strong>variant qualifier</strong> to every copy, the first one included:</p><pre>fed_pkdd_financial_pg_dev<br>fed_pkdd_financial_mariadb_dev</pre><p>The connection doesn&#39;t need it, because its technology code already makes it unique.</p><h3>When federation isn&#39;t possible: JDBC view catalogs</h3><p>Lakehouse Federation doesn&#39;t cover everything. Denodo has no federation connector. And one of my MariaDB servers had no TLS, which federation requires for MySQL. I checked the server handshake to be sure: the SSL capability flag simply wasn&#39;t there.</p><p>The workaround is a <strong>JDBC connection</strong>, a Unity Catalog connection that loads a driver JAR from a volume. JDBC connections don&#39;t create foreign catalogs, though. You query them through remote_query(), in the source&#39;s own SQL dialect.</p><p>To give these sources the same shape as a foreign catalog, I generate a <strong>JDBC view catalog</strong>: one view per source table, each a thin SELECT * over remote_query(), the same idea as Denodo base views.</p><pre>jdbc_ctulab_dev.ctu_financial.loan</pre><p>The jdbc_ prefix describes the access method, just like fed_. If the source becomes federable one day, consumers only change the prefix. Users need SELECT on the views only, because the connection privilege is checked on the view owner.</p><h3>A platform catalog for the plumbing</h3><p>Secrets, JDBC drivers and certificates need a home. The obvious candidate, the default workspace catalog, is the wrong one: it is usually bound to a single workspace, often grants broad permissions to all users, and carries no environment.</p><p>Instead, a dedicated catalog per environment:</p><pre>_platform_dev<br> ├─ secrets  (schema)          Unity Catalog secrets, no grants<br> ├─ drivers  (schema)<br> │   └─ jdbc (volume)          JDBC driver JARs, readable by connection users<br> └─ certs    (schema)<br>     └─ ca   (volume)          server/CA certificates, admins only</pre><p>The leading underscore marks it as infrastructure, and in most tools it sorts it above the data catalogs. The separate schemas exist for one reason: different grants. Everyone querying a JDBC connection needs to read the drivers, and nobody should read the secrets.</p><p>Secrets follow the newer <strong>Unity Catalog secrets</strong> model (catalog.schema.secret) instead of workspace secret scopes, and are named by intended use: conn_pg_ecom_user, conn_pg_ecom_password. I create connections with the Python SDK, reading values from secrets, so credentials never appear in a SQL statement or in the query history.</p><h3>Deploying it with bundles</h3><p>Databricks recommends Declarative Automation Bundles (formerly Asset Bundles) for multi-environment deployment. They fit the convention through one rule:</p><blockquote><strong>Bundle target names are the environment codes.</strong></blockquote><p>Then ${bundle.target} <em>is</em> the _&lt;env&gt; suffix, and no environment-specific name is ever written by hand:</p><pre>variables:<br>  billing_catalog:<br>    default: fed_billing_${bundle.target}</pre><p>Connections are not a bundle resource type, so a <strong>platform bundle</strong> deploys an idempotent setup job. It reads a versioned <strong>source registry</strong> file (config/sources_&lt;env&gt;.yml) and creates the platform catalog, connections, foreign catalogs, JDBC connections and view catalogs. The run fails if anything is missing, so a broken source surfaces in the deployment pipeline rather than in a user&#39;s query. <strong>Project bundles</strong> then consume the sources through variables.</p><p>One deliberate exception: development mode prefixes resources with the developer&#39;s name (dev_fred_…). That breaks the &quot;environment at the end&quot; rule, and it should, because a leading dev_&lt;user&gt;_ marks a personal sandbox while a trailing _dev marks the shared environment. The platform bundle never runs in development mode.</p><h3>Lessons learned the hard way</h3><p><strong>Pushdown depends on the Spark dialect, not on federation vs JDBC.</strong> I ran the same aggregate query three ways. Through JDBC to MariaDB, the filter was pushed, but the GROUP BY ran in Databricks. Through JDBC to PostgreSQL <em>and</em> through federation to PostgreSQL, the plans were identical: filter, aggregates, and grouping all pushed to the source. Both paths use the same Spark JDBC engine, and Spark picks the dialect from the URL. A jdbc:mariadb: URL falls back to the generic dialect, which doesn&#39;t push aggregates.</p><p><strong>MariaDB via the generic dialect needs </strong>ANSI_QUOTES<strong>.</strong> Spark quotes column names with double quotes, and MariaDB reads them as string literals. The symptom is baffling: value &#39;ok&#39; cannot be decoded as Integer. Adding sessionVariables=sql_mode=ANSI_QUOTES to the JDBC URL fixes it.</p><p>&#39;&#39;<strong> is not an escape in Spark SQL.</strong> Adjacent string literals are concatenated, so a doubled quote silently vanishes. When you build remote_query() strings in code, escape with a backslash.</p><p><strong>Server certificates go in as text.</strong> userProvidedServerCertificate expects PEM content, not a path, and must be paired with trustServerCertificate &#39;false&#39;.</p><p><strong>Serverless egress IPs are published, and they change.</strong> SELECT current_metastore() returns cloud and region. Filter Databricks&#39; published IP ranges with those values to build the allow-list, and re-check it periodically.</p><p><strong>Vendor-managed accounts can close the door.</strong> A Snowflake account managed by an ELT platform suddenly rejected every client IP through a network policy I couldn&#39;t change. Prefer pushing data from such platforms to a destination you control.</p><p><strong>Semantic layers don&#39;t travel over JDBC, but their queries do.</strong> Denodo 9.5 introduced metric views, similar in spirit to Databricks metric views. Databricks can&#39;t recognise them as metric views, because no semantic metadata crosses a JDBC driver. But remote_query() passes the text to Denodo unchanged, so this works, with Denodo computing the metric:</p><pre>SELECT * FROM remote_query(&#39;conn_denodo_29090_dev&#39;, query =&gt;<br>  &#39;SELECT trans_year, evaluate_metric(transaction_count) AS n<br>     FROM mv_yuga_fin_account_activity<br>    GROUP BY trans_year&#39;);</pre><p>I validated it by comparing those yearly counts with the same counts computed by Databricks on a PostgreSQL copy of the data. The difference was zero for every year.</p><h3>Takeaways</h3><ul><li>Name sources by <strong>what they are</strong>, never by where they live.</li><li><strong>One connection per catalog</strong>: a few extra objects buy you obvious lineage and painless migrations.</li><li>Put the environment <strong>only where the metastore is shared</strong>, and always at the end.</li><li>Treat the <strong>source registry as code</strong>, and let bundles derive every name from it.</li><li>When federation can&#39;t reach a source, <strong>JDBC view catalogs</strong> keep the same shape for consumers.</li><li>Before blaming federation for slow queries, <strong>check the dialect</strong> in the plan.</li></ul><p>The complete convention (with edge cases, lifecycle rules, and the full lab), the platform bundle, and the test queries are open source:</p><p><strong>github.com/Capppo/unity-catalog-federation-blueprint</strong></p><p>Contributions are welcome, especially new technology codes, table-listing queries for other JDBC sources, and real-world cases that break the rules.</p><p><em>This is an independent community project, not affiliated with Databricks or Denodo.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=432a661f406e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/databrickscommunity/naming-your-data-sources-in-databricks-unity-catalog-a-convention-tested-on-a-real-multi-source-432a661f406e">Naming Your Data Sources in Databricks Unity Catalog: A Convention Tested on a Real Multi-Source…</a> was originally published in <a href="https://medium.com/databrickscommunity">Databricks Community Articles</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Databricks News: Genie CLI, App Builder, Agent Memory, DABs [Last updated 16 November 2026]]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/databrickscommunity/databricks-news-genie-cli-app-builder-agent-memory-dabs-41820427fcc2?source=rss----8dd843bb8b39---4"><img src="https://cdn-images-1.medium.com/max/2600/1*0TvAmh9EY44ojFqraq_3mQ.png" width="2948"></a></p><p class="medium-feed-snippet">More news is coming.</p><p class="medium-feed-link"><a href="https://medium.com/databrickscommunity/databricks-news-genie-cli-app-builder-agent-memory-dabs-41820427fcc2?source=rss----8dd843bb8b39---4">Continue reading on Databricks Community Articles »</a></p></div>]]></description>
            <link>https://medium.com/databrickscommunity/databricks-news-genie-cli-app-builder-agent-memory-dabs-41820427fcc2?source=rss----8dd843bb8b39---4</link>
            <guid isPermaLink="false">https://medium.com/p/41820427fcc2</guid>
            <category><![CDATA[genie]]></category>
            <category><![CDATA[agent-memory]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[dab]]></category>
            <category><![CDATA[databricks-news]]></category>
            <dc:creator><![CDATA[Hubert Dudek Databricks MVP]]></dc:creator>
            <pubDate>Wed, 21 Oct 2026 18:21:58 GMT</pubDate>
            <atom:updated>2026-11-16T14:20:15.049Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Build Your Agentic AI App in Databricks: A Step-by-Step Guide]]></title>
            <link>https://medium.com/databrickscommunity/build-your-agentic-ai-app-in-databricks-a-step-by-step-guide-44d06abe00bf?source=rss----8dd843bb8b39---4</link>
            <guid isPermaLink="false">https://medium.com/p/44d06abe00bf</guid>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Akash Singh]]></dc:creator>
            <pubDate>Tue, 20 Oct 2026 12:02:37 GMT</pubDate>
            <atom:updated>2026-10-21T06:08:21.726Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jdtz-hPglMfgKe3eg0ceDg.png" /></figure><h3>The problem: a chatbot that doesn’t know your business</h3><p>Imagine you call a company’s support line and ask:</p><p><em>“How do I fix file sync problems in CloudVault Pro?”</em> then <em>“What’s your return policy?”</em> and finally <em>“How many returns has customer alice@example.com made this year?”</em></p><p>These three questions need three <em>different</em> places to look:</p><ul><li>Question 1 lives inside <strong>product manuals</strong> (PDF files).</li><li>Question 2 lives in a <strong>policy table</strong> (structured data).</li><li>Question 3 lives in a <strong>customer history table</strong> (structured data).</li></ul><p>A normal chatbot (ChatGPT-style) knows the internet, but it knows <strong>nothing about your PDFs, your policies, or your customers</strong>. If you ask, it may confidently make something up. That is called a hallucination, and in customer support it is dangerous.</p><p>In this article we build an <strong>AI agent</strong> that:</p><ol><li>Reads the question.</li><li>Decides where to look (manuals, policy table, or customer table).</li><li>Fetches the real facts.</li><li>Writes a clear answer based <strong>only</strong> on those facts.</li></ol><h3>Key ideas</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ToFgxH3UhcO5JkYGMAS1fA.png" /></figure><h4>AI agent vs. agentic AI</h4><ul><li>An <strong>AI agent</strong> is one worker: an LLM + tools + (optionally) memory. It breaks a task into steps and carries them out.</li><li><strong>Agentic AI</strong> is a team of such agents, often with an “orchestrator” manager deciding who works next.</li></ul><h4>RAG : Retrieve, Augment, Generate</h4><ol><li><strong>Retrieve</strong> — find the most relevant passages for the question.</li><li><strong>Augment</strong> — attach those passages to the question.</li><li><strong>Generate</strong> — the LLM writes an answer using that evidence.</li></ol><p><strong>Why not paste all PDFs into the prompt?</strong> LLMs have a context window (a maximum number of tokens they can read at once). With ~500 PDFs you would blow past it, and you would pay for every token on every question. Retrieval sends only the few relevant passages.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VsTrnPmVeltcW2-7nIu_ug.png" /></figure><h3>What we will build</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lTaA7qGfDPCYuAXEKLRlbw.png" /></figure><h3>Step 0 — Workspace</h3><p>We start by signing up for a free workspace. Think of it as an online office where all the work happens. We only need to know four rooms: Catalog (where data is stored), Compute (where things run), Playground (where we test the AI) and Serving (where the finished agent lives).</p><ol><li>You land on the workspace home page.</li><li>In the left sidebar, find the <strong>AI/ML</strong> section (Playground, Serving) and <strong>Catalog</strong>, <strong>Compute</strong>, <strong>Workspace</strong>.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f0sTjaoPdMiAKjGLicbMIA.png" /></figure><h3>Step 1 — Talk to an LLM</h3><p>Before building anything, we check that the AI model answers a question. The Playground needs no code. The Python and SQL versions show the same thing for people who want to do it in a notebook.</p><h4><strong>In the Playground</strong></h4><p>Open <strong>AI Playground</strong>, choose the model, and ask:</p><blockquote><em>Why are AI agents getting popular?</em></blockquote><p><strong>Normal</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZgB3_ENWBYYJwihv7nnxeg.png" /></figure><p>Then add a <strong>system prompt</strong>, such as Use a funny language, and ask again. The tone changes completely.</p><p><strong>By using Funny Language</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*767kLLI6Y_yIpm4yu8my0g.png" /></figure><h4>In Python</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HpdPcYvQV84C5niZpJcYdQ.png" /></figure><p><strong>Under the hood:</strong></p><ul><li>chat.completions.create is the standard API shape popularised by OpenAI and now used by many providers, which is why this one client works with many Databricks-hosted models.</li><li>max_tokens caps the reply length.</li><li>Reasoning models may return content as a <strong>list</strong> of blocks: one with type: reasoning (the model’s internal thoughts) and one with type: text (the real answer).</li></ul><h4>In SQL (the easiest way)</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pWL9lzoyzM-pJUF4ZVH-7w.png" /></figure><p>ai_query is a built-in SQL function: first argument = model, second = prompt. For a data engineer this is powerful, because you can call an LLM on every row of a table.</p><h3>Step 2 — Create the catalog, schema and volume</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iXaiuJM6ZM2-bmB1dXw7TQ.png" /></figure><p>Upload your files in the Volume</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eiowOTrQEE1WqRdeIHd7CQ.png" /></figure><h3>Step 3 — Build the unstructured knowledge base (PDFs)</h3><p>The AI can’t read PDF files directly, so we pull the text out of each PDF and save it as rows in a table. Now the manuals are data the system can search.</p><p><strong>Extract text from every PDF</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PF-4-b0E4iDKGJbvOWFMTA.png" /></figure><p><strong>Save as a Delta table</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tnRgRgwEqhtDS8282z_9-w.png" /></figure><p>We have created a product_document Table</p><h4>What about long documents?</h4><p>Each PDF was short enough to sit in one row. Real manuals are often 50+ pages. Then you <strong>chunk</strong>: split each document into passages of roughly 500–1,000 characters with a small overlap, one row per chunk. There is no universal best size; test retrieval quality and adjust.</p><h3>Step 4 — Build the structured knowledge base</h3><p>Some facts, such as policies and customer history, are already neat rows and columns. We save these as three small tables: products, policies and customer history.</p><p>We have a Sample Tables of Products, Policies, Customer_service_history</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i9KisGiRu_KJsboez_mbWQ.png" /></figure><h3>Step 5 — Enrich the documents</h3><p>Before searching, we stamp each manual’s product name and category on top of its text. It’s like writing the department name on the front of every file folder, so a search for “storage software” can find the right one.</p><p>The search index will look at <strong>one text column</strong>. So we pack the most useful context into that single column: category, subcategory, product name, then the document itself.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zOm_OnFH581Ly_wtMrGAuQ.png" /></figure><h3>Step 6 — Create the vector search (AI Search) index</h3><p>Here we build a search that works by meaning, not exact words. Each manual becomes a list of numbers (an embedding), and the question becomes one too. The closest matches win. We test this search on its own first, so we know it works before the AI is involved.</p><h4>What is an embedding?</h4><p>Computers can’t “understand” sentences, but they are excellent with numbers. An <strong>embedding model</strong> converts a sentence into a long list of numbers that captures its <em>meaning</em>. Sentences with similar meaning get similar numbers.</p><ul><li>“Press Sync Now to refresh your files”</li><li>“Use the refresh option if your files look out of date”</li></ul><p>Different words, same meaning, so their number lists sit close together. A keyword search would miss the match; a meaning-based search finds it.</p><h4>Create the endpoint</h4><p>An endpoint is the always-ready address where our search lives. Free Edition allows only one, so create it once and reuse it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q_pk1aoUAlSAXqrrgIvuEA.png" /></figure><h4>Create the index</h4><p><strong>Click on AI Search Index</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eoKJqnsyYjzWCUQWPT_DvQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kTtsI04UMwGfz2B8QKE8OA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pwA-MjVA51GHRTIBSrf4dg.png" /></figure><h4>Test the index before involving the LLM</h4><p>This is the most underrated debugging habit: <strong>always test retrieval alone first.</strong> If retrieval is bad, the LLM can’t save you.</p><p>In the index page choose <strong>Query index</strong>. You should see the CloudVault Pro document ranked first, with a score.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*en0GROJNvj9qZXsDrDfrfA.png" /></figure><p><em>this is the “R” in RAG working on its own. It proves retrieval is correct before the AI is added.</em></p><h3>Step 7 — Create the structured tools (Unity Catalog functions)</h3><p>Language models can write SQL, but generated queries can be wrong, and running them freely would require broad access to your tables. For customer data, that is a risk we can avoid. So we don’t give the agent a free-form query tool.</p><p>Instead we give it <strong>two fixed lookups</strong>: one that returns a named policy and one that returns a customer’s support history. The agent decides when to call them and what value to pass in, but the query itself is written by us and tested in advance, and Unity Catalog controls who may run it.</p><p><strong>Tool 1: policy lookup</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ei9xammRS7zpMzCDMVjozg.png" /></figure><p><strong>Tool 2: customer service history</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nEuB2-z1JaBS-1UozBSBog.png" /></figure><p><strong>Why functions instead of free-form SQL?</strong></p><ol><li><strong>The agent can only use what we built</strong> <em>(controlled access).</em> Think of a vending machine: you can pick only the items it lists. The agent can run our two lookups, but it cannot browse other tables or run a query nobody approved.</li><li><strong>The same question gets the same answer</strong> <em>(predictable results).</em> The rule for counting a customer’s returns is written once, by us. Every time the agent calls it, that same rule runs, so two people asking the same thing see the same number. A model that writes fresh SQL each time might count slightly differently.</li><li><strong>Someone decides who may use it</strong> <em>(governance).</em> Unity Catalog is the permission system in Databricks. An administrator can choose who is allowed to run each function, and the function is listed in Catalog Explorer like any table, so it can be reviewed.</li><li><strong>Each lookup has a label the agent reads</strong> <em>(clear contract).</em> The agent chooses a tool by reading its COMMENT, a short description we write. It works like the label on a button, for example “Returns the official details of a named support policy.” Write these labels as if briefing a new colleague who has never seen your data; clearer labels lead to better choices.</li></ol><h3>Step 8 — Assemble the agent in AI Playground</h3><p>Now we connect the AI to its tools: the manual search and our two lookups. We don’t tell it which one to use. It reads the question and decides. Try a question that needs two tools and watch it use both.</p><ol><li>Open <strong>AI Playground</strong> and pick the model (one that supports tool calling).</li><li>Click <strong>Tools → Add tool</strong>.</li><li>Add the <strong>AI Search</strong> tool and select product_document_index.</li><li>Add the two <strong>UC function</strong> tools: get_policy_details and get_service_history</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TnSr-LGr8dkp13T4HW8DRQ.png" /></figure><p><strong>Ask Questions</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q6APuUnBZKkIlIpNFO-ONw.png" /></figure><h3>Step 9 — Register and deploy the agent (Model Serving)</h3><p>The Playground is a test bench, so to share the agent we turn it into a live service, which Databricks calls “serving.” When nobody is using it, it goes to sleep to save resources</p><p>When nobody is using the agent, it goes to sleep to save resources, so the first answer can take about 30 seconds. If you see an error saying scale-to-zero must be enabled</p><ol><li>In Playground, <strong>clear the chat history</strong></li><li>Click <strong>Get code → Create agent notebook.</strong></li><li>A notebook opens, already containing the model, tools and <strong>MLflow</strong> logging code. You do not need to understand every line.</li><li>Find the <strong>TODO</strong> cell and set where to register the model:</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KQQ1_vTJ5GtwLI9evalx4g.png" /></figure><p>5. Click <strong>Run all.</strong></p><p><strong>Registered Model</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TkeEOLDmMcK3rFlqavjt2w.png" /></figure><p><strong>Serving endpoints</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_0f_-5Pgh38lCQP1uZsykA.png" /></figure><p><strong>Deployed endpoint answering in Playground</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V2GIGv9H7MNBeJ7O3mnZgQ.png" /></figure><p>In PDF</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cwTMYAeI7mHJH5uCnltJkg.png" /></figure><h3>Step 10 — Give it a face: the Databricks App</h3><p>A chat page is what a normal user would see. We create it from a ready-made template, with no coding. The user types a question, and they never need to know the answer may come from a PDF or a table.</p><p>Business users won’t use Playground. They need a simple chat page.</p><ol><li><strong>Compute → Apps → Create app.</strong></li><li>Choose the template <strong>Chat UI for an existing agent.</strong></li><li>Select your serving endpoint. Leave size and authorization defaults.</li><li>Name it, for example customer-portal, then <strong>Create app.</strong></li><li>Wait a few minutes for the first deployment, then open the app URL.</li></ol><p>If the app shows an error about a missing scope, open the app’s <strong>Settings → User authorization</strong>, add the <strong>Model Serving</strong> scope, redeploy, and open the app in a fresh window.</p><p><strong>App creation</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oEvoSI40FUI5J24NMX7i8Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uEYnJtIOme9Y1Mjy6Lg46g.png" /></figure><p><strong>The final chat UI</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QgBYH38uOweh_fnMS0nFPw.png" /></figure><h3>Step 11 — Watch the agent think (MLflow Tracing)</h3><p>A correct-looking answer can still be wrong underneath. <strong>Tracing</strong> records every step.</p><p>Open <strong>Serving → your endpoint → Traces</strong> and click a recent request. For “Summarize return policy for me”</p><p>1.The AI reads the question and asks for a tool</p><p>2.The tool runs and returns data</p><p>3.The AI reads that data and writes the answer.</p><p><strong>If a customer says</strong> “your bot gave me wrong instructions,” you can open the replay and see: did the bot look in the wrong place, did the search return the wrong page, or did it misread the right page?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1WsrrGX9ZjLpYzGXeJYESw.png" /></figure><h3>Step 12 — Check the agent’s answers (evaluation)</h3><p>Think of a driving test. You don’t wait for a crash to learn whether someone can drive. You give them the same set of situations and check each result against what a correct response looks like. Evaluation is that set of situations for your agent, written down so you can re-run it after every change (a new prompt, a new chunk size, a different model).</p><p><strong>Why it matters most for agents:</strong> an agent has several places to go wrong (wrong tool, wrong passage, wrong wording), and a change that fixes one question can quietly break another.</p><h4>A simple, working evaluation script</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ecV9iEXIVEWnvB3VsOyGcQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B4ciowqcugjIBcxcy7rfrw.png" /></figure><p>In my run, 5 of 7 tests passed. One failure showed raw tool text inside the answer. That is exactly what the test list is for.</p><h4>Letting an AI judge score the answers</h4><p>Text matching only asks, “does the answer contain these words?” A better question is, “is this answer correct and well-formed?” For that, Databricks provides <strong>LLM judges</strong> through MLflow’s GenAI evaluation. A judge is another AI model that reads your agent’s answer and gives a verdict, usually with a written reason.</p><p>Two judges are useful for this project:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Yus6eTwoRsFA9M9v-2JWTg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jhUaeD4p3dvwtBByMEJgFg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WYlfpxzfg_V-Cc9T4LqsqA.png" /></figure><p><em>it shows readers the difference between “contains the right words” and “a judge says the answer is correct,” using the same agent.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=44d06abe00bf" width="1" height="1" alt=""><hr><p><a href="https://medium.com/databrickscommunity/build-your-agentic-ai-app-in-databricks-a-step-by-step-guide-44d06abe00bf">Build Your Agentic AI App in Databricks: A Step-by-Step Guide</a> was originally published in <a href="https://medium.com/databrickscommunity">Databricks Community Articles</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Databricks Multi-Statement Transactions Are GA: All Tables or None]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/databrickscommunity/databricks-multi-statement-transactions-5175d2c17dd1?source=rss----8dd843bb8b39---4"><img src="https://cdn-images-1.medium.com/max/1600/1*Z2rTfpX3wLx63ToxApdOUw.png" width="1600"></a></p><p class="medium-feed-snippet">Databricks multi-statement transactions commit several Delta tables together or roll them all back. How to build pipelines on BEGIN ATOMIC.</p><p class="medium-feed-link"><a href="https://medium.com/databrickscommunity/databricks-multi-statement-transactions-5175d2c17dd1?source=rss----8dd843bb8b39---4">Continue reading on Databricks Community Articles »</a></p></div>]]></description>
            <link>https://medium.com/databrickscommunity/databricks-multi-statement-transactions-5175d2c17dd1?source=rss----8dd843bb8b39---4</link>
            <guid isPermaLink="false">https://medium.com/p/5175d2c17dd1</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Christian Hansen]]></dc:creator>
            <pubDate>Mon, 19 Oct 2026 14:22:42 GMT</pubDate>
            <atom:updated>2026-10-19T13:00:27.507Z</atom:updated>
        </item>
    </channel>
</rss>