<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Analytics at Meta on Medium]]></title>
        <description><![CDATA[Stories by Analytics at Meta on Medium]]></description>
        <link>https://medium.com/@AnalyticsAtMeta?source=rss-217b903f99fa------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Pgyu3YJhzJLG_OD8jmHgzQ.jpeg</url>
            <title>Stories by Analytics at Meta on Medium</title>
            <link>https://medium.com/@AnalyticsAtMeta?source=rss-217b903f99fa------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 01:49:28 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@AnalyticsAtMeta/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com" rel="hub"/>
        <item>
            <title><![CDATA[Scaling AI-Native Analytics Patterns at Meta]]></title>
            <link>https://medium.com/@AnalyticsAtMeta/scaling-ai-native-analytics-patterns-at-meta-b29162cb46bf?source=rss-217b903f99fa------2</link>
            <guid isPermaLink="false">https://medium.com/p/b29162cb46bf</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Analytics at Meta]]></dc:creator>
            <pubDate>Tue, 20 Oct 2026 17:07:33 GMT</pubDate>
            <atom:updated>2026-10-20T18:03:22.425Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YptGNXeR_LszrwMV" /></figure><p><em>Author: Benjamin Wan Rothschild</em></p><p>At Meta, we found that a small set of recurring patterns could answer most product analytics questions, yet teams repeatedly reimplemented core measurement primitives, such as metrics, retention calculations, experiment tables, growth accounting, and funnels. As product launches and measurement combinations multiplied, near-duplicate pipelines accumulated in the data warehouse, creating inconsistencies faster than teams could turn the data into actionable insights.</p><p>Our solution was to standardize popular measurement primitives as parameterized pipeline patterns and ensure every output ships with an agent-readable data model. This design had benefits beyond just standardizing metric calculations across the company. These parameterized patterns create a unified interface where humans and agents operate as peers. Agents can reliably author every pattern using skills and consume them via deterministic query-generation APIs, removing much of the non-determinism and hallucinations we observed with LLM-generated SQL. Now, agents can operate across the analytics stack with precision.</p><p>By treating agents as first-class producers and consumers of our data assets, we created a flywheel of increasingly powerful analytics capabilities, including automated dashboard creation, sophisticated analysis recipes, and accurate metric Q&amp;A agents, described in more detail below.</p><h3>Most analytics is a remix of a few patterns.</h3><p>Most analytics is a remix of a few patterns: How active are our users? Are they coming back? Where do they drop off? What is driving topline movement? We found a power law in measurement: a small set of analytical patterns covers the vast majority of what product teams actually need.</p><p>We cataloged our most common measurement primitives into seven core patterns:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/904/1*oe3mwcP9i_qCuVYKIJPAwg.png" /></figure><p>While your specific measurement catalog may vary, it will likely be equally concise. Standardized, predictable primitives foster trust and confidence by providing a stable interface that AI agents can use to produce and consume insights.</p><p>Standardization is powerful, but not universal. It pays off for recurring, high-trust measurements, but fails when teams need flexible, exploratory, or one-off analyses. Forcing early-stage exploration into rigid schemas can stifle discovery and innovation. Additionally, maintaining governed libraries and asset catalogs requires a continuous investment from a stable team. Organizations should balance when to enforce rigid primitives with when to allow unconstrained exploration. Standardizing a pattern helps only when widespread reuse offsets the maintenance cost.</p><h3>What we built: Analytics Libraries</h3><p>If a pattern is repeated often, define it once. That is the idea behind Analytics Libraries: reusable, parameterized, governed implementations of each canonical pattern. Instead of writing custom SQL for retention or funnels, you declare your requirements, such as your activity event, window size, and key dimensions, and the library generates the underlying SQL and pipelines. Because these declarations use structured input, an AI agent can easily generate them. Having the patterns in parameterized libraries means that essential logic, such as SQL strings and pipeline task types, cannot be hallucinated because it is generated deterministically, in our case, in a Python library used in the data pipeline.</p><p>Metric Materialization is a canonical example of this strategy. Historically, metrics were scattered across fragmented, plain SQL snippets with inconsistent calculations. Now, they live in governed artifacts that are produced using the same logic across the company.</p><p>Here is what that looks like in practice:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/709/1*ozppff77UnooiF6BaYcayA.png" /></figure><p>Since these specifications are declarative and strongly typed rather than free-form SQL, agents consistently and accurately author pipelines that match human accuracy, turning a vague request into a production-grade pipeline. The metric materialization library alone now supports over 3,250 production pipelines, with over 9,000 pipelines powered by the full catalog of Analytics Libraries.</p><h3>Why standardizing the patterns pays off</h3><p>Encoding a pattern once and reusing it empowers your team, making analytics faster, more accurate, and easier to manage. It pays off in the following ways:</p><ul><li><strong>Correctness by default.</strong> The subtle, easy-to-get-wrong parts are solved and reviewed once, then inherited everywhere. Correctness stops being something that each analyst or agent has to relearn.</li><li><strong>Speed.</strong> Authoring a parameterized pipeline with an agent is now minutes of configuration.</li><li><strong>Discoverability and governance.</strong> Standardized outputs are registered and described in a database (more details on that below), so people and agents can find them, trust them, and stop rebuilding what already exists.</li><li><strong>Compounding reuse.</strong> Every new pipeline built on a shared, well-understood primitive makes it easy to roll out new features across all pipelines.</li></ul><p>One hidden tax of analytics is duplicated work. Standardizing the patterns is how you stop paying it, and it is the precondition for handing any of this work to an agent for consumption. If you point an agent at a pile of bespoke, undocumented pipelines, it is more likely to drown in irrelevant context and land on the wrong calculation. If you point the agent at a clean catalog of governed output tables, each with a self-describing schema and a registered semantic model, it behaves like a domain expert.</p><h3>An AI-native foundation</h3><p>With a standardized foundation, agents can produce, consume, and analyze analytics questions with ease. We created skills for pipeline generation, deterministic SQL generation for querying data models, and analysis recipes for interpretation. None of this would be possible without the robust, standardized layer that Analytics Libraries support.</p><p>End-to-end, the full workflow looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/695/1*hsX-vAb2aME4M39V0YOg5w.png" /></figure><h3>Produce Pipeline and Model: skills that let anyone build a pipeline.</h3><p>The first layer is a set of plain-language skills that serve as entry points, turning a request into a production pipeline. Anyone can say “I want to measure retention for this sign-up event, split by country,” and the agent maps that request onto the right library to produce the pipeline. Before the agent starts, it also checks whether a similar measurement exists in the model database to decide whether to reuse it or start from scratch. The agent then fills in the parameters and generates the pipeline using the governed specification. This pipeline generation process leverages the automated code generation and validation loops developed for <a href="https://medium.com/@AnalyticsAtMeta/how-we-built-demate-taming-llms-for-data-engineering-at-meta-d134e69637c5">DEmate</a>, ensuring generated code and configurations adhere to Meta’s data engineering standards before deployment.</p><p>In addition to the materialized table, each pipeline outputs a data model in YML format that encodes a strongly typed representation of dimensions, metrics, and data sources for query generation.</p><h3>Query Model: a deterministic query generator the agent calls as a tool</h3><p>The hardest problem in analytical AI is trust. A large language model will happily write SQL that looks right and returns a wrong number. For this part, we try to make the SQL generation as deterministic as possible. We built a deterministic query generator and exposed it as a command-line tool. We found that an agent learns much better to use a simple API to generate SQL than to generate SQL directly. The agent calls this CLI instead of writing SQL by hand. The model handles the fuzzy part, understanding the question and choosing the inputs, while the deterministic tool turns those inputs into correct SQL.</p><p>The deterministic query generator does two main things. Data source selection and query generation. At Meta, there are often multiple tables that can resolve to the same metric, for example, a cube table vs. a fact table. We encode logic to prioritize data sources based on table retention, partition row count, and the storage layer when determining the best source to query for a metric.</p><p>Here’s an example of a CLI request for a metric.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/714/1*Aig3561-mH9GSeDwwsjhNQ.png" /></figure><h3>Analyze Model: analysis recipes for the questions after “what happened”</h3><p>Analysis recipes encode the procedures a senior analyst would follow to investigate questions such as “what’s driving the retention drop?” or “why did this metric increase?” Recipes walk through procedures using the deterministic query generator, and the LLM will help analyze and summarize the results. We have found that analysis recipes are good at answering questions about metric movements, analyzing reasons for funnel drop-offs, and finding product-market fit for new products among user demographic dimensions. For us, recipes with encoded procedures have produced more reliable results than asking agents free-form questions.</p><h3>Consume Model: dashboards that build themselves.</h3><p>Finally, the dashboard. Because each library has a standard YAML output and a deterministic query generator, we can easily power consumption experiences, such as dashboards, and provide many dashboard templates for metric investigations, growth, retention, and funnel analytics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/895/1*xkTqnBFEh9RaqD7w8aKQCQ.png" /></figure><h3>What’s next?</h3><p>We’re still early in building an AI-native analytics stack at Meta. Currently, we’re extending the libraries to support more patterns, auto-generate richer semantic models, and create more analysis recipes. The goal is easy to state but hard to achieve: anyone at Meta should be able to describe what they want to measure and receive a correct, governed answer, whether they ask a colleague or an agent.</p><p>For any analytics org, the lesson is the same: find your patterns, standardize them early, and document them well. The teams doing the unglamorous standardization work now are the ones whose agents will actually work later. You can’t automate a mess. A small set of well-defined patterns, encoded once, is a foundation you can build almost anything on, including the agents that are starting to do the work for us.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b29162cb46bf" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Measuring What Matters: How Causal Inference Turned AI Infrastructure Into a Quantifiable Business…]]></title>
            <link>https://medium.com/@AnalyticsAtMeta/measuring-what-matters-how-causal-inference-turned-ai-infrastructure-into-a-quantifiable-business-bbc49bddaa2b?source=rss-217b903f99fa------2</link>
            <guid isPermaLink="false">https://medium.com/p/bbc49bddaa2b</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[meta]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Analytics at Meta]]></dc:creator>
            <pubDate>Fri, 11 Sep 2026 23:16:30 GMT</pubDate>
            <atom:updated>2026-09-11T23:16:30.035Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1fDKfJJ6J9tYG5yqrLCMmA.png" /></figure><h3>Measuring What Matters: How Causal Inference Turned AI Infrastructure Into a Quantifiable Business Investment</h3><p><em>Authors: Tong Wu, Yang Zhang, Maggie Chen, Brad Hedges, Konstantin Golyaev, Gniewko Lubecki, Paolo Massimi</em></p><p>When A/B testing is not possible and the stakes keep climbing, observational causal inference offers a principled mechanism to attribute how much incremental revenue gets unlocked by Infrastructure spend. <em>And the answers can surprise you: the platform that looked like our best bet by adoption turned out to be mostly selection, while a quieter one was doing the real work.</em></p><h3>The Setting: ML Infrastructure for Advertising</h3><p>We developed and applied this methodology within the specific context of machine learning infrastructure designed for online advertising.</p><p>In large-scale advertising systems, ML models are the primary engine of revenue. They predict which ads to show, how much to bid, and how to rank candidates, all in real time and at enormous scale. The quality of these predictions directly determines how much revenue the platform generates. When a team ships an improved model, the revenue impact is measurable through A/B tests.</p><p>Behind these models sits a deep layer of ML infrastructure: training platforms that let engineers iterate on model architectures faster, serving systems that deploy models to production with lower latency and higher throughput, feature engineering pipelines that provide richer signals, and experimentation frameworks that accelerate the test-and-launch cycle. These infrastructure capabilities do not generate revenue directly. Instead, they make it possible, or faster, or cheaper, for ML engineers to ship the model improvements that do.</p><p>The objective of an ML infrastructure team, then, is to maximize the pace and magnitude of downstream model improvements by building better foundational tools. The key question is: how much of the resulting revenue should be attributed to the infrastructure that enabled it, versus the ML engineering work that directly produced it? This is the measurement problem we set out to solve.</p><p>While our work is situated in advertising, the pattern is general. Any organization where infrastructure teams build shared capabilities that downstream teams use to generate business outcomes faces the same attribution challenge.</p><p>Most major technology companies are pouring billions into AI infrastructure. And almost universally, the same question goes unanswered: how much incremental revenue did any of this actually unlock? Not uptime or throughput; infrastructure teams measure those well. The question is business impact: when you invest an engineering-year into a new serving platform, how much additional revenue does that produce? For most organizations, the honest answer is: we do not know. The reason is structural.</p><h3>Why A/B Testing Cannot Solve This</h3><p>Infrastructure teams build foundational capabilities. ML engineers use them to ship model improvements that drive revenue. The engineers run A/B tests, and their impact is directly measurable. But when it comes to infrastructure teams that enabled them, their contribution stays invisible. None of it shows up in A/B test results.</p><p>Infrastructure is a shared resource serving dozens of teams, and its value cannot be localized. Value is mediated through other teams’ work. And randomization is impossible: you cannot randomly assign model launches or your engineers to use a platform while others do not. Sometimes it is technically impossible, other times it is prohibitively expensive, but it is almost always highly impractical.</p><h3>The Identification Strategy</h3><p>When randomization is infeasible, we turn to observational causal inference. The unit of analysis is the individual model launch proposal, each representing a discrete ML improvement shipped to production with a measurable revenue outcome. The key treatment variable is whether that launch relied on a given infrastructure capability.</p><p>The identification relies on a selection-on-observables assumption: conditional on a rich set of covariates, infrastructure adoption is effectively as-good-as-random with respect to revenue outcomes. The core estimating equation takes the form:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/652/1*5RI7em9G0dP2bs40cu2m8A.png" /></figure><p>where Adoptedi is an indicator for whether launch i used the infrastructure capability, Xi is the confounder vector, and β₁ is the average treatment effect on the treated, i.e. the incremental revenue causally attributable to adoption.</p><p>Across the program we evaluated seven infrastructure platforms spanning the ML stack, from training-data pipelines through model productionization, putting every platform on a common footing is what let us compare investments that had previously been incomparable.</p><p>The credibility of β₁ depends entirely on our ability to control for confounding factors:</p><ul><li><em>Compute capacity.</em> Teams with larger GPU allocations ship higher-revenue launches. Omitting capacity produces upward bias if better-resourced teams adopt platforms at higher rates.</li><li><em>Human capital investment.</em> Team size and seniority correlate with both adoption propensity and launch quality. Controlling for headcount isolates the infrastructure effect from the human capital effect.</li><li><em>Model production cycle seasonality.</em> Revenue follows cyclical patterns tied to planning and launch cadences. Without time controls, seasonal upticks could be spuriously attributed to coincidental platform rollouts.</li><li><em>Model size and complexity.</em> Larger models generate disproportionate revenue. If platforms are preferentially adopted by teams training larger models, omitting model size confounds the treatment effect.</li></ul><h3>A Worked Example: When the Obvious Winner Wasn’t.</h3><p>Taking two platforms for example. The first was a production model-foundation layer that nearly every model in production runs on. Because it was everywhere, by raw usage it looked like one of the most valuable capabilities in the portfolio. The second was a training-scalability platform used by a smaller set of teams pushing the largest models; it rarely stood out on adoption. Ranked by usage, the call was obvious: keep investing in the foundation layer that everything depends on.</p><p>But usage is not value. A capability that every model touches will always look important by adoption, yet being ubiquitous is not the same as being the reason revenue moved. After conditioning on compute allocation, team size and seniority, model complexity, and launch timing, and restricting the controls to launches that could have adopted each capability but did not, the ranking inverted. The always-on foundation layer’s incremental contribution per adopting launch turned out to be small. The quieter training-scalability platform’s was roughly twice as large, and it delivered more incremental revenue despite far lower adoption. The foundation layer’s apparent value was mostly selection: strong teams would have shipped strong launches with or without it.</p><p><strong>Figure 1.1 — Two lenses on the same two platforms</strong></p><p><em>Left, the leaderboard by adoption. Right, the leaderboard once you condition on compute, team size and seniority, model complexity, and launch timing, and compare only against launches that could have adopted each platform but did not. The lines cross. That crossing is the whole finding.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZhUurjOEqubgriN-9pEmlg.png" /></figure><p>That single comparison reshaped the next planning cycle: we redirected the next increment of investment away from the capability that was merely selection toward the one that was demonstrably causal.</p><h3>Robustness and Validation</h3><p>Any observational study faces challenges in data gathering and confounder identification. We address several explicitly.</p><ul><li><em>Sample size and stability.</em> Early estimates relied on ~258 proposals from a single half. Refreshing across two consecutive halves doubled the sample, improved precision, and showed that β₁ remained stable across estimation windows, substantially strengthening the causal interpretation.</li><li><em>Counterfactual quality.</em> Not all non-adopters constitute valid controls. Proposals that could not have adopted a platform because it was technically inapplicable are excluded from the control group through close collaboration with platform engineering leads. This tightens the comparison to proposals where adoption was feasible but did not occur, reducing bias from selection on unobservables related to technical applicability. A good analogy here is to only enroll patients who are diagnosed with a condition, and then randomly allocating them into either treatment or placebo groups, while excluding people who do not present with the condition we seek to address with the medicine.</li><li><em>Cross-platform overlap.</em> A single launch may leverage multiple platforms simultaneously, risking double-counting. We address this through matching methods that balance treatment and control on overlapping platform usage, subsetting to single-platform adopters, and adjusting pooled estimates by subtracting co-adopted platform contributions.</li><li><em>Triangulating the answer.</em> Where alternative measurement approaches exist, such as mediating-metric methods, or independent causal estimates from related programs, we use them and compare results. Consistency across methodologically distinct estimates increases confidence that the effect is real rather than an artifact of model specification.</li></ul><h3>Prerequisites for Adoption</h3><p>Three data requirements are non-negotiable.</p><ol><li><em>Counterfactual group. </em>the infra innovation doesn’t roll out 100%, so that you will have a group of models or engineers that are eligible but haven’t used it.</li><li><em>Launch-level adoption tracking.</em> We must know which capabilities each launch used. This is typically the hardest prerequisite, requiring sustained collaboration between infrastructure and product teams.</li><li><em>Revenue attribution at sufficient granularity</em>. Revenue must be tied to individual launches so that the granularity of adoption matches with the impact.</li><li><em>Confounder information</em>. Compute allocation, team size, model characteristics, and launch timing must be assembled into a single analytical dataset.</li></ol><p>Beyond data, organizational buy-in matters. Infrastructure teams must accept revenue-based evaluation. Leadership must understand these estimates carry wider confidence intervals than A/B tests, but are far more informative than no measurement at all.</p><h3>Why This Matters Now</h3><p>AI infrastructure spending is accelerating faster than revenue growth. The era of justifying investment on faith is ending.</p><p>This approach generalizes to any layered technology stack where foundational capabilities enable downstream value. The structural measurement problem is universal; the methodology is portable. It requires assumptions, careful specification, and honest acknowledgment of uncertainty, but where randomized experiments are infeasible, it transforms infrastructure investment from an act of faith into a measurable discipline.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bbc49bddaa2b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Character’s Journey: Designing Content-Specific Analytics for Interactive 3D Environments ]]></title>
            <link>https://medium.com/@AnalyticsAtMeta/the-characters-journey-designing-content-specific-analytics-for-interactive-3d-environments-839da5574d3d?source=rss-217b903f99fa------2</link>
            <guid isPermaLink="false">https://medium.com/p/839da5574d3d</guid>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Analytics at Meta]]></dc:creator>
            <pubDate>Fri, 07 Aug 2026 19:12:58 GMT</pubDate>
            <atom:updated>2026-08-07T19:12:58.614Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*laXY5G4YOZqXl4rBK2WZwA.png" /></figure><p><em>Authors: Mike Kane, Vibha Virmani, Reality Labs Data Engineering</em></p><h3>TL;DR</h3><p>We built a framework called Turbo that gives Horizon Worlds creators content-specific metrics about player behavior inside their 3D worlds. Unlike traditional product analytics where we track users through funnels and flows, the interesting subject here is the character inside the world: where they go, what they interact with, how they experience a space designed by someone else. That shift forced us to rethink our approach from the ground up.</p><p>In this post, we’ll share what we learned about designing analytics for interactive 3D environments, including how we solved state management in a distributed spatial context, scaled from bespoke per-world instrumentation to genre-level metric modules, and gave creators the tools to make data-driven design decisions without needing to become data engineers.</p><h3>🎯 What You’ll Learn</h3><p>This post covers the analytics and engineering decisions behind Turbo. We think some of these lessons could apply to other domains where traditional user analytics fall short:</p><ul><li><strong>Redefining the subject</strong>: Why measuring the “character’s journey” inside a 3D world is a different problem than measuring the user’s journey through a product, and how that reframing shaped our entire approach.</li><li><strong>Schema design for user-generated content:</strong> How we built a governed, strongly-typed event schema that lets thousands of unique worlds share common metric definitions without sacrificing each world’s individuality.</li><li><strong>Scaling instrumentation from O(world) to O(genre): </strong>The module system that reduced the problem space from “every world is unique” to “worlds share genre patterns,” and why this matters for anyone building analytics for platform content.</li><li><strong>Real-time state management in 3D: </strong>How Turbo’s engine tracks player state, including session lifecycle, event sequences, and timer calculations, while persisting state across scene transitions in a distributed environment.</li></ul><h3>🎮 The Problem: Measuring a Piano as a Coffee Table</h3><p><em>“Doesn’t a Piano Make a Great Coffee Table?”</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/932/1*2cWivXbpGG-SpICbQs9gXw.png" /></figure><p>Meta Horizon Worlds lets users create, explore, and interact in immersive 3D environments. Creators build everything from full games to hangout spots and art spaces, with real incentives behind them, including the <a href="https://developers.meta.com/horizon/blog/empowering-horizon-creators-with-a-new-50m-fund-and-advanced-tools-for-mobile-and-mr-worlds/">$50M Creator Fund</a> that rewards the most engaging worlds through bonuses and competitions.</p><p>When the platform launched, creators could see standard content metrics: visitor counts, average time spent, retention curves. Useful, but generic. Those of us in Analytics recognized that analyzing 3D worlds with these metrics felt a lot like measuring a piano as a coffee table. A piano makes a great coffee table, but it’s not the best way to measure one, especially if you want to tune it.</p><p>World metrics tell us how well titles are performing. But creators needed to know why, and what they could do about it.</p><p>Here’s the core insight that changed our thinking: in most products, the user IS the subject. You track a person clicking through a funnel, filling out a form, completing a purchase. But in a 3D world, the analytics subject worth studying is the player’s character. Where did they walk? What did they interact with? Did they find the hidden room? Did they pick up the rocket launcher? Did they even make it past the lobby?</p><p>This is a different kind of problem than general funnel analytics on a user journey. It’s spatial, it’s real-time, and the “funnel” is a 3D environment where players can go anywhere, in any order. Traditional product analytics frameworks weren’t built for this.</p><h3>🔍 Proving the Value: Two Case Studies</h3><p>Before getting into architecture, here’s what this kind of analytics looks like in practice.</p><h3>Case Study 1: Super Rumble</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/554/1*m4uD-LnMoRmWSwfx9W-0rA.png" /></figure><p>When we launched our flagship Player vs. Player (PvP) Arena title “Super Rumble,” we anxiously tracked metrics like visitor count and average time spent on our dashboards. These traditional content metrics confirmed that opening weekend was healthy overall, but that’s about all they told us. We piloted Turbo to go deeper. First, we confirmed players made it past the world lobby and actually played Super Rumble, a metric we call “Game Participation %.” Low Game Participation suggests issues with the lobby design, or worse, the core experience is broken. Super Rumble showed healthy participation (&gt;90%), which eased our minds.</p><p>But then Turbo’s spatial analytics revealed something we never would have caught with standard metrics: 40% of players never visited the “Superpowers” section to equip their special ability before the fight. A huge disadvantage for those players, and a major bummer since the creators spent many hours crafting those abilities. After reviewing the data, the team redesigned that section with clearer signage and closed the gap on usage, making sure players had a chance at a fair fight. On the weapon balancing front, Turbo revealed that the Rocket Launcher was overpowered based on Knockouts (KOs) per Round by Weapon Type. Rather than waiting for players to rage-quit, the team proactively rebalanced each weapon to keep the fights fair and fun.</p><h3>Case Study 2: Kaiju</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/586/1*xUu9goGdYGThrBiyJppwAw.png" /></figure><p>Kaiju had dangerously low Game Participation (62%) during its open beta. Turbo revealed that players were abandoning the world while waiting for opponents. The developers responded with a clearer lobby flow showing players what and who they were waiting for, an automatic countdown once start conditions were met, and the option to swap roles between Kaiju and Defenders. Participation jumped to 96% just before GA release. Without content-specific metrics, that 62% would have been invisible in aggregate world metrics until it was too late.</p><h3>⚙️ Architecture: How Turbo Works</h3><p>Turbo provides a simple TypeScript API to track actions via one-liners added to existing world code. The framework handles everything else. Here’s what’s under the hood:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/982/1*PIO20iYn7J8Soc83I7vMqw.png" /></figure><h3>API Layer</h3><p>Built in TypeScript for consistency with Horizon Worlds Scripting. Creators map their custom events to a governed, enforced schema with strongly-typed payloads. “Strongly-typed” means every event has a predefined structure and data types that are validated at build time, not just at ingestion. Think of it as a contract: creators define what they want to track, Turbo enforces how that data is shaped. Because the schema is shared, every world sends data in the same structure, which dramatically simplifies downstream processing.</p><h3>The Engine: Real-Time State Management in 3D</h3><p>This is where it gets interesting from a data engineering perspective. The Engine is the core analytics runtime that manages Player State: tracking each player’s session lifecycle, event sequences, and timer calculations in real-time within the 3D environment.</p><p>What makes this challenging is state persistence across scene transitions. When a player moves from the lobby to the arena to a spectator area, the Engine maintains their analytics state continuously. It handles time synchronization, applies calculation logic every frame, and manages the lifecycle of events that span multiple scenes. Creators configure their own timer settings and event mappings, but the hard parts (state persistence, frame-level computation, distributed synchronization) are handled automatically.</p><h3>Templates &amp; Debugging Tools</h3><p>To lower the barrier to entry, Turbo includes pre-built starter packs in the asset library with genre-specific implementations. For example, PvP templates come with built-in tracking for points scored, weapons used, and match outcomes. Automatic round timers handle match lifecycles so creators don’t have to wire up their own timing logic. Creators drop these in and start measuring immediately. Automated testing tools help validate instrumentation before ship, so broken logging gets caught at build time, not in production.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LZuPw4r0-5pLIkUv9BRg6A.png" /></figure><h3>Data Backend</h3><p>Events flow from the Turbo API into Meta’s ETL pipeline, where they go through validation and deduplication before landing in dimensional models designed for genre-specific metric computation. Key pieces:</p><ul><li>Event validation and deduplication: Ensuring data integrity at ingestion</li><li>Dimensional modeling: Enabling genre-specific metric computation at scale</li><li>Capacity management: We size and scale on behalf of creators so they never have to think about data infrastructure</li></ul><p>The entire pipeline is abstracted away. Creators focus on insights instead of infrastructure.</p><h3>Creator Insights Portal</h3><p>The visualization layer where creators access interactive heatmaps, time-series trends with various slices, and metric breakdowns. This includes automatic breakdowns by interface (VR vs. mobile), which is critical for spotting engagement gaps between platforms. Detailed Turbo documentation is available in the <a href="https://developers.meta.com/horizon-worlds/learn/documentation/performance-best-practices-and-tooling/analytics/world-analytics/">Developer Portal</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*go5_1n0w9xAOUtocIOgSlw.png" /></figure><h3>📐 Scaling: From O(World) to O(Genre)</h3><p>The biggest design challenge was scale. Each world is unique, so how could an opinionated framework support each world’s individuality and still provide automatic metrics?</p><p>We solved it with a module system that operates at the genre level:</p><ul><li>Generic modules apply to all worlds. For example, Area Analytics: creators give unique names to their areas, but the module tracks visitation rates and time spent using a consistent methodology everywhere.</li><li>Genre-specific modules contain standardized events and metrics for categories like PvPs and Obbys (obstacle courses). These were developed from common patterns and analysis needs within each genre.</li></ul><p>This reduces the problem space from O(world) complexity to O(genre) complexity. A good analogy: think of an online shopping cart funnel. A metric like “conversions” has the same semantic meaning across different websites, but each site defines what counts as a successful purchase. Same principle here. Conformed metrics at the genre level give you real depth without requiring custom work for every individual world.</p><p>Turbo provides 40+ metrics depending on which modules are used. Here’s a sample of what each module covers:</p><ul><li>Area: Where do players go and how long do they stay? (spatial flow, dwell time)</li><li>Discovery: Are players finding the features and easter eggs creators built?</li><li>Friction: What negative experiences cause drop-offs? Is a challenge fun-hard or quit-hard?</li><li>PvP: KO/death ratios, weapon utilization, balance metrics. Are weapons tuned or does one dominate?</li><li>Rounds &amp; Loops: How many rounds do players complete? Where do they drop off in multi-level gameplay?</li></ul><p>Creators also automatically get metric breakdowns by interface (VR vs. mobile), essential for identifying engagement disparities between platforms. If mobile players show much higher friction than headset users, that’s a signal to rethink the design for that input method.</p><h3>🌡️ Spatial Analytics: The 3D Heatmap</h3><p>Turbo’s Area Analytics powers a fully interactive 3D heatmap that aggregates player position data to show where players spend time, where they don’t, and how they navigate the world.</p><p>We provide several spatial metrics that each tell a different story, and they’re directly actionable:</p><ul><li>Time Spent: An unexpectedly popular area? Move a store closer to increase avatar clothing visibility for in-world purchasing.</li><li>Player Exits: Where are players leaving? Is there a dead-end or confusing layout?</li><li>Crashes: If frequent crashes cluster around complex geometric objects, a creator can space them out or simplify, especially if Discovery metrics show minimal interaction with those objects.</li></ul><p>The real win with spatial analytics is how directly insights translate into map changes. Creators see the heatmap, they move the thing. No need to become data engineers or geospatial experts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BY-EnZkLSLi1hGC-yhLEZw.png" /></figure><h3>💡 Lessons for Your Own Analytics Work</h3><p>Here’s what we’d want other analytics teams to take away:</p><ol><li>Consider who the real subject is. For us, the breakthrough was realizing we needed to track the character’s journey inside the world, not just the user’s session. If you’re building analytics for a platform where someone creates an experience for someone else, the same question is worth asking: are you measuring the right thing?</li><li>Design schemas for the platform, not the instance. We could have built bespoke analytics for each world. Instead, we invested in a governed schema and genre-level modules. The upfront cost was higher, but it scaled to thousands of worlds without per-world engineering effort. Data Engineers rejoice.</li><li>Consider the tradeoffs of an opinionated framework. We chose predefined best practices and configurations because we needed to compute metrics on behalf of thousands of creators at scale. That tradeoff (less flexibility, more consistency) made sense for us. If you’re building analytics for a smaller set of consumers or don’t need to compute on their behalf, you might want more flexibility. The question is: how much do you want to standardize vs. customize?</li><li>Catch problems before the users do. Content-specific metrics let you detect and fix issues (an OP weapon, a confusing lobby, a crash hotspot) before they show up in aggregate retention numbers. By then, the players may already be gone.</li><li>Separate the insight from the infrastructure. Creators aren’t data engineers. They want answers, not pipelines. Turbo abstracts the entire backend so creators focus on “40% of players missed the Superpowers section” not “how do I set up ETL for my event stream.”</li></ol><h3>🚀 What’s Next</h3><p>We’re expanding the detailed metrics available to creators, building better support for world-specific custom metrics beyond genre modules, and exploring AI-powered analytics summaries that give creators plain-language insights without needing to interpret charts. We’re also running UXR to better understand creator needs and make sure these tools match how creators actually think and work.</p><p>Turbo is one piece of the analytics toolkit we’re building for Horizon Worlds creators, and there’s a lot more to come as the platform and creator community continue to grow.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=839da5574d3d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How We Built DEmate: Taming LLMs for Data Engineering at Meta]]></title>
            <link>https://medium.com/@AnalyticsAtMeta/how-we-built-demate-taming-llms-for-data-engineering-at-meta-d134e69637c5?source=rss-217b903f99fa------2</link>
            <guid isPermaLink="false">https://medium.com/p/d134e69637c5</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Analytics at Meta]]></dc:creator>
            <pubDate>Tue, 30 Jun 2026 17:52:56 GMT</pubDate>
            <atom:updated>2026-06-30T17:52:56.047Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OtVsq_fs53BF1_pAYnZzZA.jpeg" /></figure><p><em>By Mahesh Kshirsagar</em></p><p>You can hire the smartest driver in the world, but if you drop them into a city with no GPS, no street signs, and a unique set of unwritten traffic laws, they’re going to crash. This was the challenge we faced at Meta. Frontier LLMs are brilliant “drivers,” but Meta’s <a href="https://medium.com/@AnalyticsAtMeta/data-engineering-at-meta-high-level-overview-of-the-internal-tech-stack-a200460a44fe">Analytics Infrastructure</a> is a private city built on homegrown orchestrators, custom query engines. Out-of-the-box AI doesn’t know our “streets.” To solve this, we built <strong>DEmate</strong>: a specialized AI assistant that doesn’t just write code, but understands the unique topography of Meta’s data engineering ecosystem. In five months, we went from an experimental idea to 3,500 weekly active users landing thousands of automated diffs.</p><h3>The Context Gap: Why General AI Struggled at Meta’s Data Engineering workflows</h3><p>Popular coding agents work well with industry-standard toolsets, but they hit a wall when they encounter Meta’s internal warehouse. A general LLM might write perfect Standard SQL, but it will fail the moment it needs to use our “&lt;LATEST_DS:” partition macros or the hundreds of in-house User Defined Functions (UDFs) we’ve optimized for our stack.</p><p>Beyond the syntax, data engineering at Meta is deeply intertwined with complex operational loads. Our testing process is a prime example: we must run against production-level input while ensuring results are strictly isolated in ephemeral test tables. To an LLM, these aren’t just “quirks” — they are an invisible language. We didn’t just need a chatbot; we needed a system that could navigate the entire “job to be done” within our proprietary infrastructure.</p><h3>The Breakthrough: Recipe Architecture</h3><p>When we launched DEmate in early 2025, even early models like Claude 3.5 Sonnet showed immense potential — provided they were given the right map. We achieved an <strong>80% user acceptance rate</strong> on suggested code changes by pioneering <strong>Recipe Architecture</strong>, a system that keeps the AI on a short leash and guides it through a structured, four-phase execution loop:</p><ol><li><strong>The Cookbook (Auto-Selection):</strong> We house ~70 DE-specific recipes categorized by job roles or repositories. When a user submits a prompt, a sub-agent automatically selects the matching recipe, ensuring the AI starts with the correct “instructions” for the specific internal framework.</li><li><strong>Ingredients (Context Injection):</strong> Rather than letting the AI “hallucinate” internal logic, the recipe forces the agent to pause and ask for missing data (like a specific namespace or SQL logic). We provide the agent with precise examples of internal libraries to ensure the code is grounded in reality.</li><li><strong>Cooking (Generation &amp; Logic):</strong> This is where the agent follows a deterministic plan to fetch schema data and generate patches. It uses the “ingredients” gathered in the previous phase to build a solution that adheres to Meta’s specific coding standards and internal APIs.</li><li><strong>Final Plating (Validation &amp; Feedback):</strong> Like an Iron Chef plating a dish, the AI must verify its work before presentation. DEmate runs internal linters and status checks, feeding any errors back into the AI for an immediate “fix-it” loop. This ensures the user only sees code that is already tested and verified.</li></ol><h3>Efficiency at Scale: Recipe Chaining</h3><p>Individual recipes work really well, however real-world engineering isn’t a single “shot”; it’s a sequence of decisions. To keep our instructions <strong>DRY (Don’t Repeat Yourself)</strong> and prevent “context window fatigue,” we use <strong>Recipe Chaining</strong>. The real world cookbooks often refer reusable recipes like chicken stock into other recipes.</p><p>Instead of overwhelming the LLM with a 50-page manual, we use <strong>Progressive Disclosure</strong>. If an engineer asks to build a pipeline, DEmate triggers the “Base SQL to pipeline” recipe. Once complete, it asks: <em>“Would you like to add Data Quality (DQ) checks?”</em> If they agree, it pulls in the DQ recipe. If the table volume is high, it automatically chains into a “SQL Optimization” recipe. This keeps the AI’s attention laser-focused on one sub-task at a time, drastically reducing hallucinations. The progressive disclosure is also applicable not just to instructions, but also to custom scripts LLM can reuse to perform tasks like pulling logs, job monitoring etc.</p><h3>The Sous-Chef: AI-Powered Code Reviews</h3><p>As we evolved DEmate into a community product, developers across Meta began contributing their own recipes. With this huge influx of new recipes, humans again became the bottleneck as each recipe needed detailed review before we could make it available to all users. To maintain quality without creating a manual bottleneck, we turned to AI to review the AI.</p><p>We automated our review process using 20 highly focused rules that every new recipe must pass. These include:</p><ul><li><strong>Zero Fluff:</strong> Instructions must be direct and information-dense; if a sentence doesn’t teach the agent something new, it’s flagged.</li><li><strong>Explicit Verification:</strong> The agent must never <em>assume</em> success; it must include steps to perform validations specific for the job.</li><li><strong>No conflicting instruction: </strong>Two instructions should not conflict with each other or the global context provided in the agent.</li><li><strong>Duplication Checks:</strong> The AI scans the cookbook to ensure new submissions don’t overlap with existing skills.</li><li><strong>Trigger Validation</strong>: Every recipe must include a balanced evaluation set of positive and negative prompts to ensure the routing logic correctly triggers — or ignores — the skill.</li></ul><p>DEmate succeeded because it became a <strong>community product</strong>. Collaborators from every vertical at Meta now write recipes, fix broken ones, and codify best practices. Even with the detailed AI review we manually review recipes to make sure we follow all best practices. Every new recipe sparks a healthy debate, maturing our codebase in a Wikipedia-style community effort.</p><h3>Simmering in the Background: Asynchronous Validation</h3><p>Even though code generation was automated, engineers spent hours waiting for test jobs to finish before manually checking results and required switching context multiple times. . One of our most significant breakthroughs was solving the asynchronous validation bottleneck. In addition to manual tests, the continuous integration system also flagged issues hours after submitting the diff. Developers were constantly switching context between tasks to resolve multiple blockers they encountered in their development lifecycle.</p><p>To fix the long-running tester we integrated <a href="https://medium.com/@AnalyticsAtMeta/inside-metas-home-grown-ai-analytics-agent-4ea6779acfb3"><strong>Analytics Agent</strong></a> with DEmate. The agent handles the “waiting” in the background. It monitors long-running tester jobs and, once complete, triggers the next step to write queries against the test table based on the test hypotheses. Once the hypotheses are validated, the agent automatically updates the diff with the test results. At any point if we encounter issues, DEmate can go back and fix the issue and rerun the test as needed. This creates a fully automated testing loop that eliminates hours of manual intervention.</p><p>DEmate also checks validation errors / warnings generated by our continuous integration (CI) system for data pipelines. The agent goes back and updates the code accordingly to fix any issues after scanning for CI errors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/935/1*vtwzFPgrAYjY7gSv_VAQCg.png" /><figcaption><em>Fig 1: Before and after auto validation workflow</em></figcaption></figure><h3>A Seamless, Multi-Surface Ecosystem</h3><p>Having the right architecture was only half the battle; we also had to meet developers exactly where they work. We integrated DEmate directly into <strong>Devmate</strong> (Meta’s internal VSCode extension), which fundamentally shifted adoption by putting AI inside the IDE.</p><p>From there, we expanded DEmate’s footprint across Meta’s entire data stack: Task tracking, Daiquery (our SQL interface), Metamate, and the <a href="https://medium.com/@AnalyticsAtMeta/inside-metas-home-grown-ai-analytics-agent-4ea6779acfb3">Analytics Agent</a>. This created a seamless, <strong>multi-agent workflow</strong> that allows for sophisticated hand-offs.</p><p>For example, a Data Scientist can now provide a high-level business definition for “Daily Active Users” to the Analytics Agent, which discovers the relevant logs and builds the initial query. Once the scientist is satisfied with the results, they simply ask a follow-up: <em>“Create the metrics and cubes for this.”</em> At this point, the Analytics Agent hands the baton to <strong>DEmate</strong>. DEmate picks up the task, generates the data pipelines, and handles the diff, all via natural language. Our tools manage the complex, asynchronous testing in the background, monitoring runs and automatically updating the diff with results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/943/1*DIFIzwASOaFffhbns64I6g.png" /><figcaption><em>Fig 2: DEmate flow from multiple surfaces</em></figcaption></figure><h3>Future-Proofing: The Omni Plugin and Agent Orchestration</h3><p>As the AI landscape rapidly evolves, we realized that DEmate couldn’t be tied to a single platform. To stay agile, we built the <strong>Omni Plugin</strong> — a centralized repository of plugins and skills that syncs across multiple coding agents, including Devmate and Claude Code. This architecture allows us to switch between underlying models and agents effortlessly, future-proofing our work against the fast-moving frontier of AI.</p><p>To streamline this, we developed a dedicated <strong>DEmate CLI</strong>. When an engineer invokes it, the CLI launches Claude Code with a custom configuration and a root folder tailored to the task. Most importantly, it dynamically loads a set of skills aligned to the user’s specific organizational group. No matter which team a developer is in, DEmate ensures they have the exact “local knowledge” they need to be productive immediately.</p><h3>Closing: The Future of Intent-Based Engineering</h3><p>DEmate began as a novel way to teach LLMs how to navigate Meta’s unique internal ecosystem. Today, it is fundamentally changing the nature of data engineering. By moving away from manual boilerplate and toward a framework of structured, verifiable recipes, we aren’t just writing code faster — we are codifying the best practices of our entire engineering community. As we look forward, DEmate isn’t just a coding assistant; it is the foundation for a future where complex data infrastructure is managed through simple, high-level intent.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d134e69637c5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Meta: A Blueprint for Valuing Content When A/B Tests Are Not an Option]]></title>
            <link>https://medium.com/@AnalyticsAtMeta/meta-a-blueprint-for-valuing-content-when-a-b-tests-are-not-an-option-7880bac721f1?source=rss-217b903f99fa------2</link>
            <guid isPermaLink="false">https://medium.com/p/7880bac721f1</guid>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[product-management]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Analytics at Meta]]></dc:creator>
            <pubDate>Mon, 18 May 2026 17:51:47 GMT</pubDate>
            <atom:updated>2026-05-18T17:51:47.809Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1xsj8kGhu6h4i3xRFaARig.jpeg" /></figure><p>By Asher Zafar and Anuj Kapoor</p><p><em>This post is authored by Asher Zafar and Anuj Kapoor, Data Scientists in Meta’s Reality Labs. It is part of a series exploring how Meta’s analytics teams tackle complex measurement challenges at scale.</em></p><p>───────────────────────────────────────</p><p>How would a streaming platform like Netflix or Disney+ measure the true impact of a TV show on subscriber retention? Not just whether people watch it, but whether it <em>causally</em> keeps subscribers around. At Meta, we face an analogous challenge: quantifying how individual games and applications drive engagement for Quest VR users. Content is a primary driver of the Quest ecosystem, and the business needs to know which investments are actually working. If a particular action-adventure title keeps users returning far more than its cost would suggest, we should invest more in similar titles. If a popular genre turns out to be easily substitutable — users would just engage with something else — that changes the funding calculus and we might redirect that investment toward content with greater incremental impact.</p><p>The key idea is <strong>incremental impact</strong>. If a user’s favorite fitness app did not exist, would they have churned from the platform, or simply spent that time in another game? This post walks through a causal inference blueprint we developed to estimate the incremental effect of using a particular piece of content on a user’s likelihood of returning the following month — a metric we call <strong>“month-over-month stickiness.”</strong></p><h3>Why the Gold Standard Falls Short</h3><p>The gold standard for measuring causal impact is the randomized controlled trial, or A/B test. For content on a platform like Quest, however, running a clean experiment is often impractical. We cannot easily disentangle the inherent value of <em>using</em> content from the promotional activities that drive its discovery; we cannot remove access to an application from a control group without creating a negative user experience that contaminates results; and the nascent, dynamic nature of the VR market limits the long-term relevance of any single experiment’s findings.</p><p>Faced with these constraints, we move to the next rung on the ladder of evidence: <strong>observational causal inference</strong> [1]. This class of methods lets us estimate causal effects from existing data — a repeatable way to measure content impact without disrupting the user experience. Below we compare these methods against our use-case:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oOP24wrT5i_3pmQcWT45FA.png" /></figure><p>We chose DoubleML for its balance of flexibility, transparency, and robustness. Here’s how the process works.</p><h3>The Causal Measurement Blueprint: Implementing the DoubleML Method</h3><p>To navigate the complexities of observational data, we developed a standardized, repeatable blueprint using “Double Machine Learning” (DoubleML) [2]. This approach uses flexible machine learning to control for the two main sources of bias in observational studies: <strong>confounding</strong> (factors that influence both content usage and retention) and <strong>selection bias</strong> (users who choose certain content are inherently different from those who do not).</p><p>The process follows five steps:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CNdcILcG1IKVd8LZaNSdKQ.png" /><figcaption><em>Figure 1. The five-step DoubleML blueprint: Define, Assemble, Fit, Validate, Apply. <br>Validation failures loop back to earlier stages.</em></figcaption></figure><ol><li><strong>Define the Estimand.</strong> Get precise about the quantity we intend to measure: the Average Treatment Effect (ATE) of using a specific app on month-over-month stickiness for the active Quest user base.</li><li><strong>Assemble Data.</strong> Gather hundreds of pre-treatment covariates (user characteristics measured before treatment) per user — tenure, historical engagement, genre preferences, purchase history — that let the model control for the many reasons a user might choose a particular app and be more or less likely to retain.</li><li><strong>Fit the DoubleML Model.</strong> Train two ML models: one predicting the outcome (stickiness) and one predicting the treatment (app usage), both from covariates alone. DoubleML combines these to isolate the causal effect.</li><li><strong>Validate.</strong> Run a battery of diagnostic tests to stress-test the model’s assumptions before interpreting any results.</li><li><strong>Apply Learnings.</strong> Generate the insights that inform content strategy.</li></ol><p>Steps 1 and 2 are highly specific to the individual business setting. We’ll focus on steps 3–5 below.</p><h3>Fit the DoubleML Model: How DoubleML Isolates the Causal Effect</h3><p>The intuition behind DoubleML is straightforward. By first predicting both the outcome and the treatment using all available covariates, we compute the <em>residuals</em> — the portion of each that our covariates cannot explain. The causal effect is then estimated from the relationship between these two sets of residuals using cross-fitted sample splitting to guard against overfitting bias. After accounting for everything we know about a user, what is the relationship between the <em>unexplained</em> part of their app usage and the <em>unexplained</em> part of their stickiness? That residual-on-residual relationship is the estimated causal effect, stripped of observed confounding.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wgk6SvSp_Q-RaixKOhd1Ag.png" /><figcaption><em>Figure 2. Two ML models — one for the outcome, one for the treatment — account for all measured user characteristics. <br>The causal estimate comes from the relationship between what neither model can explain (the residuals).</em></figcaption></figure><p>This interpretation rests on a key assumption: that we have measured all relevant confounders — the conditional independence assumption. While untestable, we probe its plausibility through the validation tests below. The approach tolerates imperfect nuisance models: its bias depends on the product of errors from both stages, so moderate inaccuracy in either model has a much smaller effect on the final estimate than you might expect. For practitioners interested in applying these methods, the <a href="https://www.pywhy.org/">PyWhy</a> ecosystem and <a href="https://www.pywhy.org/EconML/">EconML</a> library are excellent starting points.</p><h3>Validation: Building Trust Before Interpreting Results</h3><p>No observational causal model should be trusted without rigorous validation. We run several diagnostic tests before interpreting any results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I5HRIfd7zR_QHX6MWVxLZQ.png" /><figcaption><em>Figure 3. Three key validation checks. All should pass before results are interpreted.</em></figcaption></figure><p><strong>Placebo Treatment Test.</strong> Assign a random, fake “treatment” — pretend users used an app they never actually used. A well-specified model should find no effect. If it does, the model is not adequately controlling for confounding.</p><p><strong>Placebo Outcome Test.</strong> Test the effect of real app usage on a pre-treatment outcome that the app could not have influenced — for example, a randomly sampled variable. Again, the model should find nothing. This confirms our covariates adequately control for confounding.</p><p><strong>Common Support Check.</strong> Verify sufficient overlap between treated and control groups using propensity score distributions, trimming observations with extreme propensity scores to ensure estimates are grounded in comparable populations. Conclusions cannot extend beyond these comparable populations.</p><p>We also verify that both first-stage models achieve meaningful predictive accuracy and assess how sensitive our estimates are to potential unmeasured confounding.</p><p>Only after a model passes these checks do we interpret its outputs.</p><h3>Apply Learnings: Turning Estimates into Decisions</h3><p>This is where the framework pays off. The model produces three key scores for every user-app combination in the population:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AQj8vJSOEWtSU10fedyoJQ.png" /><figcaption><em>Figure 4. Illustrative distributions of three outputs from the causal inference pipeline Propensity Score (Uptake), Baseline Stickiness, and CATE (Uplift). <br>Actual distributions vary by app and user population.</em></figcaption></figure><p><strong>Propensity Score:</strong> “How likely is this user to use this app organically?” High average propensity means broad appeal; low means niche. We colloquially call this <strong><em>uptake</em></strong>.</p><p><strong>Baseline Stickiness:</strong> “How likely is this user to retain anyway, without this app?” This is a critical control. Without it, we would mistakenly attribute high retention to an app when it is really just a characteristic of the users who happen to use it — the classic selection bias problem that makes naive engagement metrics misleading.</p><p><strong>Conditional Average Treatment Effect (CATE):</strong> “What is the incremental lift in stickiness <em>for this specific user</em> caused by using this app?” Estimating this at the individual level lets us understand not just <em>whether</em> an app helps retention, but <em>for whom</em>. We colloquially call this <strong><em>uplift</em></strong>.</p><p>By analyzing the joint distribution of these scores across the content portfolio, we can categorize our content on an <strong>Uplift vs. Uptake matrix</strong> to inform strategic decisions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Tl74QrwIqdW9A8TCf0ssoQ.png" /><figcaption><em>Figure 5. Illustrative Uplift vs. Uptake matrix. Each point represents a content title; position and color indicate its strategic profile.</em></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9-OMV6EY9LM50JkGzh42jg.png" /><figcaption><em>Note: The Uplift vs. Uptake framework does not incorporate investment costs (development, promotion, maintenance) required for generating future impact. Strategic implications must be weighed against each title’s investment cost to determine true return on investment.</em></figcaption></figure><p>These quadrants directly inform content funding conversations. A “Hidden Gem” with strong uplift but low uptake is a very different investment thesis than a “Popular but Substitutable” title that looks great on engagement dashboards but is not incrementally driving retention.</p><h3>Future Directions</h3><p>This blueprint gives us a scalable, repeatable way to quantify content value where traditional experiments are not practical. We combine these causal insights with other measurement approaches — survey-based models, traditional engagement metrics, and where possible, experimental evidence — to build a more complete picture.</p><p>Like all observational approaches, our estimates are only as good as our measured covariates. As we expand to new content types, sensitivity analyses that quantify robustness to potential unmeasured confounding will be an important complement.</p><p>We are actively exploring two enhancements. First, we are investigating whether <strong>simpler proxy metrics </strong>— observable engagement signals like session frequency or depth of use — can reliably approximate these causal estimates, making the insights faster and cheaper to operationalize. Second, we are exploring the next method in our comparison table — <strong>instrumental variable approaches</strong> (what we called “Causal Model w/ Instrument”) — that leverage quasi-random variation (such as recommendation system experiments) to further strengthen causal identification.</p><p>No observational model perfectly replicates the certainty of a randomized experiment. But this blueprint provides a rigorous, practical path to making smarter content investment decisions — a meaningful step up from relying on engagement metrics alone.</p><p>───────────────────────────────────────</p><p><em>For practitioners interested in these methods, we recommend the PyWhy open-source ecosystem and the book Causal Inference and Discovery in Python by Aleksander Molak. For more on how Meta’s analytics teams approach measurement challenges, explore other posts on the Analytics at Meta blog.</em></p><h3>References</h3><p>[1] Sarangan, S. (2025). “Ladder of evidence in understanding effectiveness of new products — Part I.” Analytics at Meta.</p><p>[2] Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey, W., &amp; Robins, J. (2018). “Double/debiased machine learning for treatment and structural parameters.” <em>The Econometrics Journal</em>, 21(1), C1–C68.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7880bac721f1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How We Built an AI Second Brain for 60K Knowledge Workers]]></title>
            <link>https://medium.com/@AnalyticsAtMeta/how-we-built-an-ai-second-brain-for-60k-knowledge-workers-78c507dd795b?source=rss-217b903f99fa------2</link>
            <guid isPermaLink="false">https://medium.com/p/78c507dd795b</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Analytics at Meta]]></dc:creator>
            <pubDate>Wed, 29 Apr 2026 20:36:23 GMT</pubDate>
            <atom:updated>2026-04-29T20:36:23.429Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jJGoQLSCBd8dpYUViA4DPw.jpeg" /></figure><p><em>Author: Analytics at Meta</em></p><p>Knowledge workers at Meta routinely contend with workflow fragmentation, where critical information — including meeting notes, tasks, key decisions, and code context — is siloed across disparate platforms. Each new AI conversation starts cold: the same explanations, the same links, the same ten minutes of context-setting before any real work begins.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*7g86hQ2LLoZeVtkv9xT3xg.gif" /></figure><p>So we tested a simple hypothesis: what if an AI agent had persistent, structured access to everything a person is working on, and carried that context across every interaction? Not a chatbot that answers questions, but a working partner that tracks projects, reads meeting notes, surfaces connections, and builds on prior conversations.</p><p>That AI Second Brain experiment, born in the analytics org, has since been adopted by over 60,000 people across Meta: engineers, PMs, designers, legal, finance, communications, and sales. This post covers how it was built, how it grew, and what we learned.</p><h3>How It Works</h3><p>The AI Second Brain has four main pieces.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T9AJ7YWeTcuW0agkfaZT1w.png" /></figure><h3>The PARA Workspace: A Folder Structure That Agents Understand</h3><p>The idea started with an existing productivity framework. <a href="https://fortelabs.com/blog/para/">Tiago Forte’s PARA method</a> organizes all personal information into four categories: <strong>Projects</strong> (short-term, active work), <strong>Areas</strong> (long-term responsibilities), <strong>Resources</strong> (reference material that may be useful in the future), and <strong>Archives</strong> (inactive items from the other three categories, stored for future reference). It was designed for human note-taking, but it turned out to be equally useful for AI agents. It tells the agent not just <em>what</em> information exists, but what’s active, what’s important, and where new information should go.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WmlJ0W1y6GIjwIPWOYul-A.png" /></figure><p>A PARA workspace gives the agent a durable map of a person’s work. When a user opens a new session, the agent already knows their active projects, team structure, working conventions, and recent activity. When a meeting note arrives, the agent reads the content, matches it against known project keywords, and files it into the right folder without being told where it goes.</p><p>This structure also solves a practical problem: context windows are finite. Loading every document for every project into every session wastes tokens and degrades output quality. Instead, the agent starts each session with a lean root context CLAUDE.md (a summary of who you are and what you’re working on) and drills into specific project folders only when the conversation requires it, loading each project’s CLAUDE.md file into context only when opening those folders. This concept is known as progressive disclosure, and it turned out to be one of the most important design decisions. Lean context up front, deeper detail on demand.</p><p>Most coding agents like Claude Code recommend partitioning work into projects, each with its own CLAUDE.md. That pattern works when “project” maps cleanly to a coding repo you cd into; knowledge work is messier. Meeting notes, docs, decisions, and tasks span many projects at once, projects come and go, and most aren’t repos at all. PARA’s per-project CLAUDE.md files line up with this default, but everything above them is new: a root CLAUDE.md that holds your identity and active portfolio across every session, plus a lifecycle layer (Projects, Areas, Resources, Archives) that tells the agent which projects matter right now and where new information should go. That root layer is what makes cross-project skills possible, like reading meeting notes and routing them to the right projects, or generating team reports that pull from everyone’s work.</p><h3>The Infrastructure Layer: The Bridge to Internal Tools</h3><p>Most of the information that knowledge workers need lives not in local files but in the tools they use: document editors, messaging platforms, task trackers, code review systems, wikis. The investment that made this project possible was Meta’s development of MCPs (Model Context Protocol servers) and CLIs (Command Line Interfaces) that give AI agents authenticated, scoped access to these systems.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xKXkfD6ibA-F7h28k2iIOQ.png" /></figure><p>Without this layer, the agent can only read local files. With it, the agent can pull meeting transcripts, check task status, read discussion threads, and write documents, all within the user’s own permissions.</p><h3>The Agent: The Execution Engine</h3><p>An AI agent is more than a model. It is a model plus a <em>harness</em>: the execution environment, tools, and orchestration logic that turn a question-answering system into a work engine. Without a harness, a model can respond to prompts. With one, it can run bash commands, read and write files, call APIs, recover from errors, and chain decisions in a loop until the task is done.</p><p>A capable harness provides the agentic loop (reason, act, observe, repeat), filesystem access, tool calling, MCP integration, and error recovery that sustained knowledge work requires. It does not just answer questions about your projects; it navigates folders, runs searches, calls CLIs, and writes files on your behalf. <em>(At time of writing, our deployment runs on Claude Code with the latest Anthropic model; the architecture is harness-agnostic.)</em></p><h3>The Skills: Workflows as Markdown</h3><p>Skills are reusable instructions encoded as plain markdown files plus some scripts: no compiled code, no servers, no deployment pipeline. Each skill tells the agent how to complete a specific workflow, step by step. Because they are just text, anyone (even the agents themselves) can write, modify, and share them. This turned out to be one of the strongest drivers of both adoption and community contribution. Some examples:</p><ul><li>/para-init bootstraps a new workspace from scratch. The agent scans the user’s recent posts, documents, tasks, wikis, and code reviews to infer what projects they are working on. It proposes a folder structure, generates context files for each discovered project, and populates them with relevant resources. A user goes from nothing to a fully structured workspace in a single session, with no manual file organization required.</li><li>/start-project creates a new project from a brain dump. The user describes what they are working on in free text: goals, stakeholders, open questions, relevant links. The agent then runs deep research across internal tools in parallel, looking for related documents, discussions, and prior work. It presents what it found, proposes a project structure, and creates everything (folder, context file, brief, initial tasks) after the user confirms.</li><li>/read-meeting-notes processes AI-generated meeting transcripts incrementally. The agent scans configured sources (document folders, local files), identifies notes it has not yet processed, extracts action items and decisions, and routes each note to the most relevant project using a weighted scoring system based on keyword matches, stakeholder overlap, and explicit project mentions. Running this daily keeps project backlogs current without manual effort.</li><li>/debrief:team generates a manager-level view of what an entire team accomplished over a given period. The agent resolves the org tree, then launches individual work digests for every team member in parallel, each sourcing artifacts from code reviews, tasks, posts, and documents. It synthesizes the results bottom-up into a portfolio-style report organized by project, not by person, and outputs a shareable HTML page. For a team of 10, this runs in minutes and replaces hours of status-gathering.</li></ul><p>These are just a few of the skills that come preloaded with the Second Brain. The community has since created thousands more, available in an internal library for any employee to install. Users also routinely write their own for repetitive workflows specific to their role or team, often in under an hour.</p><h3>The Snowball: 0 to 63,000 in Three Months</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ktc3WUVpw0pSICsHg60n8A.png" /></figure><p>Adoption was a slow burn until early February, when a non-technical PM published a post titled “I finally built my second brain. Here’s why you should too.” He paired a non-engineer install guide with concrete examples of what it unlocked: drafting documents from live meeting transcripts, synthesizing leadership summaries in minutes, tracking projects across weeks. Within days the system had spread across every function at Meta.</p><p>Soon after, growth outpaced infrastructure: the plugin’s shared cloud storage integration tripped API rate limits and slowed Meta’s broader AI dev environments, requiring a 10x capacity increase. The incident was the cleanest signal that adoption was real, organic, and accelerating well past what we’d planned for.</p><p>Today the plugin has over 63,000 installs across every organizational pillar at Meta, with roughly 10,000 daily active users. The community has built 9 discipline-specific packages (for PMs, data scientists, engineers, designers, and more), a team-level shared context system, and integrations for automated meeting processing, career development tracking, and visual reporting.</p><h3>What We Learned</h3><p><strong>Infrastructure comes first.</strong> The agent is only as useful as the systems it can reach. Authenticated access to internal tools (document editors, messaging platforms, task trackers, code review) is what separates a useful agent from a chatbot. Organizations building AI agent workflows should invest in this infrastructure layer before anything else (highly recommend <a href="https://www.reddit.com/r/LocalLLaMA/comments/1rrisqn/i_was_backend_lead_at_manus_after_building_agents/">this piece</a> on how to build useful CLIs). The applications will follow, often from unexpected directions.</p><p><strong>Progressive disclosure outperforms context dumping.</strong> Feeding an agent everything at once degrades output quality. The PARA structure supports tiered loading naturally: root context first, project detail on demand. Skills follow the same pattern: the agent sees only a short description until it chooses to invoke one. Lean context up front, drill as needed. There is <a href="https://arxiv.org/abs/2602.11988">some evidence</a> that too many context files can be detrimental to the agent’s performance, so also be cautious on what you are feeding the agent on each session and how you write them (<a href="https://www.humanlayer.dev/blog/writing-a-good-claude-md">here</a> is a useful guide to writing good CLAUDE.md files).</p><p><strong>Low-friction onboarding drives viral adoption.</strong> The bootstrap /para-init command, which scans recent activity and builds an initial workspace, removed the biggest barrier to adoption. Users saw real value in their first session, without spending hours organizing files manually. When the entry cost is low enough, people share it with their team the same day.</p><p><strong>Your users are your best builders.</strong> Every major feature after the initial launch (laptop support, Google Drive optimization, automated meeting note processing, visual reporting, shared team context) was built by community members, not the original author. Over a dozen contributors shipped code. Hundreds more wrote guides, answered questions, and created custom skills. Grassroots growth to 60K+ installs happened without any top-down directive.</p><p><strong>Composability creates more value than features.</strong> Because skills are public markdown files and the workspace is just a filesystem, anyone can extend the system for their own needs. Teams built skills for sprint check-ins, performance review tracking, client health dashboards, and data analysis workflows. The plugin stopped being a tool and became a platform, and each extension made the whole system more useful for everyone.</p><h3>What’s Next</h3><p>The project has moved beyond personal productivity. A team-level shared context system, internally called “Third Brain”, lets team members’ individual workspaces feed into a shared knowledge layer, and is now piloting across dozens of teams with hundreds of participants. Proactive agents run on schedules rather than waiting for prompts: morning briefings, automated meeting note processing, end-of-day digests. And the underlying architecture is converging with Meta’s broader AI platforms, bringing structured context and persistent memory to more tools and more users.</p><p>What started as one data scientist’s fix for scattered notes became a company-wide experiment in how humans and AI agents can work together on sustained, complex knowledge work.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=78c507dd795b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Inside Meta’s Home Grown AI Analytics Agent]]></title>
            <link>https://medium.com/@AnalyticsAtMeta/inside-metas-home-grown-ai-analytics-agent-4ea6779acfb3?source=rss-217b903f99fa------2</link>
            <guid isPermaLink="false">https://medium.com/p/4ea6779acfb3</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Analytics at Meta]]></dc:creator>
            <pubDate>Mon, 30 Mar 2026 16:34:21 GMT</pubDate>
            <atom:updated>2026-03-30T16:34:21.940Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bVNBBAeeQQyoW8uhtFDkjw.jpeg" /></figure><h3>From Hack to Company-Wide Tool</h3><p>The hypothesis was simple:<strong> can an AI agent perform routine data analysis tasks autonomously?</strong> Data scientists tend to get asked similar questions over and over, working within a familiar set of tables. An agent seeded with context about which tables a person queries, and how they use them, might be able to handle much of this work on its own.</p><p>To test the idea, a data scientist on the team used Meta’s internal coding agent to hack together a prototype on their devserver: an agent that could execute SQL against the internal data warehouse, with access to a few colleague’s query history for context.</p><p>The first real-world trial started simply enough: after sharing the prototype with a colleague, they asked it to diagnose a drop in a health monitoring metric. The agent identified the right tables, ran several diagnostic queries on its own, and ultimately traced the root cause to a recent code change.</p><p>That was an ah-ha moment that shifted the conversation. Up until then, most data scientists had used AI for surface-level tasks like fixing SQL syntax, searching internal docs. But a fully autonomous agent that could run queries, reason about results, and decide what to investigate next got people excited enough to want to try it for many different applications!</p><p>So the agent moved off a devserver and into production. A small alpha group of data scientists gave direct feedback. A broader beta surfaced edge cases, for example, queries the agent got wrong, tools people needed, tables it couldn’t find. Each round of iteration showed progress. We would fix what broke, add what was missing, open it up to more users, which luckily is the kind of fast iteration Meta’s builder culture is designed for. Workplace posts and word-of-mouth did the rest. People would try it, get surprised by the results, and share it with their team and keep using it.</p><p>Within weeks, there were hundreds of weekly active users. As momentum grew, a dedicated engineering team formed around the project. Six months later, 77% of Meta’s Data Scientists and Data Engineers use Analytics Agent on a weekly basis, along with roughly 5x as many users from non-data roles.</p><p>Below is an example of “Analytics Agent” doing analysis on its own growth!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6YbyEr90nYMR-wl6TweKSg.png" /></figure><p>But why did this work? How could a weekend prototype do work that normally takes an experienced analyst hours?</p><h3>The 80/20 Hypothesis: Making the Problem Tractable</h3><p>The answer lies in a surprising property of data work: it’s often far more repetitive than most people realize. Most analysis builds off of previous work rather than net-new exploration: 88% of queries by Data Scientists rely solely on tables they’ve queried in the preceding 90 days</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vd3ysGu7AwlBtkCX6f_BNQ.png" /></figure><p>This creates a constrained environment, and constrained environments are where AI excels. Meta’s data warehouse has <strong>millions</strong> <strong>of tables</strong>, but any individual analyst’s work typically lives within a few dozen. Just as AlphaGo succeeded by mastering a game with fixed rules rather than the open-ended real world, Analytics Agent often succeeds because each analyst’s work lives within a bounded, learnable domain.</p><p>By giving Analytics Agent access to the history of tables each user has queried, in addition to AI generated descriptions of those tables based on past queries run against them, we seed it with the most critical context: what data matters to this person, how they use it, and what questions they tend to ask. This is the foundation everything else is built on.</p><h3>How Analytics Agent Works</h3><p>An agent needs the same tools and context a human Data Scientist would have to accomplish a task. Our goal is to provide this scaffolding for Analytics Agent and otherwise get out of the way of the model’s intelligence.</p><p>Two of the most important elements of scaffolding are around enriching the user prompt with context (Discover Data &amp; Gather Context in the diagram below) and providing the right reasoning logic (Iterative Reasoning Loop). In addition we put a lot of effort into providing transparency to users on what data is used to answer their questions (“Answer”).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E-zg9s7qwbBCr3cuq2OKdQ.png" /></figure><h3>Discover Data &amp; Gather Context</h3><p>Before our agent begins any analysis it attempts to gather resources that will enable it to succeed, just like a Data Scientist. We try to ensure that the agent has the right personal context (e.g. users past history of analysis) as well as the right business context.</p><p>Data scientists at Meta develop deep expertise in a slice of the business. They analyze this data using SQL, Python, and R, but their work rarely gets committed to a repository. Instead, it lives scattered across query editors and notebooks, invisible to any system that might learn from it.</p><p>To bridge this gap, we build “shared memory” with each analyst. Offline LLMs process every query an employee has run, generating descriptions of the tables they use, how they use them, and what kinds of analyses they perform. These summaries, along with example queries and column-level documentation, are continuously refreshed and stored where the agent can retrieve them on demand.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CBn88I0c_gHdCRb0Dehl8A.png" /></figure><p>This pipeline constructs the analyst’s “domain” which is a bounded starting point that makes the problem tractable. Meta’s data warehouse serves 70,000+ employees building everything from VR headsets to Facebook Marketplace. Without narrowing the scope upfront, the agent would waste time sifting through irrelevant context and would be far more likely to land on the wrong tables entirely. AI agents can’t yet accumulate context the way humans do over months and years on a team. Personal query history bridges that gap by providing the cumulative knowledge that makes the agent genuinely useful from the very first interaction.</p><ul><li>You can even “clone” a colleague by pointing Analytics Agent at their query history which gives you instant access to their domain expertise without needing to ask them (e.g. Answer this question as if you were PersonA).</li></ul><p>Beyond personal history, the agent needs access to the broader knowledge ecosystem: documentation, data warehouse metadata, data pipeline source code, semantic models. We index all of these and make them searchable which gives the agent the same reference materials a human analyst would consult.</p><p>With the right context/ability to gather context the agent, just like a human, is equipped to do the analysis.</p><h3>Iterative Reasoning Loop</h3><p>Data analysis is a domain where an AI agent can close the loop entirely: write code, execute it, see real results, and decide what to do next. The cycle that normally happens between an analyst and their SQL editor now happens inside the agent.</p><p>An analyst can ask “why did signups drop last Tuesday?” and the agent will query the signup table, notice the numbers look normal, check for a logging change, find a deploy that altered the event schema, and surface the root cause. This happens all in a chain of queries that build on the last. While analysts can see every query and steer, Analytics Agent can often self correct.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DU7XLfp1id_FKeRZy4AiKA.png" /></figure><p>Once the agent determines it has correctly identified the right context and has reached a solution, it returns an answer to the user.</p><h3>Answer</h3><p>When we provide an answer to a user, transparency becomes critical. <strong>In analytics, trustworthy answers are the product </strong>which poses a real challenge given the nondeterministic nature of LLMs. Our approach to this is to show everything. Every data point Analytics Agent surfaces is accompanied by the SQL query that produced it, front and center.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*noM3YmRRfK1CEM6RIkM9xw.png" /></figure><h3>Advanced Features</h3><p>Of course, once we had the agent working users wanted ways to add their own customizations and shortcuts to provide the agent with even richer context. An AI that already knows your team’s data, your metrics, and your best practices is even more powerful than one that just has your personal query history.</p><p>To make that possible, we built a layered knowledge system around three concepts borrowed from the kitchen: Cookbooks, Recipes, and Ingredients. Together, they transform Analytics Agent from a general-purpose tool into a domain-specific expert that gets smarter over time.</p><h3>The Architecture at a Glance</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W23eyrt_SfuBLVHLKdBbDg.png" /></figure><h3>Cookbooks</h3><p>A Cookbook is the entry point. It’s a centralized package that bundles everything the agent needs to become a domain-specific expert: recipes, ingredients, business context, instructions, and suggested prompts. When someone starts a conversation from a Cookbook, the agent already knows the team’s tables, metrics, best practices, and analytical patterns with no setup required.</p><h3>Recipes</h3><p>Inside a Cookbook, Recipes define how the agent should analyze. Think of a Recipe as a “standard operating procedure” that encodes the analytical approach your team would use if a senior analyst were answering the question themselves.</p><p>Recipes can be explicitly chosen by the user or auto-selected, and the agent reads short descriptions of all enabled recipes and picks the most relevant one based on the question being asked. A Recipe typically includes some or all of the following:</p><ul><li><strong>Reference Experts:</strong> Points the agent to specific people whose SQL query history it should learn from. This surfaces battle-tested query patterns, table choices, and filter logic.</li><li><strong>Context</strong> <strong>(Instructions)</strong>: Persistent guidance appended to every conversation such as business rules, expected value ranges, SQL patterns, response templates, and domain-specific terminology.</li><li><strong>Custom Validations</strong>: Natural-language validation rules that a separate AI checks against the agent’s output before presenting results. For example: “WAU should be &lt; 8 billion” or “Always filter by is_test=false.” This scales domain expertise into automated quality control.</li><li><strong>Tool Controls</strong>: Controls which tools the agent can use (e.g., only SQL queries, no Scuba), keeping it focused on the right data sources.</li></ul><p>The key design principle: Recipes define what to do (workflows, analysis steps, response formats), but not what the data means. Data definitions belong in a separate layer called Ingredients.</p><h3>Ingredients</h3><p>Recipes tell the agent how to analyze, but it also needs to know what the data actually means. That’s where ingredients come in.</p><p>Ingredients are structured knowledge assets, like semantic models or wiki pages, that teams link to a Cookbook and apply through Recipes. A semantic model, for example, describes not just that a column called <strong>l7_active</strong> exists, but that it means “users active in the last 7 days, excluding churned accounts, measured at the country-day grain.” A wiki page might document a team’s metric definitions, mandatory filters, or known data quality issues.</p><p>By linking these knowledge sources as Ingredients and routing them through Recipes, teams encode institutional context once and make it available to every conversation in that domain. The agent doesn’t need to rediscover that “revenue” means “net revenue after refunds” or that two tables should be joined on <strong>user_id</strong> and <strong>ds</strong> rather than <strong>user_id</strong> alone; it already knows, because the Ingredient told it so.</p><p>Teams can add several types of Ingredients:</p><ul><li><strong>Semantic models: </strong>Pointers to the canonical data documentation for a domain, so the agent doesn’t waste time searching millions of warehouse tables.</li><li><strong>Documentation &amp; URLs:</strong> Links to metric definitions, wiki pages, and business context that ground the agent’s understanding.</li><li><strong>Text Snippets: </strong>Free-form notes, naming conventions, known data quality issues, mandatory filters, or anything an experienced analyst would “just know.”</li><li><strong>Memories:</strong> Corrections and learnings the agent accumulates over time from user feedback, so mistakes aren’t repeated.</li></ul><h3>How the Layers Work Together</h3><p>The relationship between these concepts follows a clear hierarchy:</p><ul><li><strong>Cookbooks are the package;</strong> they bundle the right recipes and ingredients together for a specific team or domain, with built-in learning loops that improve over time.</li><li><strong>Recipes define how to analyz</strong>e; the workflow, the steps, the validation rules.</li><li><strong>Ingredients provide the what;</strong> the context agents need to understand a business domain.</li></ul><p>The net effect: instead of every analyst on a team needing to know which tables to query, what filters to apply, and how to validate results, that knowledge is encoded once and shared with everyone through a Cookbook. The agent behaves like a domain expert from the first question, and gets better with each conversation.</p><h3>Key Learnings &amp; Lessons</h3><p>The foundation of Analytics Agent rests on a counterintuitive insight: Data scientists are often asked to deliver similar types of analyses and insights within their domain creating significant opportunities for automation to amplify their impact; and that’s exactly where AI agents thrive. By seeding the agent with personal context, equipping it with the right tools, and keeping every step transparent, we’ve built something that doesn’t replace the data scientist but scales them, giving everyone access to reliable answers. In building this agent we learned a ton of lessons. Below they are listed in no particular order.</p><h3>Start With a Falsifiable Bet</h3><p>We had a specific hypothesis that, given 88% of queries by Data Scientists on any given day rely solely on tables queried in the preceding 90 days, an AI agent could perform these routine data analysis tasks autonomously. This wasn’t a vague “AI could help with analytics” pitch, it was a specific, measurable hypothesis based on data.</p><ul><li><strong>The lesson</strong>: don’t start with a solution looking for a problem. Start with data that proves the problem is tractable.</li></ul><h3>Personal Context Is a Killer Feature</h3><p>Meta’s data warehouse has millions of tables serving 70,000+ employees. Without narrowing the scope, any agent drowns in irrelevant context. The breakthrough was building “shared memory” with each analyst, offline LLM pipelines process every query an employee has run, generating descriptions of their tables, usage patterns, and analytical style. When the agent receives a question, it already knows the person’s domain before writing a single line of SQL.</p><ul><li><strong>The lesson</strong>: an AI agent without personalized context is just a chatbot with database access. Context is what transforms it into a domain expert.</li></ul><p>Furthermore, while running SQL is table stakes, what differentiates good analysis from bad is domain knowledge, like which tables are canonical, what filters are mandatory, and which values are “normal.” This led to the Recipes, Ingredients, and Cookbooks architecture: a layered knowledge system where teams encode analytical best practices into reusable configurations.</p><ul><li><strong>The lesson</strong>: the agent is only as good as the domain knowledge it has access to. Build systems that make it easy for experts to teach the agent, not just use it.</li></ul><h3>Show Your Work</h3><p>In analytics, a wrong number presented confidently is worse than no number at all. Every data point Analytics Agent surfaces is accompanied by the SQL query that produced it, front and center. The Thinking UI shows planning and reasoning steps in real time. Users don’t have to trust the agent blindly, and they can verify it, just as they’d review a colleague’s SQL.</p><ul><li><strong>The lesson</strong>: for analytical AI, showing your work isn’t a nice-to-have, it’s the core product requirement. Accuracy you can’t verify isn’t useful.</li></ul><h3>Your Community Is Your Product Team</h3><p>In H2 2025 alone we had 750+ feedback posts, 130+ wins &amp; best practices posts, 40+ community talks. When users prototyped Python matplotlib integration, the team productionized it. When power users asked for programmatic access, the team shipped a feature for this within weeks. By the time of open beta, 4,500+ community-created recipes had been used 150,000 times.</p><ul><li><strong>The lesson</strong>: treat your earliest users as co-builders, not beta testers. Their investment compounds and power users become evangelists, and the community extends the product faster than any roadmap.</li></ul><h3>Ship Early, Learn Fast</h3><p>Analytics Agent went from a weekend prototype on a devserver to a company-wide tool used by thousands in roughly six months. The first version was crude, but when it successfully traced a metric drop to a recent code change on its first real-world trial, it proved the concept. The team raised its adoption goal from 20% to 55% mid-half when growth exceeded expectations, and still overshot it. Each iteration made the next one faster, and each new user made the product better for everyone.</p><ul><li><strong>The lesson</strong>: in a fast-moving AI landscape, the biggest risk isn’t shipping too early, it’s shipping too late. Real user feedback is worth more than any amount of fake testing.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4ea6779acfb3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Beyond the Numbers: Meta’s CMO and VP of Analytics Alex Schultz on Career Shifts, and the AI…]]></title>
            <link>https://medium.com/@AnalyticsAtMeta/beyond-the-numbers-metas-cmo-and-vp-of-analytics-alex-schultz-on-career-shifts-and-the-ai-30d52ba6ebfa?source=rss-217b903f99fa------2</link>
            <guid isPermaLink="false">https://medium.com/p/30d52ba6ebfa</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[marketing]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[facebook]]></category>
            <category><![CDATA[analytics]]></category>
            <dc:creator><![CDATA[Analytics at Meta]]></dc:creator>
            <pubDate>Tue, 24 Feb 2026 16:58:32 GMT</pubDate>
            <atom:updated>2026-02-24T17:37:20.626Z</atom:updated>
            <content:encoded><![CDATA[<h3>Beyond the Numbers: Meta’s CMO and VP of Analytics Alex Schultz on Career Shifts, and the AI Revolution that Will Redefine Analytics Roles</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D5uZ0RSsqGFPjkrJMfoFNg.jpeg" /></figure><p>For nearly his entire career, Alex Schultz has been driven by data. Before his shift into the CMO role at Meta, Alex established himself a leader in analytics, a foundation he credits for his later success. His transition didn’t involve abandoning his roots, rather, he used his deep understanding of data to build trust and make marketing “interpretable” for the business, allowing for some big, creative swings.</p><p>In the below Q&amp;A Alex shares his insights for the next generation of data and marketing leaders. He delves into the critical lessons he’s learned, from the importance of translating complex numbers with empathy for your audience, because “being ‘right’ is not enough”, to the core principles of his recently published book, <em>Click Here: The Art and Science of Digital Marketing and Advertising. </em>Read on as Alex discusses the evolution of Meta’s North Star metric, the disruptive force of AI in the industry, and the timeless nature of marketing principles.</p><p><strong><em>1. You have worked in Analytics for nearly your entire career before making the switch to marketing. What advice would you give to someone who is interested in starting out in Analytics?</em></strong></p><p>Being “right” is not enough. The hardest lesson I learned in Analytics — and one that many data people struggle with — is that having the correct data doesn’t matter if people hate listening to you. Early in my career, I was cut out of meetings with senior executives even though I was right about the SEO strategy. My HR VP told me: “They are listening to you, but they are not hearing you”. I was so aggressive with my facts that I was putting them into “amygdala hijack”.</p><p>You have to realize that the executives and peers you are meeting with have context-switched ten times before they got to your meeting. If you come in with complex spreadsheets and no empathy for their headspace, you will fail. You must learn to translate. You cannot just present the numbers; you have to package them in a way that respects the emotional and cognitive load of your audience.</p><p><strong><em>2. And on that note, what advice do you have for Analytics leaders navigating transitions to another function?</em></strong></p><p>First, don’t abandon your analytics roots. Use them to build a fortress of trust around your department. Once the business trusts your math on the things that can be counted, they will trust your intuition on the things that can’t be measured.</p><p>Second, keep an open mind and learn. When I started, I didn’t respect the “creative brief.” I came from direct response; I thought it was just people banging on about words. I had to humble myself and learn from mentors like Antonio Lucio and David Droga to understand that the brief is actually the core of the campaign. This is now really useful with AI because the brief is really just a well defined prompt.</p><p>My expanded role of CMO worked because I didn’t try to stop being a data guy. I used my data background to make marketing “interpretable” for the business, and my management skills to support those who understood the other areas better than me.</p><p><strong><em>3. During your career at Meta, have you seen the North Star Metric change?</em></strong></p><p>I have seen our North Star evolve in three distinct phases: the original mission of connection, a temporary strategic pivot, and our current focus on future technologies.</p><p>When I first joined, the North Star was incredibly simple: “connect the world online”. This wasn’t just a slogan; it acted as a “razor” that simplified decision-making and priorities. Because this was our clear goal, we prioritized user growth over revenue. For example, we refused to do “homepage takeovers” (massive ads that blocked the screen), which MySpace was doing at the time. We knew those ads would make money, but they would hurt the user experience and slow down our goal of connecting people, so we didn’t do them. We used Monthly Active Users (MAU) as the metric to track this goal, but we were careful not to confuse the metric with the goal itself.</p><p>As we succeeded in connecting nearly two-thirds of the world, we realized we had largely achieved our original mission. We had to update the North Star to reflect where we were going next. The North Star is now to “build the future of human connection and the technologies that enable it”. This shift justifies why we pivoted from just being a social media company to investing heavily in AI, the metaverse, and wearables. The goal is no longer just about connecting people on a phone, but about getting them to use the next computing platform through us.</p><p>There was a specific period recently — when our stock price dropped significantly — where our tactical priorities had to shift temporarily. While the ultimate North Star remained connecting people, “revenue” and “profitability” superseded many less urgent priorities for a short period to ensure the business remained viable. I believe you have to distinguish between a fundamental change in your mission and a temporary change in strategy because the business climate demands it.</p><p>The clarity of our North Star is what differentiates Meta from other companies I have worked at, like eBay, where I felt we struggled because we didn’t know if our goal was to maximize market cap or accelerate commerce. At Meta, we always knew: first it was user growth, and now it is future technology.</p><p><strong><em>4. What role did Analytics play in shaping the Northstar?</em></strong></p><p>I like to use the analogy of a ship captain to explain the role of analytics in defining our North Star. The North Star is the destination the captain is steering toward (e.g., delivering passengers safely). Analytics doesn’t decide the destination, analytics acts as the radar, sonar, and engine monitoring systems and more. These metrics act as “guardrails” to ensure the ship doesn’t run aground or overheat while pursuing the North Star. They tell you if you are failing to reach the goal, but they are not the destination itself.</p><p>Analytics helps you use the North Star as a “Razor” for Trade-offs. Once the North Star is defined, analytics provides the data necessary to prioritize it over conflicting goals. I often say that the biggest enemy to your number one priority is your number two priority. Analytics allows the company to measure trade-offs.</p><p>Analytics also play a critical role in refining the North Star’s metric, and surrounding it with guardrail metrics where it is imperfect so it reflects true value rather than “vanity metrics.”</p><p>And finally, when determining if a new North Star is viable, analytics focuses on retention as the “science” of product-market fit. If retention is low, the product isn’t working, regardless of the “soul” or belief behind the North Star.</p><p><strong><em>5. How do you see AI progress shaping the way Meta and its peers operate? How do you see it evolve?</em></strong></p><p>Based on my experience at Meta and the shifts I’m seeing in Silicon Valley right now, I would answer that by looking at three distinct areas: how we work internally, how the business model of the internet is shifting, and the specific framework I use to predict what comes next.</p><p>Internally, the way we operate is changing faster than people realize. I sit in meetings now with an AI agent open next to me — I wasn’t doing that a year ago. I ask it, “What does the real time data say about how Oakley Meta Glasses are selling?” during the Super Bowl campaign and it gives me answers that make me better at my job.</p><p>We are seeing a massive shift in engineering called “vibe coding”. I recently watched a senior leader at Meta — someone who wouldn’t normally be writing code anymore — build a web parser in a day or so that would have previously taken a team of engineers a month to build. Companies like Anthropic are seeing 90-something percent of their code written by AI.</p><p>The lesson here is simple: AI isn’t going to take your job; somebody else using AI is going to take your job. Whether you are a marketer or an engineer, you have to learn the tool. It’s just like learning mobile or the internet; if you don’t skill up, you will be replaced by the person who did.</p><p>When I look at how this shapes operations for us and our peers, I break it down into three specific buckets:</p><ol><li>Efficiency (Doing existing things faster): We are not going to have humans creating 47 variations of a banner ad anymore; AI will do that. Routine coding tasks like migrations and prototyping will be fully automated. This is where jobs will be reduced.</li><li>Viability (Making expensive things affordable)<strong>:</strong> This is a huge growth area. For example, we can now offer customer support chat for people who lose access to their Facebook accounts. Previously, that was economically impossible to staff with humans. AI makes it ROI-positive, which actually <em>creates</em> jobs because we need humans to handle the hand-offs,.</li><li>Possibility (Doing the impossible): This is the most disruptive bucket. Five years ago, social media was “connected” (friends/family). Today, the majority of time spent on Instagram and Facebook is “unconnected” content (Reels),. This was impossible before modern AI systems could semantically understand content and user intent to show you a video from a stranger that you actually like.</li></ol><p>For the wider industry, specifically peers like Google, the disruption is existential. We are moving from a world of Search Engine Optimization (SEO) to AI Engine Optimization (AEO).</p><p>If you look at Stack Overflow, they have been decimated because engineers stopped searching for answers and started asking AI co-pilots. In the future, I believe we will have AI agents marketing to other AI agents. But human preference will also matter, if a user’s AI agent knows they like Tide detergent, it will buy Tide automatically. My job as a marketer will be to convince that agent to switch brands.</p><p>The big question for the industry is: Does the future look like Google (ad-supported search) or does it look like OpenAI (subscription-based chat) or a mix of the two or does it look like something else entirely? If it stays ad-supported, the industry survives. If users pay for search and see no ads, that is incredibly disruptive to the entire internet economy, and this is true even if just the most wealthy users pay to have no ads in their AI because the value of ads will drop.</p><p>We are fully in the game. We are putting AI into wearables like our Ray-Ban Meta glasses because we believe voice and visual interfaces are the future. We are encouraging advertisers and brands to use our AI ad tools and they are, in fact almost all our advertisers are using at least one of our AI-driven products. And there are now a billion monthly actives using Meta AI.</p><p>But ultimately, the companies that win will be the ones that use AI to make the “impossible” possible, rather than just using it to make the routine a little bit faster</p><p><strong><em>6. What books shaped your perspectives on SI and superintelligence?</em></strong></p><p>Look, if you want to understand where we are going with Superintelligence and how “weird” the world is going to get, I recommend three specific essays that give you a framework to think about the range of outcomes.</p><ol><li>“<a href="https://ai-2027.com/">AI 2027</a>” written primarily by Daniel Kokotajlo, a former OpenAI researcher and co-director of the AI Futures Project: This is what I consider the “Max Doom” view of the world.</li><li>“<a href="https://blog.samaltman.com/the-gentle-singularity">The Gentle Singularity</a>” by Sam Altman, CEO of Open AI: This represents the very positive view of how this transition happens.</li><li>“<a href="https://darioamodei.com/essay/machines-of-loving-grace">Machines of Loving Grace</a>” by Dario Amodei, CEO and Co-founder of Anthropic: I think this essay sits nicely in the middle of those two extremes.</li></ol><p>If you read those three, it gives you the mental framework to decide for yourself how fast this is coming and what the outcome looks like. Personally, I am an AI maximalist and a positive guy; I tend to believe in the positive view of the world that Mark Zuckerberg put out in his essay on superintelligence.</p><p>When we talk about the <em>implications</em> of Superintelligence — specifically what happens to humanity when we no longer <em>have</em> to do the thinking — there is a sci-fi book that a ton of people here in Silicon Valley put on their reading lists, “<a href="https://www.amazon.com/Player-Games-Iain-Banks/dp/0061053562">The Player of Games</a>” by Iain M. Banks. This book is essentially a thought experiment on the end game of superhuman AI. It explores what is unique about humanity and what people get excited about in a world where AI does everything for them. I often say that I hope we end up in that “Culture” series future — that’s the outcome I’m rooting for.</p><p>Finally, whenever I try to prognosticate about the future of technology, I have to reference, “<a href="https://www.amazon.com/Profiles-Future-Inquiry-Limits-Possible/dp/0030697832/ref=sr_1_1?crid=22THZJ84U3PSW&amp;dib=eyJ2IjoiMSJ9.pzCx4XPxgHgYqPU26S6U7Jd4n-vG5RJ3b9fiDrW0UM7d1rgVRTZFaL7rr17rWmcTw4PujngWuGqYIEo5l0kpOuYKAosxCYmdM6c0oMFGEtGlsx_JwFC61JAF5QonMJtdnG6AY9AAJiQNu6n5qPachczW90NTMU_YLz1qvSWgZcVN3ZupK1R7TNJkVei9XEaSRmuaRsG1NpBAJ3ytnoZ2mxsLLUMio3nFOZfqB8J1Gss.O4_o-0UduVhkiGXhN4Pd2LjN4GXTkynrJPmhQQ43ZjQ&amp;dib_tag=se&amp;keywords=Profiles+of+the+Future&amp;qid=1771952199&amp;s=books&amp;sprefix=profiles+of+the+future%2Cstripbooks%2C169&amp;sr=1-1">Profiles of the Future</a>” by Arthur C. Clarke. He makes the point that any prophet of the future is doomed to be incorrect. It reminds me to be humble because, as Clarke points out, predicting the details of the future is nearly impossible</p><p><strong><em>7. You recently published a book. What are the key messages you’d want the reader to take away from it?</em></strong></p><p>There are a few reasons I wrote <a href="https://www.amazon.com/dp/0316597597?ref=cm_sw_r_ffobk_cp_ud_dp_M1VCDAZ5BY82SJHJRNRC&amp;ref_=cm_sw_r_ffobk_cp_ud_dp_M1VCDAZ5BY82SJHJRNRC&amp;social_share=cm_sw_r_ffobk_cp_ud_dp_M1VCDAZ5BY82SJHJRNRC&amp;bestFormat=true"><em>Click Here</em></a>. The banal reason is that there was a gap in the market — I wanted to write the modern <em>Ogilvy on Advertising</em> because there wasn’t a single book I could hand to my team that covered the last 20 years of digital channels. The deeper reason is that I am sick and tired of the negativity around our industry. I wanted to say that growth is good, marketing drives growth, and we should be proud of it.</p><p>If you read the book, here are the absolute key takeaways I want you to walk away with:</p><p>1. Incrementality is Everything (Don’t be a “Useful Idiot”)</p><p>This is a core principle of my life and the book. I frankly stole a term from David Ogilvy: the “useful idiot.” In the modern age, a useful idiot is a marketer who uses post-click tracking to measure their results.</p><p>If you just look at who clicked an ad and then converted, you are taking credit for things that would have happened anyway. You have to use rigorous testing — lift studies for social, matched market tests for search — to prove that <em>absent</em> your marketing, that sale would not have happened.</p><p>2. Principles are Timeless, Channels Change</p><p>I want people to stop panicking about new technology. The tools change, but the principles of database marketing have been a continuum for 100 years.</p><p>Direct Mail = AI Chatbots: Sending a postcard in the 1980s required a database, a target, and a message. Chatting with an AI agent today requires the exact same things; it just happens faster.</p><p>If you understand the principles of the past (like email or SMS marketing), you are ready for the future of AI agents.</p><p>3. Goals are Not Metrics (The North Star)</p><p>This is the chapter that business leaders tell me is the most useful. You must have a North Star goal, and you must understand that a metric is just an imperfect description of that goal. A captain’s goal is to get the passengers safely to their destination. The fuel gauge and the radar are metrics. You don’t optimize for the fuel gauge; you optimize for the destination. Metrics are just guardrails.</p><p>4. Kill the Binary: Use “Science” to Buy “Art”</p><p>I want to kill the idea that there is “Brand Marketing” and “Performance Marketing.” All marketing performs. I treat my relationship with the CFO as a transaction. If I measure the direct response stuff perfectly (the Science), I earn the permission to do the unmeasurable stuff (the Art).</p><p>Because I can prove the ROI of our ads, the board lets me close Fifth Avenue to have Lewis Hamilton do donuts in a Formula 1 car. I can’t measure the ROI of the donuts, but the rigorous measurement of everything else buys me the freedom to take that risk.</p><p>5. AI is Not Going to Take Your Job</p><p>Finally, regarding the future: AI isn’t going to take your job; somebody else using AI is going to take your job. You need to learn the tools. It’s just like the transition to mobile or the internet. If you don’t skill up, you will be left behind.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=30d52ba6ebfa" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why We Shouldn’t Condition on Posttreatment Variables in Experiments]]></title>
            <link>https://medium.com/@AnalyticsAtMeta/why-we-shouldnt-condition-on-posttreatment-variables-in-experiments-5746220133ca?source=rss-217b903f99fa------2</link>
            <guid isPermaLink="false">https://medium.com/p/5746220133ca</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[facebook]]></category>
            <dc:creator><![CDATA[Analytics at Meta]]></dc:creator>
            <pubDate>Mon, 24 Nov 2025 19:17:22 GMT</pubDate>
            <atom:updated>2026-03-30T15:21:43.998Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HumRoTjBaN5qvmvdT7Rw9Q.jpeg" /></figure><p><em>Author: Anja Kilibarda</em></p><p><strong>Tldr;</strong> Conditioning on posttreatment variables prevents us from being able to causally identify the effects of our experimental treatments because it introduces bias into our estimates. Biased estimates can lead us to make bad decisions, like shipping features that don’t work. Do not condition on posttreatment variables, or if you did, interpret the estimates as correlational.</p><h3>What is Posttreatment Bias?</h3><p>In experimental settings, posttreatment bias occurs when we condition on variables measured after a treatment is administered, either by controlling for such variables or partitioning the data into groups defined by them. For instance, suppose I launch an experiment testing a new feature in my app that sends people daily reminders. I want to know if this feature makes people use the app more. I estimate a linear regression of app usage on whether people got the reminders, and I control for how many times they opened the app since the experiment started. How many times someone opened the app since the experiment started is posttreatment, because it’s measured after the treatment is introduced. People sometimes condition on posttreatment variables like this to try to explain how a treatment works (e.g., via ‘mediators’) or to exclude people who didn’t exhibit certain desired behaviors (e.g., didn’t spend enough time on the treatment feature).</p><p>But conditioning on posttreatment variables ruins our experiments because it<strong> biases our treatment effect estimates </strong>in ways that we<strong> cannot in practice correct</strong>. This can lead to incorrect conclusions and bad decisions. Let’s use an example to illustrate.</p><h3>Example: Ray-Ban Meta Ad, Awareness, and Purchases</h3><p>Suppose we run a marketing experiment where we randomly assign some users on Facebook to see a Ray Ban Meta (RBM) ad, while others are assigned to holdout (ie. they don’t see the ad). Random assignment allows us to draw causal conclusions because it ensures that the treatment and control groups are the same in every way, except for whether or not they get the treatment — in our case, the RBM ad. We recover the average treatment effect (ATE) via a simple regression model (under standard assumptions):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/390/1*kmENFUE-VOClpsB9Js-8wA.png" /></figure><p>where Yᵢ is RBM purchases and Zᵢ is the treatment indicator.</p><p>Suppose we get curious about <strong>why</strong> exposure to the ad led to increased RBM purchases. We hypothesize that the ad drove RBM purchases because it increased RBM awareness, and we want to test this hypothesis.We included an awareness question in a sentiment survey we fielded among experiment participants after the campaign began and we can thus estimate the following model:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/548/1*gHhlwXCxSDQ20VbM4AGKyw.png" /></figure><p>where Mᵢ is awareness as measured in our post-campaign survey. Puzzlingly, we find that the coefficient b₂ is negative — controlling for awareness, our ad campaign drove <em>down</em> RBM purchases! How can this be? Simply put, it is not: by conditioning a variable measured post-treatment (Mᵢ, awareness), we have introduced bias into our estimate of the ATE.</p><p>In estimating Eq. 2, we end up comparing sales among people who had low awareness <em>despite</em> being exposed to the ad to those who have low awareness but were not exposed to the ad.<strong> If the ad worked, these groups are not the same and cannot serve as counterfactuals for each other.</strong> People who see the ad but whose awareness doesn’t increase might consist disproportionately of individuals who are completely uninterested in tech — like luddites. By contrast, people who do not see the ad and whose awareness doesn’t increase will be more moderate in their interest.</p><p>Comparing these groups leads us to b₂ — the false conclusion that our ad drove down purchasing because the control/low awareness people (our moderates) are likely to buy RBMs at a higher rate than our treated/low awareness people (our luddites). By conditioning on the posttreatment variable Mi, we have now introduced bias from imbalance in another variable — tech interest — that isn’t in the model and that we haven’t even measured, and <strong>we can no longer estimate an unbiased causal effect. </strong>In other words, the groups we are comparing are now no longer the same in every way except for the treatment — they are also now different on tech interest and a potentially infinite number of other things correlated with tech interest. We thus cannot isolate a causal effect for our RBM ad–the whole point of our experiment–only a correlational effect.</p><h3>How Bad Could it Be?</h3><p>Suppose the true value of our ATE was a 2 percentage point increase in Ray Ban Meta purchases this month. Suppose that if we observe an effect at least as large or larger than 1.8 percentage points in our data, we continue investing in this ad campaign; else we drop it and forfeit our investment. To estimate our ATE, we run a regression model where purchasing is regressed on a treatment indicator and we control for awareness, just like in our example.</p><p>Figure 1 plots the estimates we would get depending on how strongly correlated an unobserved confounder is with awareness, our mediator, and purchasing, our outcome. We simulate 100 scenarios, with increasing levels of correlation. For ease of understanding, the x-axis reports the average correlation between the confounder and the mediator and the confounder and the outcome. At very low levels of correlation, we still recover the true ATE of 2 percentage points. But it gets real bad, real quick, as the correlation between the confounder and our mediator and outcome increases. In more than half of the simulations, we recover an estimate less than 1.8 percentage points, which would lead us to drop a campaign that is in reality successful! In this example, we know how bad the confounding is because we’ve constructed it ourselves. In reality, we wouldn’t know how bad the confounding is because it is unobservable. We could be anywhere along the x-axis and we just wouldn’t know it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HEd8iZFou50-Kcn78ZLTlg.png" /><figcaption>Average Correlation of Confounder w/ Outcome &amp; Mediator</figcaption></figure><h3>What Should We Do?</h3><p><strong>Don’t condition on posttreatment variables: </strong>Don’t add them in regressions, don’t cut your data by them, don’t drop participants based on their reactions to things measured posttreatment (e.g., failed manipulation checks).</p><p><strong>If you did condition on posttreatment variables, interpret estimates like you would correlations:</strong> Use descriptive language (e.g., “is associated with”, “is correlated with”) instead of causal language (e.g., “causes”, “leads to”, “generates”).</p><p><strong>Consider </strong><a href="https://www.publichealth.columbia.edu/research/population-health-methods/instrumental-variables"><strong>instrumental variables regression</strong></a><strong>: </strong>If you have measured an instrument that is correlated with the mediator and can in theory only affect the outcome through the mediator, you might consider IV regression.</p><p><a href="https://journals.sagepub.com/doi/pdf/10.1177/25152459211047227"><strong>Implicit mediation analysis</strong></a><strong>: </strong>Randomly assign people to different treatments you believe operate through different pathways. For example, some people get a feature that could only affect the outcome by affecting awareness, others to an otherwise identical feature that should work by affecting awareness and consideration, and others to holdout. If you find the treatments moved your metric similarly, you would conclude that awareness, not consideration, is what we should care about. This is a great option because it’s rooted in the unbiased framework of comparing randomly assigned people to treatments.</p><h3>In Summary</h3><p>It’s common to see experimental analyses condition on posttreatment variables to understand mediation, exclude non-compliant subjects, or evaluate subgroups. However, the strength of causal inference in experiments relies on random assignment, ensuring treatment and control groups are identical except for the treatment itself. When we condition on posttreatment variables, this equivalence is lost — if the treatment affects these variables, the groups are no longer comparable on all other factors. As a result, any observed differences may not be attributable solely to the treatment, turning causal data into mere correlations and undermining the experiment’s purpose.</p><p>Given the effort involved in setting up experiments, it’s crucial to preserve their integrity. Conditioning on posttreatment variables risks invalidating the causal conclusions we’re after, and we should avoid practices that compromise the value of our experimental designs.</p><p><strong>Suggested Reading:</strong></p><p>Bullock, John G. and Donald P. Green. 2021. “<a href="https://journals.sagepub.com/doi/pdf/10.1177/25152459211047227">The Failings of Conventional Mediation Analysis and a Design-Based Alternative</a>,” Advances in Methods and Practices in Psychological Science, 4(4).</p><p>Gerber, Alan S., and Donald P. Green. 2012. <em>Field Experiments: Design, Analysis, and Interpretation</em>. New York: W.W. Norton.</p><p>Montgomery, Jacob, Brendan Nyhan and Michelle Torres. 2018. “<a href="https://onlinelibrary.wiley.com/doi/abs/10.1111/ajps.12357">How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It</a>,” American Journal of Political Science, 3(62).</p><p>Rosenbaum, Paul. 1984. “<a href="https://faculty.washington.edu/peterg/Vaccine2006/articles/Rosenbaum1984.pdf">The Consequences of Adjustment for a Concomitant Variable That Has Been Affected by the Treatment</a>,” Journal of the Royal Statistical Society, Series A, 5(147).</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5746220133ca" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Ladder of evidence in understanding effectiveness of new products — Part I]]></title>
            <link>https://medium.com/@AnalyticsAtMeta/ladder-of-evidence-in-understanding-effectiveness-of-new-products-part-i-ad8dee70906c?source=rss-217b903f99fa------2</link>
            <guid isPermaLink="false">https://medium.com/p/ad8dee70906c</guid>
            <category><![CDATA[facebook]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[instagram]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Analytics at Meta]]></dc:creator>
            <pubDate>Fri, 31 Oct 2025 20:19:25 GMT</pubDate>
            <atom:updated>2025-10-31T20:19:25.146Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w876C5yO0SBJ14LLG6VQYg.jpeg" /></figure><h3>Ladder of evidence in understanding effectiveness of new products — Part I</h3><p><em>Author: Sarayu Sarangan</em></p><h3>Background</h3><p>Randomized Control Trials (RCTs) are considered the gold standard when it comes to measuring the effectiveness of products launched at Meta. Whitepapers that explore observational and randomized experimental methods have time and again shown that causality is best established using Randomized Controlled Trials (RCT), i.e., we observed X effect because of Y</p><p>However, in some product development scenarios, causal inference methods may be necessary, either alongside or instead of experiments.</p><p><strong><em>The article </em></strong>will be broken into two posts. The first part will discuss scenarios for using various analysis methods and activation. The second part will discuss the methods themselves in detail.</p><p><strong><em>Part 1</em></strong></p><ol><li><strong><em>Scenarios for when to leverage experiments, causal inference methods, or both</em></strong></li><li><strong><em>Analysis influence, decisions, burden of proof and active communication</em></strong></li><li><strong><em>Recommendation for using the framework</em></strong></li><li><strong><em>Stepping through the framework using an example</em></strong></li></ol><p><strong><em>Part 2 will go deeper into exploring methods, related assumptions and a hierarchy of methods to consider when using causal inference.</em></strong></p><h3>Part 1</h3><h3>Scenarios for when to leverage experiments or causal inference methods or both</h3><p>When it is possible to randomize the treatment at the right level of granularity and with a sufficient number of observations, we should aim to understand product effectiveness by running Randomized Control Trials (RCTs). Random assignment of treatment to the population of interest eliminates bias and helps establish causality, i.e. we observed effect Y because of treatment X. A number of factors go into selecting experimental tool(s) such as product/ feature being tested, stage of the product development lifecycle, funding, resources (e.g. compute, storage capacity, etc.) to name a few.</p><p>Nevertheless, there are instances where justifying the need for experiments is necessary, or when experiments alone are insufficient for broad generalization, high-confidence conclusions, or in rare cases, even feasible.</p><p>The various scenarios and potential methods are illustrated in the diagram below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i2xw1P4nxjXe_TMsVv8Yvw.png" /><figcaption><em>Figure 1. Visual summary of project constraints and recommended methods</em></figcaption></figure><ol><li><strong>When experiments are feasible:</strong> Whenever feasible, Randomized Control Trials (RCTs) should be the definitive method for demonstrating the causal impact or effectiveness of newly introduced features or products.</li><li><strong>When experiments are feasible, but sample size is small: </strong>As an example, in early stages of product development, adoption of the product or feature may not have reached critical mass. This could lead to under powered tests, resulting in low confidence in the study, impacting ability to establish causality and generalize outcomes of the experiment. In such cases, we could consider leveraging causal inference methods to increase confidence and/or generalizability of the results.</li><li><strong>When experiments are feasible but cost of running experiments is very high:</strong> Running experiments can be costly and difficult to narrow down treatment arms, even when feasible. In such cases, causal inference methods can provide early directional evidence of product effectiveness. This is particularly useful when experiments are expensive to conduct due to factors like limited compute or capacity, or when multiple experiments compete for the same population, potentially reducing power. Additionally, when prioritizing multiple features or products, causal inference methods can estimate expected improvements or ROI with reasonable confidence, enabling a ranked prioritization of potential treatment options.</li><li><strong>When experiments are not a feasible option:</strong> Sometimes, it’s not possible to randomize the variable of interest. This occurs, for instance, when a product or feature must be launched to the entire eligible population due to high demand, making it impossible to reserve a control group for a counterfactual analysis. In such situations, causal inference where possible can be leveraged to understand effectiveness post launching the product, but decisions made off of such methods should be weighed carefully against past experiments if available.</li></ol><h3>Analysis influence, decisions, burden of proof and active communication</h3><p>Analyses conducted by data scientists can have varying implications, influencing decisions throughout the product development lifecycle. As the reach and impact of the analyses broaden, the demands for rigorous proof and the associated analysis intensity increase, as illustrated in the table below.</p><p>Data scientists must be aware of the limitations inherent in various methods and independently evaluate each situation to determine if the chosen methods are sufficiently robust to inspire confidence. When interpreting both experimental and non-experimental outcomes, data scientists should differentiate between “causation” and “strong association.” Experimental results directly indicate causation, whereas quasi-experimental and purely observational methods require justification of the unconfoundedness assumption before causality can be claimed. However, causation is not always necessary, as a strong association can be valuable in certain contexts.</p><p>Data scientists are also responsible for actively and transparently communicating these trade-offs to their product partners. This communication should clearly articulate the level of confidence derived from the different methods employed, as well as the potential influence these insights might have on critical product decisions. This proactive and open dialogue ensures that the entire cross-functional team — including product managers, engineers, designers, and marketing specialists — is fully aware of any caveats, uncertainties, or potential biases in the data. By fostering this shared understanding, the team can collectively make more informed decisions, mitigate risks, and ultimately put forth products that are robust, reliable, and genuinely meet user needs, driving long-term value for both the business and its customers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*avP-ED46HSgDrMTW2qkTKQ.png" /></figure><h3>Recommendation for using the framework</h3><p>To effectively implement this framework, data scientists should consider the following:</p><ul><li>Step 1 — Understand project constraints</li><li>Step 2 — Discuss analysis influence with cross-functional partners. This should explicitly cover what decisions the analysis will influence.</li><li>Step 3 — Propose methodologies to understand product effectiveness, given project constraints and analysis influence</li><li>Step 4 — Inform confidence level, given project constraints, analysis influence and proposed methodology</li></ul><h3>Stepping through the framework using an example</h3><p>As an example of making the framework come to life, we iterate through the development of a new Ads solution, <a href="https://www.facebook.com/business/help/8975791382450840">Omni channel Ads</a> from early stage exploration to launching the product.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P4uVC8-5jZtviuh3hdc8tw.png" /></figure><p><strong>Early Prototype Stage (Product Strategy -&gt; Alpha and Alpha -&gt; Beta):</strong></p><p>In the initial development phase, limited advertiser adoption meant we relied on causal inference analyses to establish a baseline understanding of effectiveness based on adoption and usage of existing features and products (e.g. Reach optimization using specific targeting features that drove online and offline purchases). In the absence of having the actual product, advertisers were leveraging existing solutions to meet their needs. After building the Proof-of-concept (i.e. Omnichannel optimization product that drives online and offline purchases), we ran small-sample experiments to gather preliminary evidence of product effectiveness over existing product usage. A medium confidence level was sufficient at this stage to guide product direction.</p><p><strong>Scaling and Launch (Beta -&gt; Launch):</strong></p><p>As product adoption grew, large-scale Randomized Controlled Trials (RCTs) became feasible, significantly increasing our confidence in the solution’s effectiveness at scale. Prior to launch, a higher confidence level was required due to broader exposure to advertisers and thus broader influence of the analysis. This led to multiple rounds of large-scale RCTs to ensure high confidence in the product’s impact for advertisers.</p><p><strong>Feature Prioritization and MVP:</strong></p><p>Throughout product development, numerous features were considered to be added to the Minimum Viable Product (MVP), but RCT capacity was constrained. Causal inference analyses were employed to understand the association between potential features and product performance, aiding in prioritization. While these analyses offered low confidence individually, they were sufficient to identify promising candidates for more thorough evaluation through large-scale experimentation, ultimately building higher levels of confidence.</p><h3>Principles for using causal inference methods</h3><p>Part 2 of this series will delve into specific methods. However, data scientists should keep these high-level principles in mind when selecting a design and analysis strategy for a given problem, with the goal of minimizing:</p><ul><li><strong>Confoundedness:</strong> Ensuring that what we identify as the cause is indeed the true cause, or at least a significant part of it.</li><li><strong>Bias in our estimate:</strong> Striving for an assessment of size and direction (negative/positive) that is as close to reality as possible.</li><li><strong>Variance in our estimate:</strong> Aiming for the most precise estimates possible.</li></ul><p>In part 2 of the series, we will dive deeper into specific methods, addressing specifically the following aspects:</p><ul><li>Scoping the problem</li><li>Framework for hierarchy of causal inference methods</li><li>Checking assumptions and conducting sensitivity checks</li><li>Considerations for heterogeneous treatment effects</li></ul><h3>Footnotes</h3><p>1. Sufficient in terms of Type I and Type II error rates — an RCT that cannot provide the confidence that a measured effect is indeed present/absent fails to achieve its goal.</p><p>2. This assertion is based on the assumption that random assignment was successful, test groups are balanced on confounders as expected, that there are no defiers and that attrition is symmetric between test and control groups.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ad8dee70906c" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>