<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Zepto Tech on Medium]]></title>
        <description><![CDATA[Stories by Zepto Tech on Medium]]></description>
        <link>https://medium.com/@tech.culture?source=rss-47a9eb8a04a5------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*Zdo4al9KE5LuqNxm</url>
            <title>Stories by Zepto Tech on Medium</title>
            <link>https://medium.com/@tech.culture?source=rss-47a9eb8a04a5------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 02:14:49 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@tech.culture/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com" rel="hub"/>
        <item>
            <title><![CDATA[From Data to Action: Designing for Faster Decisions in Delivery Hub Operations]]></title>
            <link>https://blog.zepto.com/from-data-to-action-designing-for-faster-decisions-in-dark-store-operations-aac167dd07b9?source=rss-47a9eb8a04a5------2</link>
            <guid isPermaLink="false">https://medium.com/p/aac167dd07b9</guid>
            <category><![CDATA[quick-commerce]]></category>
            <category><![CDATA[product-design]]></category>
            <category><![CDATA[ux-design]]></category>
            <dc:creator><![CDATA[Zepto Tech]]></dc:creator>
            <pubDate>Tue, 03 Nov 2026 06:25:22 GMT</pubDate>
            <atom:updated>2026-11-04T11:37:26.363Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EGVWW7uhiRDG6J_4.png" /></figure><p>Imagine it’s 7:00 AM. You step into the delivery hub (a mini-warehouse), and the delivery clock is already ticking. You are responsible for fulfilling 2,500 orders today. Even a 30-second delay can ripple through subsequent orders and throw off delivery times.</p><p>You sit down in front of your laptop and open up your usual eight browser tabs. One is tracking stock levels, another is tracking staff schedules and efficiency, the third is tracking customer orders and so on.</p><p>At the same time, your phone won’t stop buzzing with WhatsApp messages, where the customer support team is coordinating with you to solve customer order issues. Outside, a truck is waiting at the loading dock with replenishment stock and needs your signature. A packer can’t find an item and needs your help.</p><p>So many things are happening in parallel that you are constantly wondering:</p><ul><li>Is your inventory adequate for the orders today?</li><li>Do you have enough packers? Are people on a break, or are you understaffed?</li><li>Have you planned the shift roster well enough for the peak hours? What will you do if people don’t show up?</li></ul><p>You are not actively running the store; you are drowning in dashboards, just trying to figure out what went wrong, what could go wrong, and hoping you catch the issue early so that it doesn’t affect a customer order.</p><p>This was the core problem we set out to solve when we decided to build a dashboard for our delivery hub managers. To protect the delivery promise, they needed to know what was happening in their store, spot potential bottlenecks early, and decide where to intervene without losing valuable time.</p><h3>Figuring out the real bottleneck</h3><p>As Zepto grew from around 400 delivery hubs to over 1,000 in just a year, our Warehouse Management System grew alongside it. But scaling the system also meant scaling the complexity of running a store.</p><p>Hundreds of workflows had been built across the app, and Delivery Hub Managers had to navigate them every day to keep operations running smoothly. From keeping track of inventory across the store and managing packer shifts and rosters to ensuring orders went out on time, they were constantly juggling multiple responsibilities across different parts of the system.</p><p>To understand how these challenges played out on the ground, we shadowed Delivery Hub Managers as they went about their day. What we found was that the problem wasn’t a lack of data, but the lack of a single source of truth.</p><p>Information was scattered across spreadsheets, apps and devices, forcing managers to spend a significant part of their day piecing together numbers and reconciling mismatches instead of focusing on running their stores.</p><p>We found managers relying on a series of manual workarounds to keep things running:</p><ul><li>They downloaded raw data shared by the central operations team and transferred it to custom spreadsheets, with graphs and pivot tables, to make sense of the data</li><li>They manually tracked order defects, staff utilisation, SLA breaches and other metrics to get a consolidated view of store performance.</li><li>They maintained leaderboards on whiteboards to track packer rankings and encourage healthy competition.</li></ul><p>Individually, these tasks might have seemed small. But together, they created a significant operational bottleneck. Managers were jumping between browser tabs, spreadsheets, whiteboards, messages and phone calls, spending valuable time piecing together information instead of acting on it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RTAKMjo6aOFnXCNr.png" /><figcaption>The ground reality</figcaption></figure><h3>Designing for clarity, speed and scale</h3><p>We realised that the problem wasn’t a lack of data, but the effort required to interpret it and the delay in getting it. This led us to one central question: <strong>What does a manager need to know now to keep the store running smoothly?</strong></p><h3>Figuring out what is actionable</h3><p>We looked at the daily bottlenecks across store operations and identified four key areas that managers needed to monitor: <strong>Customer Experience, Manpower, Inventory and Returns.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*k97cIJ3bt25vOqJW.png" /></figure><p>We sat down and validated our approach with multiple Delivery Hub managers and regional operations managers to understand whether the metrics we had noted down reflected their day-to-day needs.</p><p>For each area, we prioritised the metrics that would help them identify operational issues and decide where to intervene.</p><p>Their feedback helped us identify what was relevant, what needed more prominence, and what could be deprioritised.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8rvScD1X0QjqkL3g.png" /></figure><ul><li><strong>Customer Experience<br></strong><em>Late Orders, Customer Complaints and IPP (Items packed per packer)<br></em>These would help the managers spot delays and packing inefficiencies before they affect more orders.</li><li><strong>Manpower<br></strong><em>Packer Distribution, Shift Details, Packing Device Status<br></em>These would help managers identify staffing gaps and improve their shift rosters to keep up with the demand.</li><li><strong>Inventory<br></strong><em>Putaway status, Task adherence, Cost per order, Bin Utilisation<br></em>These would help managers spot inventory bottlenecks and keep store operations moving.</li><li><strong>Returns<br></strong><em>Customer Return Status, Vendor Return Status<br></em>These would help managers track pending return orders and prevent further losses due to negligence.</li></ul><p>We brought these key metrics to the main dashboard, where they were immediately visible and moved the rest of the data tables behind a click.</p><h3>Solving data latency</h3><p>Now that we knew what we wanted to show, we moved on to the next problem, i.e. data latency.</p><p>Most critical metrics updated at a lag of a couple of hours to one day. Managers had no way of knowing if something was going wrong in real time.</p><p>One manager shared an incident where six orders were pending and getting delayed, but no one on the floor knew about it because there were no alerts or live updates.</p><p>This lag created a major trust deficit between the store teams and the central operations teams.</p><p>We knew a better UI and metric visibility wouldn’t solve this problem if they didn’t get this data in real time or near real time.</p><p>So we sat down with the product, analytics and store operations teams to figure out how to solve this latency issue. Making every metric available in near real time would have significantly increased server costs. We therefore needed to prioritise which metrics required frequent updates and which could tolerate a longer delay.</p><p>We had to prioritise and evaluate each metric to understand how fast it needed to update. After having a discussion with the tech team, we categorised these metrics into 3 buckets: <strong>Real Time, &lt; 5 Minute, 30+ Minutes</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FWey7FxDHtKj8klY.png" /></figure><p>Once we aligned on the required refresh frequency for each metric, we could finalise the technical approach while balancing server costs against the visibility managers needed.</p><h3>Designing for scale</h3><p>We didn’t want every new metric to become a one-off design and development effort.</p><p>Creating custom widgets for each metric would have made the dashboard harder to maintain and the experience increasingly inconsistent as we added more capabilities.</p><p>Instead, we built a modular system using six standardised tile patterns.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tQHWJaknPbqaGh7O.png" /><figcaption>The six building blocks of the dashboard</figcaption></figure><p>Each metric card on the dashboard was built by combining these reusable tile patterns.</p><p>This gave engineers a consistent foundation to build on and made the dashboard easier to extend and maintain.</p><p>More importantly, it ensured that as the dashboard grew, Delivery Hub Managers could rely on familiar interaction patterns instead of having to learn a new way to interpret every metric.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*sMllaEhQAtdW84VT.png" /></figure><p>Instead of cluttering the main screen with large tables, we used progressive disclosure to hide granular details until needed.</p><p>The information is structured across three levels:</p><ul><li><strong>L0 (Summary Tiles):</strong> High-level numbers that managers can check at a glance while walking around the store floor.</li><li><strong>L1 (Side Drawer):</strong> Clicking on a tile opens a side drawer with additional context about what is driving that metric.</li><li><strong>L2 (Detailed Tables):</strong> Full data tables accessible when managers need to investigate specific issues or perform root cause analysis.</li></ul><h3>Impact on store operations</h3><p>The dashboard changed how Delivery Hub Managers understood and ran their stores. By bringing operational metrics together in one place, we replaced fragmented spreadsheets, manual reports and scattered updates with a single, consolidated view of store performance.</p><p>The changes were most visible in three areas:</p><ol><li><strong>From chasing data to acting on issues<br></strong>Managers no longer had to download raw data, maintain custom spreadsheets or manually compile daily reports. The information they needed was available on the dashboard, freeing them to spend less time reconciling numbers and more time addressing operational bottlenecks.</li><li><strong>From fragmented reporting to a shared source of truth<br></strong>With the same consolidated view available to Delivery Hub Managers and central operations teams, both could work from a common understanding of store performance. This reduced dependence on manual reporting and made it easier to identify bottlenecks, align on priorities and follow up on issues.</li><li><strong>From delayed visibility to timely intervention<br></strong>We significantly reduced the time it took for key metrics to become available. This meant managers could spot emerging issues sooner and respond while there was still time to prevent them from affecting store operations.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MDvZHAyNJjAEqLWv.png" /></figure><p>Ultimately, the dashboard shifted the manager’s role from piecing together what had happened to understanding what needed attention next.</p><h3>What we learnt</h3><p>This project reinforced something important about designing for data-heavy operational environments: the interface is only one part of the solution.</p><blockquote><em>The real work lies in understanding how operations run on the ground, deciding what information matters, and making that information useful when people need to act on it.</em></blockquote><p>Five principles shaped our approach:</p><ol><li><strong>Understand the workarounds before designing the solution<br></strong>The spreadsheets, manual trackers and WhatsApp groups we uncovered weren’t just inefficiencies. They revealed what the existing systems failed to provide. Understanding these workarounds helped us identify the actual gaps we needed to solve.</li><li><strong>Align on metrics before designing the interface<br></strong>A dashboard is only as useful as the information it presents. Aligning central operations, analytics and store managers on metric definitions and their intended use helped us build a shared understanding of store performance.</li><li><strong>Match data freshness to operational urgency<br></strong>Not every metric needs to update in real time. We prioritised faster updates for metrics that required immediate intervention while allowing others to refresh at a longer interval. This helped us balance operational needs with technical constraints and server costs.</li><li><strong>Design for the pace of operations<br></strong>Delivery Hub Managers work in fast-paced environments where they need to absorb information and make decisions quickly. The dashboard had to make critical metrics easy to scan, surface issues that needed attention, and avoid adding unnecessary cognitive load during busy shifts.</li><li><strong>Build systems, not one-off solutions<br></strong>With hundreds of workflows and an expanding set of metrics, designing each component independently wouldn’t scale. Standardised, reusable tile patterns helped us maintain a consistent experience while making the dashboard easier to extend and maintain.</li></ol><h3>Closing thoughts</h3><p>Looking back, this project was about more than consolidating metrics onto a single screen. It was about changing how store managers accessed and acted on information.</p><p>The biggest takeaway was that internal tools are often treated as reporting interfaces, but their real value lies in <strong>reducing the time between a problem emerging, someone recognising it, and action being taken.</strong></p><p>By designing around the realities of store operations, we built a dashboard that could scale with the business while helping managers focus on what mattered most: keeping their stores running smoothly and delivering on Zepto’s delivery promise.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aac167dd07b9" width="1" height="1" alt=""><hr><p><a href="https://blog.zepto.com/from-data-to-action-designing-for-faster-decisions-in-dark-store-operations-aac167dd07b9">From Data to Action: Designing for Faster Decisions in Delivery Hub Operations</a> was originally published in <a href="https://blog.zepto.com">Zepto TechXPress</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Optimizing SQL Warehouse Costs: Building Dataset Materialization in Apache Superset]]></title>
            <link>https://blog.zepto.com/optimizing-sql-warehouse-costs-building-dataset-materialization-in-apache-superset-2f5fa34682af?source=rss-47a9eb8a04a5------2</link>
            <guid isPermaLink="false">https://medium.com/p/2f5fa34682af</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[apache-superset]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Zepto Tech]]></dc:creator>
            <pubDate>Fri, 09 Oct 2026 09:51:21 GMT</pubDate>
            <atom:updated>2026-10-09T09:51:21.072Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*e2WEVhhiN7gW24tI.png" /></figure><p><strong>Same number of queries. Dramatically less warehouse runtime. So what changed?</strong></p><p><em>The chart above show the same dashboard’s runtime and query count over time. Query volume barely moved — but total runtime dropped dramatically. This is the story of how we taught Superset to precompute virtual datasets: safely, concurrently, and without ever serving stale data.</em></p><p><strong>TL;DR</strong></p><blockquote><strong><em>The Problem:</em></strong><em> 16K+ virtual datasets in Superset were re-executing the same heavy, 200-line CTE queries once per chart — even when charts shared the same underlying dataset — driving up warehouse spend.</em></blockquote><blockquote><strong><em>The Solution:</em></strong><em> We built transparent, on-demand </em><strong><em>Dataset Materialisation</em></strong><em> inside Apache Superset. It converts virtual dataset SQL into precomputed physical tables, without changing analyst workflows or serving stale data.</em></blockquote><blockquote><strong><em>The Impact:</em></strong><em> Total runtime, fetch latency, and compute costs all dropped sharply, while query volume and user experience stayed unchanged. Warehouse costs fell </em><strong><em>30–35% </em></strong><em>overall. Runtime dropped by up to </em><strong><em>60% on average, and 90–95% on our highest-frequency dashboards</em></strong><em>.</em></blockquote><h3>The Core Challenge: Same Expensive Work, Repeated</h3><p>Every time someone opens a dashboard in Superset, each chart sends a query to the warehouse.</p><ul><li>Charts can be built on virtual datasets — SQL queries stored in Superset rather than physical tables.</li><li>While the chart query may be a simple aggregation, the dataset SQL underneath can contain large scans, joins, and transformations.</li></ul><p>Consider a 200-line, CTE-heavy dataset query used by 12 charts. Superset can end up executing that expensive underlying query 12 times in parallel. The charts may ask different questions, but they repeatedly pay for the same scans and joins.</p><p><strong>The opportunity was therefore not to reduce the number of dashboard queries, but to reduce the amount of expensive work each query has to do.</strong></p><p><strong>At our scale</strong> — 16K+ virtual datasets on one major data source and hundreds of owners across business and product teams — manual SQL rewrites, physical-table creation, or owner-managed schedules were not going to scale.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LMB0_afpUF-qSkoj.png" /></figure><h3>Our Approach: What We Built</h3><p>To address this, we built the <strong>Dataset Materialisation</strong> feature in Superset: an <strong>enabled virtual dataset can be precomputed into a physical warehouse table</strong>. Once that table exists, charts query the precomputed results instead of re-running the original dataset SQL.</p><p>The important part is that this happens <strong>without changing how people use Superset</strong>. Analysts keep their SQL, dashboards keep their charts, and the system changes the execution path underneath.</p><p><strong>Before</strong></p><p><em>Chart → Virtual Dataset SQL (Heavy Scans/Joins) →Warehouse</em></p><p><strong>After</strong></p><p><em>Chart → Lightweight SELECT *</em> <em>→ Materialised Table on Warehouse</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3iOTtw6mr3SR0TQI.png" /></figure><blockquote><strong><em>How to enable</em></strong><em>: Materialisation is enabled through the dataset editor/API with a configurable refresh frequency. When the materialised result becomes stale, it is rebuilt from the original dataset SQL.</em></blockquote><h3>Let’s Deep Dive: Core Architecture &amp; Mechanisms</h3><h3>1. Keeping User Experience Unchanged</h3><p>There’s a subtle problem with transparently turning a virtual dataset into a physical table: the system needs to execute something different from what the analyst wrote, while the analyst still sees and edits their original SQL.</p><p>We keep three concepts separate:</p><ul><li><strong>Source SQL</strong>: The analyst’s original query — also the source for future rebuilds.</li><li><strong>Runtime Stub</strong>: A lightweight SELECT from the materialised table, used by charts after a successful build.</li><li><strong>Physical Table</strong>: The precomputed result stored in the warehouse.</li></ul><p>The editor and dataset views always resolve back to the Source SQL; only the chart execution path uses the Runtime Stub. <strong>The result is a transparent optimisation — the user works with the same SQL, while the warehouse does far less work.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*26DG0HClG1-K3tIF.png" /></figure><h3>2. Controlling When a Refresh Happens</h3><h3>We Only Refresh When Someone Needs the Data</h3><p>The obvious approach would be a periodic background job that refreshes every enabled dataset. But that means paying for refreshes even when nobody is using the dataset. <strong><em>Some dashboards may not be opened for days; others may be seasonal or heavily used only during certain periods.</em></strong></p><p>Hence, we use <strong>lazy refresh</strong>: when a dashboard is opened, Superset checks whether its materialised datasets are stale. If one has exceeded its configured refresh interval, the system triggers a rebuild. Unused datasets incur no refresh work.</p><blockquote><em>This </em><strong><em>ties</em></strong><em> </em><strong><em>refresh cost to actual usage</em></strong><em>, while keeping frequently used dashboards fresh.</em></blockquote><h3>Refresh Intent: Staleness ≠ Automatic Rebuild</h3><p>Superset already bypasses its application cache in some flows — but a cache bypass shouldn’t automatically mean rebuilding a warehouse table, or routine actions like auto-refresh or tab switches would trigger expensive CTAS runs.</p><p>We introduced an <strong>explicit refresh intent:</strong></p><ul><li>none: Never rebuild the warehouse table.</li><li>if stale: Rebuild only when the configured refresh interval has been exceeded.</li><li>force: Rebuild immediately, regardless of age.</li></ul><blockquote><strong><em>Lazy refresh</em></strong><em> decides when to check for staleness — </em><strong><em>Refresh Intent</em></strong><em> decides whether a check is allowed to trigger a rebuild at all.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bNHNRPQc3wRHSia4.png" /></figure><h3>3. Controlling Where a Refresh Happens</h3><p>When a chart hits a stale materialised dataset, a worker must trigger the warehouse rebuild, update metadata, and return fresh data on the same request. We evaluated three execution models for running this rebuild:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yBh4dLONsM2bp7_K.png" /></figure><p>We first tried a <strong>background Celery worker</strong>, but it fell short on two fronts:</p><ul><li><strong>Cost control</strong>: Coordinating on-demand triggers across Celery and web workers meant complex multi-layer locking.</li><li><strong>Latency</strong>: Users are already waiting when they open a dashboard. Offloading to Celery and polling for completion just adds round-trip overhead and queue-delay failures.</li></ul><blockquote><em>We settled on </em><strong><em>Inline Execution</em></strong><em> instead: the same web worker that detects staleness runs the CTAS itself, updates metadata, and returns fresh data — all in one request, bounded by a 5-minute timeout.</em></blockquote><blockquote><strong><em>Why it wins</em></strong><em>: no queue, no polling, no cross-system coordination. One request, one direct path.</em></blockquote><h3>4. Making Refreshes Safe Under Real Traffic</h3><p>Once refreshes happen on demand, concurrency becomes a problem. A dashboard can have several charts on the same stale dataset — if they hit different web workers within milliseconds, each could try to rebuild the same table independently.</p><p><strong>Without coordination:</strong></p><pre>Worker 1→ CTAS<br>Worker 2→ CTAS<br>Worker 3→ CTAS<br>Worker 4→ CTAS</pre><p>We needed exactly one refresh to win. Postgres enforces this with a <strong>partial unique index</strong>: only one RUNNING refresh row can exist per dataset. The first worker to insert it owns the rebuild - every other worker&#39;s insert fails, and they skip the rebuild.</p><p><strong>One request owns the rebuild; the rest do no duplicate work.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uHNzj3Vk4mqODiW0.png" /></figure><h3>Not Every Request Needs to Wait</h3><p>Losing requests behave differently depending on intent. When a dataset is stale, all sessions follow the same behaviour: requests poll for CTAS completion before reading the refreshed materialised table. For a force refresh, only requests with force intent wait for the rebuild, bounded by a timeout. Other user sessions continue serving the existing materialised table. A short dedup window prevents repeated force requests from triggering back-to-back rebuilds.</p><blockquote><em>Normal traffic stays responsive; users who explicitly want fresh data get a clear path to it.</em></blockquote><h3>5. Built Once, Warehouse-Agnostic</h3><p>The orchestration should not need to know how each warehouse replaces a physical table. We therefore isolated warehouse-specific rebuild behaviour behind a lightweight adapter.</p><ul><li><strong>Shared Laye</strong>r: owns freshness, concurrency, refresh intent and audit logging.</li><li><strong>Adapter</strong>: handles warehouse-specific concerns such as CTAS execution, atomic replacement or staging, table qualification, and session timeouts.</li></ul><p><strong>Result: the materialisation workflow is shared, while warehouse-specific mechanics stay behind a small interface.</strong></p><h3>6. The Trade-off We Accepted</h3><p>Force refresh runs two queries instead of one — a CTAS rebuild, then a SELECT against the fresh table. The added SELECT is lightweight, so the extra runtime is barely measurable, especially on frequently refreshed dashboards.</p><h3>The Impact</h3><p>We rolled out dataset materialisation gradually — starting with internal dashboards, then expanding to our highest-traffic ones.</p><p>The impact showed up across multiple warehouse metrics: <strong>total query runtime, data scanned, warehouse queue time, result fetch time, and dashboard/chart refresh latency all dropped</strong> — while query count stayed largely the same.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_EnhD5JjCfVSIpJG.png" /></figure><blockquote><em>Overall, warehouse costs dropped </em><strong><em>30–35%</em></strong><em>. Runtime fell by up to </em><strong><em>60% on average, and as much as 90–95% on our highest-frequency dashboards</em></strong><em>.</em></blockquote><p>The hard part was never the CREATE TABLE AS SELECT. It was making materialisation work transparently inside Superset while preserving existing behaviour and handling freshness, concurrency, and refresh races.</p><p>And from the user’s perspective, nothing changed.</p><p><strong>The same dashboards. The same SQL. Less work for the warehouse.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2f5fa34682af" width="1" height="1" alt=""><hr><p><a href="https://blog.zepto.com/optimizing-sql-warehouse-costs-building-dataset-materialization-in-apache-superset-2f5fa34682af">Optimizing SQL Warehouse Costs: Building Dataset Materialization in Apache Superset</a> was originally published in <a href="https://blog.zepto.com">Zepto TechXPress</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What Do You Add? | Recommending the right next item when intent is already there]]></title>
            <link>https://blog.zepto.com/what-do-you-add-recommending-the-right-next-item-when-intent-is-already-there-7a35a408b175?source=rss-47a9eb8a04a5------2</link>
            <guid isPermaLink="false">https://medium.com/p/7a35a408b175</guid>
            <category><![CDATA[recommendation-system]]></category>
            <category><![CDATA[product-management]]></category>
            <dc:creator><![CDATA[Zepto Tech]]></dc:creator>
            <pubDate>Wed, 07 Oct 2026 07:30:25 GMT</pubDate>
            <atom:updated>2026-10-07T07:30:25.567Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*K3CrrY5NbccFpZhd.png" /></figure><p>You have almost certainly done this.</p><p>Your cart is at ₹340. Bread, milk, a bag of onions, a shampoo you were running low on. You are one tap from paying. And then you notice the line at the top of the screen: <em>add ₹59 more to unlock ₹75 off.</em></p><p>So you do the arithmetic. If I add something for around ₹60 and save ₹75, the new item is effectively free. Better than free, actually. And it is not as though ₹60 of groceries goes to waste. There is always something the house needs.</p><p>That calculation is not a trick being played on you. Most of the time it is simply correct, and both sides come out ahead: you get more for less, and the order is worth more to us. This is one of the rare moments in commerce where the incentives genuinely point the same way.</p><p>And then you get stuck.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*za9_mK6uJxYVGgpC.png" /></figure><p>Because knowing you <em>want</em> to add ₹59 of something is a completely different problem from knowing <em>what</em>. So you go back to browsing. You search for a couple of things half-heartedly. You scroll a category you had no intention of scrolling. Sometimes you find something. Often you give up, pay at ₹340, and leave the ₹75 on the table, mildly annoyed but not enough to do anything about it.</p><p>This is the part worth sitting with, because it is unusual. <strong>The intent is already there.</strong> The user is not being persuaded, not being sold to, not resisting. They have decided they are willing to spend a little more and are actively looking for a reason to. What stands between them and that purchase is not desire. It is a decision they cannot make quickly enough for it to be worth making.</p><p>Worse, the moment is fragile. Someone on the cart page is in checkout mode. They have finished shopping and are trying to leave. Sending them back into search or a category listing breaks that momentum, and momentum, once broken, frequently does not come back. Every second of browsing is a second in which the whole order is at risk, not just the ₹59.</p><p>So the question is narrow and quite hard: in a few seconds, on a page the user is trying to exit, what do you put in front of them?</p><h3>Why the obvious answer isn’t enough</h3><p>The instinctive move is to find something priced at ₹59 and show it. That gets you part of the way, and no further.</p><p>The gap is not a fixed number. It is ₹59 for this user and ₹412 for the next, and the right answer looks nothing alike. Someone ₹59 away needs one well-chosen item. Someone ₹412 away is being asked to make a real decision about a real amount of money, and no quantity of ₹30 fillers gets them there in a way that feels like help rather than pressure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JbSVIHvBS_VpwX5S.png" /></figure><p>Price fit alone also produces absurd results. A ₹59 item the user would never buy is not a recommendation, it is noise. Their arithmetic does not stop running just because a coupon is dangling. Whatever we show still has to be something they actually want, at a price that still makes the maths work.</p><p>And the existing recommendation surfaces do not solve it either. Cart recommendations are usually built on similarity: you bought pasta, here is pasta sauce. That is a good answer to “what goes with this” and a poor answer to “what should I add to finish my order,” which is a different question with a price constraint attached to it.</p><p>What we needed was a recommendation that is <em>aware of where the user is standing</em>: how much is in the cart, how far the next milestone is, and what this particular person tends to buy. That awareness is the whole product.</p><h3>What belongs in the running</h3><p>Recommendations are built in two stages: assemble a pool of plausible candidates, then rank them. Our pool draws on three different ideas about what might help, and the reason for three rather than one is that each covers a failure of the others.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DCjJaA7JeqkcHjRq.png" /></figure><p><strong>Complementary items</strong> are the familiar kind of recommendation: things that go with what is already in the basket. These are easy to say yes to, because the user has already demonstrated they want the thing it goes with. On their own, though, they tend to cluster around whatever dominates the cart, which is a problem when the cart is three unrelated things.</p><p><strong>Habitual items</strong> come from the user’s own history. At checkout, the strongest predictor of what someone adds is what they always buy. These are the low-regret additions: no evaluation required, no risk of buying something that sits in the fridge untouched. Their weakness is that a person’s habits may contain nothing that fits the gap they happen to be facing.</p><p>One thing to note is that habit alone is not enough, though. The strongest signal that someone buys a product every week is also the reason not to show it to them the day after they bought it. A recommendation slot spent on rice somebody already has is a slot spent on nothing, and with only a handful of positions on the shelf, that is expensive. So repeat items are held back until enough of a user’s typical cycle has passed for the suggestion to be plausible again, and they return to contention as that moment approaches.</p><p><strong>Everyday top-ups</strong> cover the rest. Some categories get added at the last moment far more often than others, not because they pair with anything specific but because they are small, useful and easy to decide on. We lean on those categories deliberately, shaped to each user rather than shown blindly, so a top-up suggestion is still something that person plausibly wants.</p><p>Together the three balance the qualities the moment demands: relevance to this cart, confidence from familiarity, and the small useful things that make closing a gap painless.</p><h3>How we rank them</h3><p>This is where the problem becomes specific to the cart page, because ranking has to answer a question ordinary recommendation ranking never asks: <em>does the price of this item fit the gap?</em></p><p>Every candidate is scored on three things. Two are conventional: how well it fits the current cart, and how much this user tends to like it. The third is built for this surface, and it is where most of the thinking went.</p><p><strong>Milestone Delta Score </strong>measures the relationship between what a product costs and what the user still needs to spend. The scoring is deliberately asymmetric, because overshooting and undershooting fail in different ways and should not be punished the same.</p><p>An item that lands just above the gap is the ideal outcome. It closes the milestone in a single tap without spending much more than necessary, and the ranking chases that case hard.</p><p>An item that badly overshoots is penalised, and the more it overshoots the worse it does. Someone ₹59 short is not looking to spend ₹400. The wider the distance between what they need and what you are proposing, the more the recommendation reads as an upsell rather than a favour.</p><p>An item priced below the gap is the subtle case, and it is the one most systems get wrong by treating it as useless. It does not finish the job in one tap, so it cannot score perfectly. But plenty of these items are still genuinely good suggestions, and how good depends on how people actually buy them rather than on the shelf price alone. We score them accordingly, which means cheap items are neither dismissed nor allowed to crowd out better answers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9dsda8PT-4XgkV39.png" /></figure><p>The effect of all this is easiest to see where it overrules everything else. Imagine a user ₹50 short, and a loaf of bread that fits their cart perfectly and that they buy every single week. On any conventional recommendation ranking, bread wins outright. Here it comes second to a ₹55 pack of yoghurt they like slightly less, because that item ends the problem in one tap and the bread leaves them short. A recommendation engine with no sense of the gap would have got this backwards. That inversion is the entire reason the signal exists.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-QsNXzAr5WsPZDbm.png" /></figure><p>One last pass enforces variety, so that a single category cannot take over the shelf and every position is doing different work.</p><h3>Small gaps and large gaps are different problems</h3><p>The three signals are always present, but the balance between them shifts with the size of the gap. This turned out to matter more than almost anything else in the design.</p><p>When the gap is small, milestone delta score<em> </em>carries most of the weight. A user ₹50 from their milestone is in a solvable situation: there exists an item priced to close it that they also happen to want, and the job is simply to find that item. Relevance and personalisation still matter, but largely as tiebreakers among the many products that fit the price. Get this right and the interaction takes one tap.</p><p>When the gap is large, the balance deliberately tips the other way, toward relevance and personalisation, with milestone delta score pulled back.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ILoT24KNMtMTqcGE.png" /></figure><p>This is counterintuitive at first. A big gap needs a big-ticket item, so surely price fit should matter <em>more</em>? In practice the opposite holds. Leaning hard on price fit at a large gap surfaces expensive products, and expensive products require consideration. Nobody researches a ₹400 purchase from a widget on the cart page while trying to check out. Recommend one and you get neither the add nor the goodwill.</p><p>So past a certain point we stop optimising for closing the gap and start optimising for something the user genuinely wants. The realistic goal changes: not one tap to the finish line, but real progress with a product they are happy to own. Some of those users reach the milestone over two or three additions. Some never reach it and still bought something they wanted, which is not a failure.</p><p>Put simply, a small gap is a pricing puzzle and a large gap is a recommendation problem. Treating both the same way is the most natural mistake to make here, and it is the one the weighting exists to prevent.</p><h3>How it helped</h3><p>More users reached the milestone they were short of. The widget turned a question people were answering by browsing, or not answering at all, into something they could resolve in a tap or two without leaving the cart page, and a meaningfully larger share of them checked out with the coupon applied. That is the whole of what they wanted when they stopped to do the arithmetic.</p><p>Carts grew with them. Add-to-cart from the cart page rose, and because the items were ones people were glad to own rather than filler chosen to hit a number, average order value went up rather than simply shifting around. The additions were real demand that previously went unserved, not purchases moved from one surface to another.</p><p>Which is the unusual thing about this moment and the reason it was worth building for. The user wanted to spend a little more. We wanted the order to be worth a little more. Neither of us needed convincing, and all that stood in the way was a decision nobody had made easy. Answering the question they were already asking turned out to be the entire product.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7a35a408b175" width="1" height="1" alt=""><hr><p><a href="https://blog.zepto.com/what-do-you-add-recommending-the-right-next-item-when-intent-is-already-there-7a35a408b175">What Do You Add? | Recommending the right next item when intent is already there</a> was originally published in <a href="https://blog.zepto.com">Zepto TechXPress</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Zepto beyond Groceries | How the grocery app became a shopping destination]]></title>
            <link>https://blog.zepto.com/zepto-beyond-groceries-how-the-grocery-app-became-a-shopping-destination-aabdb373648a?source=rss-47a9eb8a04a5------2</link>
            <guid isPermaLink="false">https://medium.com/p/aabdb373648a</guid>
            <category><![CDATA[quick-commerce]]></category>
            <category><![CDATA[product-management]]></category>
            <category><![CDATA[customer-experience]]></category>
            <category><![CDATA[innovation]]></category>
            <category><![CDATA[product-engineering]]></category>
            <dc:creator><![CDATA[Zepto Tech]]></dc:creator>
            <pubDate>Tue, 29 Sep 2026 07:55:20 GMT</pubDate>
            <atom:updated>2026-09-29T07:55:20.469Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Hc6mQoDdP8c4uNgd.png" /></figure><p>Zepto started as the fastest way to buy groceries, and grocery shoppers arrive knowing what they want: milk, atta, the biscuits they buy every week. We built for that. Compact cards showed a picture, name, pack size and price. Dense listings let you scan more products per scroll, and search took you to something you could already name.</p><p>Then we started selling T-shirts, makeup, air fryers, toys, skincare and a lot more.</p><p>Customers opened the app for these without a decision made. A T-shirt buyer compares fabric, fit, colour and size. An air fryer buyer weighs capacity and specs across models. Someone shopping for skincare may still be figuring out which product is right for them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Es2NIacktRYn9oV-.png" /><figcaption>How different products require different attributes in the decision making</figcaption></figure><p>Getting these products to customers quickly wasn’t the only problem we were trying to solve. We also had to help them choose.</p><p>Fast delivery still mattered. But the decision now happened inside Zepto, and we had designed the app for people who showed up with it already made.</p><p><strong>That’s where the grocery experience stopped working.</strong></p><p>We spoke with roughly 300 customers across different persona and studied how people searched, browsed, carted and ordered in these categories over the last year. The gap showed up in six places, from discovering a category to receiving the order.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FcrCn9luwcA8o1wj.png" /><figcaption>Six questions from ~300 customer conversations, in the order a purchase happens</figcaption></figure><p>On their own these looked like separate problems: discovery, conversion, merchandising, cart behaviour. But they pointed to the same gap.</p><p>We had built an experience that was very good at helping customers find something they already understood. Now we were asking it to help them decide what to buy.</p><p>That became the product problem we spent much of the next year working on.</p><blockquote><em>Sizes came up in almost every apparel conversation. Customers said they didn’t want to open a product to find out whether it came in their size, because a T-shirt that doesn’t is not a product they need to look at. What we were treating as a detail on the product page, they were treating as the first filter.</em></blockquote><h3>What we built</h3><p>We solved for each question that user had. Most of them change parts of the app that grocery shoppers never needed: a place to browse new categories, product cards built for comparison, and prices that show offers before checkout. The overview below maps each question to what we built, and the sections that follow take them in the same order.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4xQORjDPuTnT3lUo.png" /><figcaption>How customer questions across the shopping journey shaped what we built.</figcaption></figure><h3>SuperMall</h3><p><em>New way to show customers what Zepto sells</em></p><p>You can’t search for something you don’t know exists. Search helps only customers who already know what to type. Most customers think of Zepto as a grocery app, so they don’t think to search it for a hair dryer or a Bluetooth speaker. We could stock the product, price it well and deliver it in ten minutes, and still never make the sale.</p><p>We built <strong>Super Mall </strong>to give Fashion, Electronics, Beauty and Home a place customers could browse into, with no search query needed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0MrINhMkxlRb0bQ4.png" /><figcaption>Super Mall — dedicated destination for the new categories</figcaption></figure><h3>Better Merchandising: Making the full range visible</h3><p>Customers who reached a category still saw only a small part of it. Merchandising widgets kept surfacing the same few products, so a customer returning on a later visit found the same shelf and assumed that was all we had.</p><p>Adding more products wouldn’t have fixed this, because new products would have stayed as hidden as the ones we already stocked. We changed which products customers saw instead:</p><ul><li>Occasion-led collections, such as gifting or festive picks</li><li>Age-led discovery for toys and kids’ products</li><li>Widgets that swap out products a customer has already seen or skipped</li></ul><p>The measure we cared about afterwards wasn’t how many products we had listed. It was how many of them customers actually reached. ATC share spread across a noticeably wider set of products instead of concentrating on the same few, and the share coming from our top 20 products fell meaningfully.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WZN1hpkfThYbnmmt.png" /><figcaption>Same familiar experience but with fresh products every few sessions</figcaption></figure><h3>Category led search</h3><p><em>Redesigning search to help customers decide</em></p><p>A customer who types “garbage bag” knows what they want, and a dense 3×3 grid gets them to it faster. A customer who types “T-shirt” is comparing colour, cut, material and fit, which nine thumbnails show poorly. In research, apparel shoppers told us they saw nine T-shirts on the screen and couldn’t tell one from the next. Some didn’t want to open a T-shirt to learn it wasn’t in their size.</p><p><strong>Choosing the grid for each query</strong></p><p>Home holds both “bedsheet” and “garbage bag.” Shoppers judge a bedsheet by how it looks and pick a garbage bag by size, so we couldn’t give Home a single layout. We tagged high-volume queries by intent instead, using signals such as how often customers opened product pages before adding to cart. We moved queries like “T-shirt” to a 2×2 grid with larger images, and queries like “garbage bag” kept the 3×3 grid.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HEofyZsV_5uvuI-x.png" /><figcaption>Redesigned search layout, and the product cards</figcaption></figure><p><strong>Shortlisting details on the card</strong></p><p>A grocery card works with an image, name, pack size and price. A sunscreen card needs SPF and skin type. Earbuds need playback time. Apparel needs material and available sizes.</p><p>Sizes came up in almost every apparel conversation. Customers wanted to see which sizes were in stock before opening a product. If the T-shirt doesn’t come in their size, they had to tap into the page to learn that, and it wasted their time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*oE9Iw-eeCf_kDUVi.png" /></figure><p>Most of these details already existed on the product page, so comparing ten products meant opening ten pages.</p><p>The bigger cards needed images in a new aspect ratio. Reshooting thousands of products would have held up the launch, so we used AI to extend the photos we already had.</p><p>We picked these using research / filter usage / most-viewed product page attributes, and left the rest on the product page. A phone has dozens of specs, and we left the rest on the product page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zoHRHw8-UynPm5p3.png" /><figcaption>Making the bigger purchase feel affordable via Best Price and EMI</figcaption></figure><h3>Showing the price customers pay</h3><p>On a grocery card, the listed price is close to what you pay. That stopped being true for higher-ticket products. Product discounts, bank offers and coupons often brought the price down, but customers saw the lower number only after they added the product to cart.</p><p>The card is where a customer decides whether a product deserves a closer look. Someone who saw ₹1,000 and thought it too expensive scrolled past before any offer appeared. A customer comparing five air fryers compared them on prices they wouldn’t end up paying.</p><p>We followed a two-pronged approach to solve this problem:</p><ul><li><strong>Best Price: </strong>Cards in browse and search show the lowest price a customer can get after applicable offers, and name the offer that unlocks it. If a product is listed at ₹1,000 and a bank offer takes it to ₹900, the card shows ₹900 and tells the customer how to get it. Customers now compare products on the price they can pay, instead of discovering the discount at checkout.</li><li><strong>EMI on cards and product pages: </strong>Customers are used to seeing EMI and no-cost EMI when they buy expensive products online. Most didn’t expect it on Zepto, so they didn’t look for it, even when it was available. The few who found it did so at payment, if they got that far. We now show the monthly amount on cards and product pages. A customer looking at a ₹10,000 purchase also sees ₹656 a month, and can judge whether it fits their monthly budget before ruling the product out.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Jpxcm238zcRF2cCM.png" /><figcaption>Making the bigger purchase feel affordable via Best Price and EMI</figcaption></figure><blockquote><em>The offers were always there. The customer just had to work to find them, and the point at which they gave up was earlier than the point at which we were showing them.</em></blockquote><h3>Letting customers decide over several visits</h3><p>Customers kept products they weren’t ready to buy in the cart, so they could find them again later. When they needed the cart for an immediate order, they removed those products and lost track of them.</p><p>For purchases that take more than one visit, we built:</p><ul><li><strong>Wishlist.</strong> Customers save a product without adding it to the cart. The cart stays clear for today’s order, and the saved product is still there after checkout.</li><li><strong>Recently Viewed and Continue Browsing.</strong> A customer who spends ten minutes narrowing down four air fryers and closes the app can pick up where they left off three days later, without searching again.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xCCyg7MRRMliSltc.png" /></figure><p>Customers who wishlisted converted about 2x as often as those who didn’t. We don’t credit Wishlist with all of that lift, since customers who save products are more interested to begin with. The stronger signal is how they saved: about 40% saved three or more products, and we read that as customers comparing options across visits.</p><h3>And then there was Trust</h3><p>A customer buying a ₹3,000 pair of earbuds takes on more risk than one buying milk. They need confidence at two points: when they choose the product, and when it arrives.</p><p>Before the order, we built <strong>Ratings</strong>. For an unfamiliar product, seeing what other customers bought and how they rated it helps make the choice easier.</p><p>But trust did not end at checkout.</p><p>With a higher-ticket purchase, customers also wanted confidence that what arrived was actually what they had ordered and expected.</p><p><strong>Open Box Delivery</strong> moved that verification to the doorstep. Customers open and inspect the product at the doorstep before accepting the order. If it’s damaged or doesn’t match what they ordered, they catch it there, and don’t have to accept it and then start a return.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cHN3FN_roborDV1Y.png" /><figcaption>Removing the risk of via ratings, and open box delivery</figcaption></figure><p>We built each of these for a step in the same purchase. Take a customer shopping for a T-shirt. She has to know Zepto sells T-shirts and see enough of the range to find one she likes. She needs to tell the options apart, see the price she’ll actually pay, and come back later to decide. Then she needs to trust what she orders and what arrives. We spent this year building each of those steps.</p><h3>Overall Impact</h3><p>All the above changes has significantly improved the conversion funnel, and the penetration of these categories on Zepto. Over the past year, search to add-to-cart across these categories rose about 18%. First-time category buyers, the group furthest behind repeat buyers, gained the most.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rF8VMxRKfJfxilHS.png" /><figcaption>Search to ATC Funnel Impact</figcaption></figure><h3>What we’re working on next</h3><p>Most of this year went into establishing that these categories need a different experience. The next stretch is about depth.</p><p><strong>Search that adapts to the customer.</strong></p><p>Search picks a layout based on the query. The same query plays out differently by customer: one who has bought in the category scans and picks, while a first-time buyer scrolls further, rewrites the query and leaves more often. We’re building search that adjusts to how well the customer knows the category.</p><p><strong>Getting customers to buy the whole category from us</strong></p><p>Speed built quick commerce, but it doesn’t hold back Electronics or Fashion. A customer buying a weekend gift isn’t counting minutes, and ten-minute TV delivery doesn’t help if nobody is home. We’re working out what else customers need to buy these categories from us by default, such as scheduled delivery, installation, insurance.</p><p><strong>Winning the considered purchase</strong></p><p>Customers buy in these categories rarely, so we grow more through bigger orders than more orders. Many customers buy an accessory from us and the device elsewhere. We’re working on earning the purchase they think hardest about.</p><p><strong>Reviews on the product page</strong></p><p>Ratings tell a customer how buyers scored a product. Before a first purchase, they also want the reasons: whether a T-shirt’s fabric pills after a wash, or how earbuds sound on calls. We’re adding written reviews to product pages, so customers can read what buyers liked and what disappointed them.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aabdb373648a" width="1" height="1" alt=""><hr><p><a href="https://blog.zepto.com/zepto-beyond-groceries-how-the-grocery-app-became-a-shopping-destination-aabdb373648a">Zepto beyond Groceries | How the grocery app became a shopping destination</a> was originally published in <a href="https://blog.zepto.com">Zepto TechXPress</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Right People, Right Shift, Right Time: Optimizing Workforce Planning at Zepto]]></title>
            <link>https://blog.zepto.com/right-people-right-shift-right-time-optimizing-workforce-planning-at-zepto-11a8e706c7c3?source=rss-47a9eb8a04a5------2</link>
            <guid isPermaLink="false">https://medium.com/p/11a8e706c7c3</guid>
            <category><![CDATA[optimization]]></category>
            <category><![CDATA[forecasting]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[workforce-planning]]></category>
            <category><![CDATA[linear-programming]]></category>
            <dc:creator><![CDATA[Zepto Tech]]></dc:creator>
            <pubDate>Thu, 03 Sep 2026 07:33:58 GMT</pubDate>
            <atom:updated>2026-09-03T07:33:58.157Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5Rs1bChF8TastE3s.png" /></figure><p>In quick commerce, every minute matters and so does having the right number of people on the floor at the right time.</p><p>At Zepto, workforce planning across a large and rapidly evolving network of dark stores and superstores across India is anything but static. Order volumes change sharply through the day, inbound trucks bringing in fresh inventory from the warehouse have their own variability, and the same store teams must balance order fulfilment with putaway, inventory audits, expiry checks and other operational tasks. The challenge isn’t simply deciding how many people a store needs; it’s determining when they are needed, what shifts they should work, and how those shifts should be assigned to people while respecting real-world constraints.</p><p>For a long time, much of this complexity was solved locally by Store Managers using spreadsheets, intuition and experience. That may work for an individual store, but at Zepto’s scale, we needed something fundamentally different: a centralized workforce planning system that could translate fluctuating demand into optimized shifts and rosters across the network.</p><p>This is the story of how we built that system combining forecasting, mathematical optimization and production engineering to turn a deeply operational problem into a scalable decision engine.</p><h3>The Need for a Centralized Solution</h3><p>The complexity becomes clearer when we look at what a typical dark store team actually does.</p><p>Picking and packing customer orders is only one part of the workload. Throughout the day, the same team also receives inventory from mother hubs and vendors, moves products from staging areas onto shelves, and performs critical inventory hygiene activities such as audits, expiry checks and removal of damaged or defective products.</p><p>The challenge is that <strong>these workloads don’t arrive uniformly through the day</strong>. Customer orders have distinct peaks, inbound inventory has its own arrival patterns, and several background activities have flexibility in when they can be performed. Staffing for the peak keeps fulfilment well covered, but can leave excess capacity during leaner hours. Staffing closer to the average creates the opposite problem, with too few people when demand spikes.</p><p>Historically, Store Managers navigated these trade-offs using their experience and local context. But as the network grew, we were effectively solving the <strong>same optimization problem independently across every store</strong>, with different assumptions and no common planning framework.</p><p>We wanted to turn this around. Instead of asking each Store Manager to determine the right manpower plan, we wanted a centralized system that could answer three questions:</p><ul><li><strong>How much manpower does each store need, and when?</strong></li><li><strong>What combination of shifts best matches that workload?</strong></li><li><strong>How do we assign those shifts to actual people while respecting availability, preferences and operational constraints?</strong></li></ul><p>These questions became the foundation of our <strong>workforce optimization engine</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Diasjarr2JwpYHpX.png" /></figure><h3>Decoupling Complexity: A Two-Step Architecture</h3><p>Once we knew what the system needed to solve, the next question was how to formulate it.</p><p>At first glance, workforce planning looks like a standard optimization problem. Given the expected workload for every hour, determine the shifts required and assign employees to those shifts. In practice, however, these two decisions are tightly coupled.</p><p>Every additional shift option increases the possible ways in which employees can be rostered. At the same time, each employee brings their own constraints around availability, weekly offs, shift timings and other operational requirements. Solving shift design and individual rostering together therefore creates a large combinatorial optimization problem that becomes increasingly expensive as the number of employees, shifts and constraints grows.</p><p>Rather than solving everything in one monolithic optimization model, we <strong>decoupled the problem into two sequential stages</strong>:</p><p><strong>Step 1: Shift Generation<br></strong>Determine the optimal set of shifts and the number of people required in each shift, based on the store’s expected workload and operational constraints.</p><p><strong>Step 2: Personnel Rostering<br></strong>Map those shift requirements to individual employees while respecting availability, preferences, weekly offs and other ground-level constraints.</p><p>This separation gave us an important architectural advantage. The first stage could focus purely on <strong>matching manpower capacity to workload</strong>, while the second could focus on <strong>turning that ideal plan into a roster that works for real people</strong>.</p><p>It also allowed us to choose the right optimization technique for each part of the problem.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZQmQQm3z_EAC3Exk.png" /></figure><h3>Step 1: Shift Generation via MILP</h3><p>The first stage answers a fundamental question: <strong>given the workload expected at a store throughout the day, what combination of shifts gives us the right manpower at the right time?</strong></p><p>We formulate this as a <strong>Mixed-Integer Linear Programming (MILP)</strong> problem and solve it independently for each dark store using the open-source <a href="https://highs.dev/"><strong>HiGHS solver</strong></a>. The optimizer takes the expected hourly workload as an input and generates the shifts and corresponding headcount required to complete that work with the available manpower.</p><p>But simply matching headcount to workload is not enough. A mathematically efficient shift plan also needs to be practical for teams to operate on the ground. We therefore incorporated a set of operational constraints into the model:</p><ul><li><strong>Work Completion:</strong> At every hour, sufficient manpower capacity must be available to complete the expected workload.</li><li><strong>Weekly Planning:</strong> Shift requirements are generated for the entire week rather than optimizing each day independently.</li><li><strong>Schedule Regularity:</strong> To keep schedules predictable and easy to manage on the ground, the optimizer maintains consistent shift templates across the week. <strong>Weekdays follow one shift template, while weekends follow another</strong>, allowing the plan to reflect differences between weekday and weekend demand patterns.</li><li><strong>Balancing Fulfilment and Store Operations:</strong> The optimizer must cover fluctuating order peaks while ensuring activities such as putaway and inventory hygiene can also be completed without affecting customer fulfilment.</li><li><strong>Shift Handover:</strong> Every shift requires a minimum staffing level, with sufficient overlap between consecutive shifts to ensure smooth handovers.</li></ul><p>This formulation allows the optimizer to do more than simply <strong>staff for the busiest hour</strong>. It finds a combination of shifts that follows the store’s workload curve through the day while using available manpower efficiently.</p><h3>Step 2: Preference-Driven Personnel Rostering</h3><p>Once the optimal shift plan is generated, the next challenge is to turn it into a roster for actual employees.</p><p>This introduces a different set of constraints. Employees need weekly offs, sufficient rest between shifts, and schedules that account for their availability and shift preferences. Trying to incorporate all of these individual-level considerations directly into shift generation would bring back much of the complexity we deliberately separated in the first step.</p><p>We therefore solve personnel rostering in <strong>two stages</strong>.</p><h3>Structural Allocation</h3><p>We first use <a href="https://developers.google.com/optimization/cp/cp_solver">Google OR-Tools’</a> <strong>CP-SAT</strong> to construct a weekly roster using synthetic employee slots such as P1, P2, P3 and so on. At this stage, we are not concerned with which actual employee fills a slot. The objective is to ensure that the roster is structurally feasible and provides the required headcount for every shift. We used CP-SAT here over HiGHS as it performs much better when all the variables are integers.</p><p>The model also incorporates fundamental rostering constraints, including a <strong>weekly off for every employee and a minimum 12-hour gap between consecutive shifts</strong>.</p><p>The output is essentially a set of feasible weekly work patterns that now need to be matched to real people.</p><h3>Preference-Driven Matching</h3><p>This is where employee preferences enter the picture.</p><p>We formulate the assignment as a <strong>bipartite matching problem using min-cost max-flow</strong>. On one side are the actual employees available at the store, and on the other are the synthetic roster slots generated in the previous stage.</p><p>Each possible employee-to-roster assignment carries a penalty based on how far it deviates from the employee’s preferences, including <strong>preferred shift start times and weekly offs</strong>. The algorithm then finds the assignment that fills the required roster while minimising the total preference penalty across employees.</p><p>This gives us an important separation of concerns:</p><p><strong>CP-SAT ensures that the roster works operationally. Min-cost max-flow helps ensure that the roster works for the people assigned to it.</strong></p><p>With the core optimization architecture in place, however, we encountered another challenge. <strong>A mathematically optimal roster does not automatically make it operationally viable.</strong></p><p>That meant bringing a much richer set of ground realities into the model.</p><h3>Tuning the Math to Ground Realities</h3><p>Building the optimization models was only part of the challenge. <strong>Making their recommendations work on the ground was where things got interesting.</strong></p><p>A solver operates on the constraints we give it. A dark store, however, operates in the real world. Employees depend on public transport, students have classes, labor availability varies across locations, and different store formats have fundamentally different operating characteristics.</p><p>As we scaled the system, we continuously translated these ground realities into constraints and configurations that the optimizer could understand.</p><h3>Building for an Inclusive Workforce</h3><p>Creating a safe and inclusive workplace is an important consideration in how we design our rosters. Across our network, there are stores where women form a significant share of the workforce, including <strong>100+ dark stores where women outnumber men</strong>.</p><p>We incorporated appropriate rostering constraints into the optimization framework to account for the safety and operational requirements of women employees while generating shift plans.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*I5NILzoZ5yTOZRRQ.png" /></figure><h3>Aligning Shifts with Public Transport</h3><p>A shift is useful only if employees can actually get to the store.</p><p>We found that early-morning and late-night shifts could remain unfilled in locations where public transport was unavailable during those hours. To account for this, we introduced <strong>store-specific blackout windows</strong> that prevent the optimizer from recommending shifts during periods when commuting options are limited.</p><p>Since these constraints differ by location, Store Managers can provide local inputs that are incorporated directly into the planning process.</p><h3>Working Around Academic Schedules</h3><p>College students (over the legal working age), form an important part of our flexible workforce, particularly in helping stores manage evening demand. Their availability, however, needs to work around academic schedules.</p><p>We introduced student-specific shift constraints that allow the optimizer to schedule them outside class hours, creating predictable work patterns while still utilizing this flexible capacity where it is most valuable.</p><h3>Accounting for Inbound Variability</h3><p>Not every workload has a predictable arrival time.</p><p>Supply trucks from mother hubs can be delayed by traffic and other operational factors, making rigid inbound windows difficult to plan around. Instead of assuming a fixed arrival pattern, we introduced <strong>dynamic workload smoothing</strong> that distributes putaway activities across lower-volume outbound periods.</p><p>This helps flatten workload peaks while protecting manpower capacity during periods of high customer demand.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JT663YhtyN7y0Fjk.png" /></figure><h3>Planning Under Manpower Constraints</h3><p>The ideal manpower requirement is not always the manpower available on the ground.</p><p>This is particularly relevant in rapidly expanding markets where local labor availability can temporarily fall below what the unconstrained optimizer recommends. We therefore introduced active manpower upper bounds and a shortage-aware optimization formulation.</p><p>When manpower is constrained, the optimizer prioritizes available capacity around peak customer fulfilment periods, rather than spreading the shortage uniformly throughout the day.</p><h3>Extending the Model to Superstores</h3><p>As Zepto expanded its superstore network, the same framework had to accommodate a different operating model.</p><p>Superstores carry a larger assortment and serve a wider delivery radius. Their demand and replenishment patterns make shelf availability a more continuous activity throughout the day, rather than something concentrated within a narrow window.</p><p>We adapted the framework with <strong>store-format-specific processing speeds, task durations and input configurations</strong>, allowing the same optimization architecture to support these larger-format stores.</p><p>The result was not one rigid optimization model, but a configurable planning framework that could adapt to the operating realities of individual stores while retaining a common optimization backbone across the network.</p><h3>Closing the Loop: From Optimization to Execution</h3><p>Generating an optimized roster is only useful if it can be executed on the ground.</p><p>Store operations are inherently dynamic. Unexpected absenteeism, a local event, changes in transport availability, or other operational disruptions can quickly make even a well-optimized weekly plan outdated. Rather than treating the generated roster as a static output, we built a <strong>closed-loop planning system</strong> that allows ground teams to respond to these changes while maintaining central visibility.</p><h3>Bringing Local Context into Planning</h3><p>Store Managers often have context that no centralized system can know in advance. Before the optimizer runs, they can provide store-specific inputs such as blackout hours and local transport constraints. These inputs become part of the optimization itself, allowing the generated roster to reflect local operating conditions.</p><p>This gives us the benefits of centralized optimization without losing valuable context from teams on the ground.</p><h3>Re-optimizing When Reality Changes</h3><p>Not every disruption can be anticipated during weekly planning.</p><p>When conditions change during the week, Store Managers can update relevant constraints through our HRMS portal and <strong>trigger re-optimization on demand</strong>. The system recalculates the plan and generates an updated roster within minutes, rather than requiring managers to manually rebuild schedules.</p><h3>Building Guardrails Around the Optimizer</h3><p>Giving local teams the ability to provide inputs also introduces another engineering challenge: <strong>those inputs need to be reliable.</strong></p><p>We built automated schema and validation checks around the planning pipeline to catch incorrectly formatted inputs and invalid parameter values before they affect optimization runs. Alerts integrated with Slack pinpoint failures to our engineering and operations teams, helping identify and resolve pipeline issues quickly.</p><h3>Measuring What Happens on the Ground</h3><p>Finally, we close the loop by comparing the optimizer’s recommendations with what actually happens at each store.</p><p>Central operations teams monitor <strong>generated plans, planned versus actual headcount, and shift-level adherence</strong> through live dashboards. This gives us visibility into where recommendations are being followed, where they are not, and where the model or operational process may need further refinement.</p><p>Today, this feedback loop has helped us achieve <strong>85%+ adherence across the network</strong>, while still giving Store Managers the flexibility to respond when local conditions change.</p><p>The result is a system that is neither completely centralized nor completely decentralized. <strong>Optimization provides the common planning backbone, while local inputs and continuous re-optimization keep the plans grounded in operational reality.</strong></p><h3>The Impact</h3><p>What started as an effort to improve manpower planning has evolved into a centralized workforce optimization system operating across Zepto’s network of dark stores and superstores.</p><p>The system generates weekly shift plans and rosters automatically, while continuously incorporating operational constraints and changing ground realities. More importantly, it has translated mathematical optimization into measurable impact:</p><ul><li>85%+ Adherence: Recommended shift plans see more than 85% adherence across the network, demonstrating that optimized schedules can work at scale without losing touch with ground realities.</li><li>Workforce Efficiency: The system helped optimize approximately 2.5% of the dark store workforce and 15% of the superstore workforce, creating additional capacity to absorb order growth more efficiently.</li><li>Significant Financial Impact:<strong> </strong>Optimizing manpower allocation across stores has <strong>unlocked substantial annualized savings</strong>, driven by better alignment of staffing with workload and reduced idle capacity.</li><li>One Planning Framework Across the Network: What was previously solved independently by Store Managers is now powered by a common, data-driven planning framework, while retaining the flexibility needed for local operating conditions.</li></ul><p>Workforce planning is a classic Operations Research problem. But deploying it in quick commerce required much more than formulating an optimization model. It meant combining mathematical optimization, scalable engineering, operational context and continuous feedback from the ground.</p><p>At Zepto, this is how we think about applied Data Science and Management Science: not as models that produce recommendations in isolation, but as decision systems embedded directly into how the business operates.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=11a8e706c7c3" width="1" height="1" alt=""><hr><p><a href="https://blog.zepto.com/right-people-right-shift-right-time-optimizing-workforce-planning-at-zepto-11a8e706c7c3">Right People, Right Shift, Right Time: Optimizing Workforce Planning at Zepto</a> was originally published in <a href="https://blog.zepto.com">Zepto TechXPress</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Route IQ: How We Reconstruct Delivery Paths from Noisy GPS at Scale]]></title>
            <link>https://blog.zepto.com/route-iq-how-we-reconstruct-delivery-paths-from-noisy-gps-at-scale-6898d3260ed2?source=rss-47a9eb8a04a5------2</link>
            <guid isPermaLink="false">https://medium.com/p/6898d3260ed2</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[geospatial]]></category>
            <category><![CDATA[last-mile-delivery]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[route-optimization]]></category>
            <dc:creator><![CDATA[Zepto Tech]]></dc:creator>
            <pubDate>Mon, 24 Aug 2026 06:40:30 GMT</pubDate>
            <atom:updated>2026-08-24T06:40:30.304Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZSpyNZxoTzq1cmgD.png" /></figure><p>Every day, millions of customers watch a tiny vehicle icon glide across their screen, a pristine purple line tracing the journey from store to doorstep.<br>Behind that polished UI lies a spatial nightmare.</p><p>In last-mile delivery, GPS signals degrade in narrow alleys, flyovers, and dense tech parks. Combined with low-end hardware and aggressive battery-saver modes, raw GPS data doesn’t trace a route it spatters coordinates. Pings lag, vanish for minutes, bounce off glass facades, or inexplicably teleport a rider three blocks into a lake.</p><p>For a data science team, this isn’t a visual glitch; it’s an operational vulnerability. Bad spatial data distorts true distance, corrupts unit economics, and feeds downstream machine learning models false realities.</p><p>Route IQ is our solution: an end-to-end spatial pipeline built to transform millions of chaotic, teleporting GPS traces into high-fidelity ground truth.<br>In this blog, we’ll walk through our approach to turning noisy GPS pings into meaningful rider routes:</p><ol><li><strong>The Need:</strong> Why raw GPS data isn’t enough to understand last-mile rider behaviour.</li><li><strong>Cleaning the Mess:</strong> Filtering noisy, jittery pings into smooth, believable traces.</li><li><strong>Map Matching:</strong> Snapping cleaned pings to the roads riders actually travel using OSRM and confidence scoring using Dynamic Time Warping.</li><li><strong>Learning the Consensus:</strong> Identifying the routes riders typically take from a store to a customer using H3 level Hidden Markov Model, Greedy Walk and Dijkstra fallback.</li></ol><h3>1. Why GPS Data Isn’t Enough for Last-Mile Intelligence?</h3><p>A rider’s phone emits a GPS ping every few seconds, painting a breadcrumb trail across the city. Before processing, a typical trace looks like a toddler’s scribble: pings inside buildings across the street, sudden 400-meter teleports forward and back, or a cluster of 50 pings jittering around a single red light, adding phantom distance to the trip.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/0*apnjlaKvVptfLkai.gif" /><figcaption>A real GPS trace before processing — pings jumping across roads, creating phantom segments</figcaption></figure><p>Computing standard Haversine distance over these raw coordinates frequently yields estimates <strong>30% to 40% off</strong> from the actual distance traveled. Because rider distance is a foundational metric for compensation, ETA modeling, and geofence optimization, this inaccuracy cascades across the entire business.</p><p>Route IQ answers four critical operational questions:</p><ol><li><strong>What route did the rider actually take?</strong> Filtering noise and resolving confidence gaps where standard map-matching fails.</li><li><strong>Is the rider stalling somewhere?</strong> Disentangling signal degradation from actual stops at traffic signals, crossings, or vendors.</li><li><strong>What route <em>should</em> the rider have taken?</strong> Learning historical “consensus paths” that riders naturally prefer for specific origin-destination pairs.</li><li><strong>Did this rider deviate? </strong>Scoring real-time divergence against learned consensus paths to catch distance padding or identify better shortcuts.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*h-grzKgdTGpZ7V0a.png" /><figcaption>End-to-end pipeline architecture: 6 stages from raw GPS pings to deviation scoring</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6pumaYWM3oB4McHg.png" /><figcaption>Before and after passing through the pipeline</figcaption></figure><h3>2. Cleaning the Mess: A Multi-Factor Approach to GPS Noise Removal</h3><p>A rider’s phone might report their location with 5-meter accuracy one second and 200-meter accuracy the next, depending on satellite visibility, urban canyon effects, whether the phone is in a pocket, or whether the device just woke from a power-saving sleep cycle. The result is a coordinate trail that looks less like a route and more like a toddler’s crayon drawing over a city map.</p><p>The naive fix “just remove points that are too far from the previous one” fails quickly. A single noisy ping between two good ones creates two bad segments, not one. Remove the wrong point and we would have deleted real movement while keeping the outlier. We needed something smarter.</p><h3>Joint Anomaly Scoring</h3><p>Instead of binary outlier detection, we compute a <strong>joint anomaly score</strong> for every ping as a weighted combination of normalised kinematic and spatial features. The goal is to capture different kinds of GPS anomalies without relying on any single rule:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*o93sT68Ppxsg8pb1.png" /></figure><p>where wₖ represents empirically derived feature weights and fₖ(pᵢ) represents normalised feature functions capturing acceleration anomaly, bearing variance, and spatial deviation from the local median centroid. Each feature targets a distinct kinematic or spatial violation:</p><ul><li><strong>Speed anomaly:</strong> Detects implausible speeds between consecutive pings.</li><li><strong>Acceleration anomaly:</strong> Flags sudden, physically unrealistic changes in velocity.</li><li><strong>Bearing anomaly:</strong> Identifies sharp directional changes, while accounting for the fact that turns at low speeds can be legitimate.</li><li><strong>Interpolation deviation:</strong> Measures how far a ping deviates from the expected position between its neighbouring pings.</li><li><strong>Spatial outlier:</strong> Compares the ping against the local spatial median to catch isolated jumps that kinematic checks may miss.</li><li><strong>Edge adjustment:</strong> Adds a small correction for the first and last pings, which have less neighbour context.</li></ul><p>The weights wₖ were tuned empirically. Speed features dominate because they provide the highest signal-to-noise ratio. A 200 km/h implied speed over a 3-second interval is unambiguous regardless of other features.</p><h3>Iterative Greedy Removal</h3><p>With anomaly scores computed, the algorithm enters a loop:</p><ol><li>Check if any segment in the trip exceeds vₘₐₓ = 60 km/h</li><li>If yes, compute A(pᵢ) for all pings</li><li>Remove the single ping with the highest anomaly score: argmaxᵢ A(pᵢ)</li><li>Recalculate all features and repeat</li></ol><p>The greedy one-ping-at-a-time removal is critical: removing a ping changes its neighbours’ features, potentially making a normal ping anomalous or a suspicious one normal. Batch removal misses these cascading effects. The loop stops when no segment exceeds vₘₐₓ or fewer than 2 points remain.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gFri2synEkVjrvpe.gif" /><figcaption>Iterative ping removal: the worst-scoring ping (red) is removed each iteration until all segments are below 60 km/h</figcaption></figure><h3>Anchoring the Start: The Artificial First Ping</h3><p>Before cleaning, the pipeline adds a <strong>synthetic first ping at the store’s GPS coordinates</strong> with the dispatch timestamp. Since the first real GPS fix can arrive seconds later and hundreds of meters away, this anchor provides a reliable starting point for the trip.</p><h3>Handling Stationary Riders: Stall Detection</h3><p>After bad pings are removed, a second pass detects <strong>stalls. </strong>We identify periods where the rider stays within 30m of a reference point for 240 seconds despite GPS jitter. These can generate dozens of noisy pings and add 100–300 phantom meters to trip distance. Replacing them with a single centroid removes this inflation while preserving the actual location.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2eowg9oLrpJGK0hD.png" /><figcaption>Before and after stall detection and cleaning</figcaption></figure><h3>Interpolation &amp; Kalman Smoothing</h3><p>After cleaning and stall compression, sparse temporal gaps (&gt;4 seconds) are filled with linearly interpolated points. Then a <strong>Kalman filter</strong> smooths the entire trajectory. The state vector is four-dimensional — [lat, lon, v_lat, v_lon] — modelling the rider as a constant-velocity particle with Gaussian process noise.</p><p>Why Kalman over a simple moving average? Kalman naturally handles irregular time gaps (the state transition scales with dt) and the velocity component provides momentum. The filter &quot;expects&quot; the rider to keep moving in roughly the same direction, which suppresses residual jitter that survives the earlier cleaning stages.</p><h3>3. OSRM Map-Matching &amp; Confidence Scoring</h3><p>A cleaned GPS trace is a significant improvement over raw pings, but it still floats in free space. The coordinates don’t know about roads, one-way streets, or the fact that the rider couldn’t have cut through that apartment complex. The next step is <strong>map-matching</strong>: aligning the GPS trace to the actual road network using OSRM (Open Source Routing Machine).</p><h3>Handling Scale</h3><p>At millions of trips per pipeline run, OSRM calls are the primary bottleneck. Two mechanisms keep this tractable:</p><ul><li><strong>Smart downsampling:</strong> OSRM’s match endpoint has a practical limit of ~100 coordinate pairs per request. Trips with more pings are evenly sampled while keeping first and last points. The step size adapts to trip length.</li><li><strong>Distributed rate limiting:</strong> The pipeline runs on Spark with multiple executors hitting OSRM in parallel. The entire stage uses Spark’s mapInPandas. Each partition processes API calls in parallel, results write directly to Delta. No data collected to the driver.</li></ul><h3>Best Match Selection</h3><p>OSRM can return multiple road-snapped matchings for a trip, sometimes as fragmented routes. Rather than choosing the highest-confidence match, we select the one whose total distance is closest to the <strong>haversine distance from the cleaned pings</strong>. This helps avoid partial matches.</p><h3>The Final Confidence Score</h3><p>We needed a holistic confidence label — not just “how confident is OSRM?” but “how confident are <em>we</em> that the final route and distance for this trip are correct?” The Final Confidence Score is a categorical label (High / Medium / Low) computed from four independent signals, each catching a different failure mode:</p><p><strong>Signal 1: OSRM Confidence Filtering.</strong> We discard OSRM matchings with confidence <strong>&lt; 0.05</strong> or anomalous values <strong>&gt; 1</strong>, treating them as unreliable. These trips fall back to the cleaned ping polyline and haversine distance, ensuring a bad map-match never degrades the output.</p><p><strong>Signal 2: DTW Deviation between Ping and OSRM Polylines.</strong></p><p><strong>Dynamic Time Warping (DTW)</strong> measures how closely two sequences follow each other by finding the best alignment between their points, even when they differ in length or sampling frequency. It’s a technique used to dynamically compare time series data when the time indices between comparison data points do not sync up perfectly. Here, we use <strong>haversine distance</strong> to measure the spatial gap between each aligned pair.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/343/0*ad5gywkZ9JSLdohL.png" /><figcaption>Dynamic Time Warping</figcaption></figure><p>We use DTW to compare the <strong>cleaned GPS trace</strong> with the <strong>OSRM-matched route</strong>. The resulting average deviation tells us how well the road match reflects the rider’s actual movement. If the deviation exceeds <strong>10% of the path length</strong>, we flag the match as unreliable. This adds an extra check to prevent cases where OSRM snapping might return incorrect routes.</p><p><strong>Signal 3: Maximum Timestamp Gap.</strong> We flag trips where the largest gap between consecutive raw pings exceeds <strong>180 seconds</strong>, since the missing interval cannot be reliably reconstructed. Trips with gaps below <strong>45 seconds</strong> provide strong coverage confidence.</p><p><strong>Signal 4: Start and End Lag.</strong> We flag trips where the first or last GPS ping is more than <strong>a fixed threshold </strong>from dispatch or arrival, indicating incomplete route coverage.</p><h3>What the Confidence Levels Mean</h3><p>The four signals combine into a confidence label that travels with every trip:</p><ul><li><strong>High Confidence:</strong> Dense coverage, low DTW deviation, valid OSRM confidence, and minimal start/end lag. The OSRM route and road-network distance are used as the final output.</li><li><strong>Medium Confidence:</strong> Some gaps or divergence, but no major quality issues. The OSRM route is retained, with caution for trip-level decisions.</li><li><strong>Low Confidence:</strong> A major issue such as a large timestamp gap, excessive DTW deviation, invalid OSRM confidence, or significant start/end lag. The OSRM match is discarded, and we fall back to the cleaned ping polyline with haversine distance.</li></ul><p>This ensures a bad map match never makes the output worse than the cleaned GPS data, while giving downstream systems a simple way to filter or weight trips by reliability.</p><h3>4. Learning the Road: Historically Consistent Paths from Millions of Trips</h3><p>So far, every trip has been processed in isolation. But the strongest signal is <strong>collective behaviour</strong>: the routes riders <em>actually take, </em>including OSM inaccuracies, service-lane shortcuts, and turns riders avoid.</p><p>This section explains how Route IQ extracts these <strong>consensus paths</strong> from historical trips, creating a ground-truth routing layer built entirely from observed behaviour.</p><h3>Step 1: Spatial Bucketing with H3</h3><p>The first challenge is defining what “same origin-destination pair” means. Two customers 50 metres apart should likely share a consensus route, while customers 2 km apart should not.</p><p>We use <strong>Uber’s H3 spatial index</strong> at two resolutions:</p><ul><li><strong>Store location → Resolution 10:</strong> ~76 m average edge length, providing a relatively precise origin bucket.</li><li><strong>Customer location → Resolution 9:</strong> ~201 m average edge length, intentionally coarser to group nearby customers into the same destination bucket.</li></ul><p>Each trip’s rider polyline is then converted into an <strong>ordered sequence of H3 cells at resolution 10</strong>. This represents the route as a chain of spatial tiles rather than raw latitude/longitude points. Consecutive duplicate cells are removed so that lingering in one cell does not create repeated entries.</p><p>This gives us a clean abstraction: every trip is a sequence of H3 cell IDs, and every store-destination pair defines a group of such sequences.</p><h3>Step 2: The Hidden Markov Model</h3><p>We frame consensus path learning as inference over a <strong>Hidden Markov Model (HMM)</strong>. The key insight is that the rider’s <em>true</em> route through the road network (the hidden state sequence) is not directly observable. What we observe are noisy GPS pings that have been discretized into H3 cells. The HMM separates these two layers cleanly.</p><p>Formally, for each (store, destination) group, the HMM is defined by the tuple: λ = (S, O, A, B, π)</p><ul><li><strong>Hidden states — </strong>S = {s₁, s₂, …, sₙ} — The set of all H3 cells (resolution 10) that appear in any trip for this OD pair. Each state represents a physical location on the road network that a rider could actually occupy.</li><li><strong>Observations — </strong>O = {o₁, o₂, …, oₜ} — The observed H3 cell sequence for a single trip, derived from the rider’s GPS pings. These are noisy: GPS error means the observed cell may differ from the cell the rider actually occupied.</li><li><strong>Transition matrix — </strong>A<strong> </strong>— Where aᵢⱼ = P(sⱼ | sᵢ) is the probability of moving from cell to cell , estimated from the empirical transition counts across all trips in the group:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*juTO2VEtuweNa9-8.png" /></figure><ul><li>This matrix encodes the collective routing knowledge of all riders who’ve served this OD pair. A cell at a fork in the road will have two outgoing transitions with probabilities proportional to how many riders took each branch. Transitions with aᵢⱼ &lt; 0.02 are floored to zero — they represent GPS noise or one-off detours, not genuine route choices.</li><li><strong>Emission probabilities (B) </strong>— Where bⱼ(oₜ) = P(oₜ | sⱼ) models the probability of observing cell when the rider is truly in state . We use an isotropic Gaussian based on haversine distance between cell centroids:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ldLsbSqyPisWhPfN.png" /></figure><p>where μᵢ denotes the centroid of cell (i)and σ ≈ 15 m (converted to degrees as σ_deg = 15/111,000), reflecting typical urban GPS accuracy. A state that is physically close to the observed cell gets high emission probability; distant states are exponentially suppressed.</p><ul><li><strong>Initial distribution </strong>π<strong> </strong>— Concentrated on the store’s H3 cell, since all trips in a group originate from the same store.</li></ul><h3>Step 3: Viterbi Decoding — MAP Inference over the HMM</h3><p>Given the observed cell sequence O = {o₁, o₂, …, oₜ} for a single trip, we seek the most likely hidden state sequence S* = {s₁*, …, sₙ*} that maximises the posterior probability:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*e4A0VTSJNc-Jp823.png" /></figure><p>This is solved efficiently using the <strong>Viterbi algorithm.</strong> The same dynamic programming approach used in speech recognition and computational genomics. We work in log-space to avoid numerical underflow:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*z1b9xE335q_4kP_h.png" /></figure><p>where Vₜ(j) is the log-probability of the most likely path ending in state at time . The optimal path is recovered via backtracking.</p><p>Think of this as “spell-checking” each trip against the group’s collective vocabulary. The output is a corrected cell sequence that is consistent with both the observed GPS data and the collective transition patterns. Trips with unusual detours get smoothed toward the majority path; trips already on the main route are largely unchanged.</p><h3>Step 4: Clustering to Discover Multiple Routes</h3><p>A critical insight: <strong>there isn’t always one correct route</strong>. A store near a major intersection might have riders splitting between two completely different roads to reach the same destination.</p><p>We solve this by <strong>clustering the Viterbi-decoded sequences</strong> based on H3 cell overlap. Two trips are in the same cluster if they share at least 50% of their cells (measured bidirectionally — the overlap must be significant relative to <em>both</em> trips’ lengths).</p><p>The result: typically <strong>1–3 route clusters per OD pair</strong>. In dense urban areas, 2 routes is common; 3 is rare but real.</p><h3>Step 5: Consensus Path Extraction using Greedy Walk with Dijkstra Fallback</h3><p>For each cluster, we need to extract a single “consensus path”. The most representative route. We do this by building a <strong>cluster-specific transition graph</strong> (using only transitions from trips in that cluster) and walking it.</p><p><strong>Greedy walk (primary method):</strong> Start at the store’s H3 cell. At each step, move to the neighbouring cell with the highest transition probability that hasn’t been visited yet. Continue until reaching a cell that falls within the destination hex (res-9) or running out of viable next steps.</p><p>The greedy walk works well when the cluster is tight. Most trips follow essentially the same route with minor variations. But it can fail when the graph is sparse or has dead ends: the walker might reach a cell with no unvisited high-probability neighbours before arriving at the destination.</p><p><strong>Dijkstra fallback:</strong> If the greedy path fails to reach the destination (last cell outside the destination hex and &gt;100m away), we use Dijkstra on the same graph with -log(probability) edge weights to find the highest-probability path.</p><p>The consensus path is encoded as a polyline and stored alongside metadata: the route probability (what fraction of trips in the group belong to this cluster), the path distance in meters, and which method produced it (greedy vs Dijkstra).</p><h3>Step 5b: Filtering to Qualifying Routes</h3><p>Not every cluster deserves a consensus path. We keep at most <strong>the top 2 routes</strong>, and only if each accounts for more than <strong>25% of trips</strong> in the group. This filters out tiny clusters that represent one-off detours rather than genuine alternative routes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Xg0fmb3jk3m78Lb2.gif" /><figcaption>From spaghetti to consensus: individual trip polylines converging into learned consensus routes through Viterbi decoding, clustering, and path extraction</figcaption></figure><h3>5. What Comes Next: From Clean Routes to Route Intelligence</h3><p>Once we have consensus paths, new possibilities open up:</p><p><strong>Deviation detection:</strong> For each live trip, Dynamic Time Warping measures divergence from the consensus path, normalized by path length. A score of 2% indicates close alignment, while &gt;20% suggests significant deviation potentially a shortcut, detour, distance padding, or road closure.</p><p><strong>Distance validation.</strong> If the consensus path distance is 1.8 km and a rider’s trip reports 3.2 km for the same OD pair, that’s a strong anomaly signal.</p><p><strong>ETA calibration.</strong> Consensus paths carry implicit timing information. If most riders cover a particular route in 8–12 minutes, a promised ETA of 6 minutes is probably unrealistic.</p><p><strong>Rider behaviour analytics.</strong> Aggregate deviation scores per rider across hundreds of trips. Riders who consistently deviate in the same direction might have found a genuinely better route. Riders who deviate erratically might be struggling with navigation.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6898d3260ed2" width="1" height="1" alt=""><hr><p><a href="https://blog.zepto.com/route-iq-how-we-reconstruct-delivery-paths-from-noisy-gps-at-scale-6898d3260ed2">Route IQ: How We Reconstruct Delivery Paths from Noisy GPS at Scale</a> was originally published in <a href="https://blog.zepto.com">Zepto TechXPress</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Billion Events a Day: How We Built Ads Analytics That Actually Works]]></title>
            <link>https://blog.zepto.com/a-billion-events-a-day-how-we-built-ads-analytics-that-actually-works-c28904bbf8de?source=rss-47a9eb8a04a5------2</link>
            <guid isPermaLink="false">https://medium.com/p/c28904bbf8de</guid>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[apache-flink]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[clickhouse]]></category>
            <dc:creator><![CDATA[Zepto Tech]]></dc:creator>
            <pubDate>Mon, 27 Jul 2026 07:46:04 GMT</pubDate>
            <atom:updated>2026-07-27T07:46:04.621Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3hLYuYe0YzlzaIPa.png" /></figure><p><em>From a naïve Flink pipeline to a ClickHouse-powered analytics engine — the full story of deduplication, late events, aggregation windows, and the lessons we learned the hard way.</em></p><h3>The Problem</h3><p>When we serve an ad, we get back a stream of user events — views, clicks, and more — flowing in. We needed to consume this stream and answer some fundamental questions: <em>How much has each advertiser spent? Across which campaigns, search terms, categories, placements, and devices?</em> And critically — the same analytics engine would power advertiser billing and real-time budget enforcement.</p><p><strong>Functional Requirements</strong></p><ul><li>Spend analytics across campaigns, placements, keywords, and product categories</li><li>Revenue attribution on the same dimensions</li><li>Real-time budget enforcement — stop serving ads when a campaign’s spend limit is hit</li></ul><p><strong>Non-Functional Requirements</strong></p><ul><li>Near real-time analytics to prevent overspend</li><li>High reliability — downtime directly causes overspend</li><li>Exactly-once semantics — duplicates overcharge advertisers, missing events lose revenue</li><li>~1 billion events per day throughput</li></ul><h3>Phase 1 — The Flink Approach</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Qh7u6HbJpVV8WqAm.png" /></figure><p>Our first instinct was straightforward: write Flink jobs using 5-minute tumbling windows to aggregate event counts over the request dimensions. The result was a compact, queryable set of records in Postgres — regardless of how much traffic came in, the number of rows for a given day and dimension set stayed fixed. Querying for an hour or a day was just a matter of summing up 5-minute buckets.</p><p>This worked. Until it didn’t.</p><p><strong>Issue 1: No deduplication</strong></p><p>Pre-aggregating events made deduplication impossible. Events could be replayed due to failures, leading to duplicate counts. We also had business-level deduplication requirements — for example, a user should only be charged for their first click on an ad. With only aggregated data available, there was no reliable way to handle either of these cases.</p><p><strong>Issue 2: No raw event history</strong></p><p>Once aggregated, the click-level data was gone. Debugging discrepancies — <em>“why did the count spike here?”</em> — had no answer. We needed the ability to trace individual events.</p><p>The conclusion was clear: we needed to store raw events and run aggregations over them. That meant an OLAP database.</p><h3>Phase 2 — Rebuilding on ClickHouse</h3><p>We evaluated our needs against standard OLAP requirements: heavy aggregate queries over specific columns (bid, impression_count), time-series analytics, columnar storage, and high write throughput. ClickHouse was the clear fit.</p><h3>Infrastructure Setup</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xpPDsmCDm8ELWso7.png" /></figure><p>We started with a single shard, three-replica cluster. Each node runs on 4 CPU / 16GB RAM / 1TB storage. All three replicas act as masters — writes land on any node and replicate to the others, so reads can be served from any node too. This gives us high availability out of the box: a failed node doesn’t take down the cluster, and rejoining a recovered node is automatic.</p><p>ZooKeeper handles coordination: it tracks which parts of a table live where, orchestrates merges, compactions, and replication. One important safety rail — if a node loses its ZooKeeper connection, it transitions its replicated tables to <strong>READONLY</strong> mode. It stops accepting writes while the other nodes continue, and once reconnected, it catches up and resumes. No silent data divergence.</p><p>For cluster management we used the <strong>Altinity Kubernetes Operator</strong>, which let us handle configuration, user management, and service endpoints entirely through YAML. Metrics flow into Grafana using Altinity’s published dashboard templates. Backups to S3 are configured via Altinity ClickHouse Backup with a defined schedule and rotation policy.</p><h3>Ingesting Raw Events</h3><p>ClickHouse’s Kafka Engine made ingestion straightforward. We attached materialized views to the engine — one per version of our event schema — each feeding into the same canonical table contract. We also captured Kafka virtual fields (_topic, _partition, _offset) in each row, giving us a precise audit trail for debugging event loss or corruption.</p><p>The raw events table uses ReplicatedMergeTree: columnar storage, sparse primary index, custom partitioning, and cross-node replication.</p><h3>Deduplication Strategy</h3><p>The obvious candidate was ReplacingReplicatedMergeTree, where ClickHouse itself deduplicates on merge. We rejected it for three reasons:</p><ol><li><strong>Eventual consistency only.</strong> Deduplication happens during part merges. At our ingest rates, merge frequency isn’t guaranteed — reads between merges will see duplicates.</li><li><strong>No custom dedup logic.</strong> Dedup rules differed by ad slot type. A single merge key couldn’t express that.</li><li><strong>Irreversibility.</strong> Once duplicates are merged away, you can’t reconstruct them. Future rule changes would have no raw data to reprocess against.</li></ol><p>Our solution: <strong>store everything, deduplicate at query time.</strong> Each ingested record includes a _timestamp field set to the arrival timestamp in nanoseconds. To get the canonical version of any event, we group on the fields that define uniqueness and use argMax to pick the latest values:</p><pre>SELECT<br>    eventId,<br>    advertiserId,<br>    campaignId,<br>    bidType,<br>    argMax(bid,  _timestamp) AS bid,<br>    argMax(event_timestamp, _timestamp) AS created_at,<br>    argMax(session_id, _timestamp) AS session_id<br>FROM ad_impressions<br>GROUP BY<br>    eventId, advertiserId, campaignId, campaignId .....<br>-- eventId + advertiserId + campaignId + campaignId ... define uniqueness<br>-- argMax picks the latest value for all non-key fields</pre><p>This keeps raw history intact, supports custom dedup logic per event type, and remains fully reprocessable if rules change.</p><h3>Partitioning and Ordering</h3><p>With 1B events a day, how data is physically laid out has a direct impact on both query latency and ingestion throughput.</p><p><strong>Partition key:</strong> Day partitioning on event time. Real-time ingestion writes to a single partition. Analytics queries rarely span more than two days, so ClickHouse reads at most two partitions instead of scanning a large monthly slab.</p><p><strong>Order key:</strong> (advertiserId, campaignId, bidType, …). Our most common query pattern filters by advertiser and campaign, so those come first. Columns are ordered by increasing cardinality to maximize compression — similar values cluster within the same granule, reducing storage and speeding up scans.</p><h3>Building the Analytics Layer</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*M08DDCrE-heNBFYX.png" /></figure><h3>The Aggregation Problem</h3><p>With raw events stored and deduplicated, the next challenge was aggregating them into queryable metrics across all dimensions. We chose 5-minute windows — fine-grained enough for near real-time budget enforcement, and easy to roll up into hours, days, or months by summing buckets.</p><p>Our first attempt used a ClickHouse materialized view on the raw events table with an AggregatingMergeTree engine. It failed in two ways: dedup logic couldn&#39;t be applied before aggregation, and late-arriving events got bucketed to the wrong window.</p><p>We switched to a <strong>scheduled worker</strong> — a cron-like job that fires every hour and runs a SQL command to:</p><ol><li>Deduplicate raw events over the last 4 hours using argMax</li><li>Tag each event to the correct 5-minute window using toStartOfFiveMinute(event_timestamp)</li><li>Insert the aggregated result into the analytics table, overwriting prior calculations for those windows</li></ol><p>The 4-hour lookback handles late events: an event delayed by network issues or replay still lands in the right window based on its original event_timestamp, not its arrival time.</p><h3>Controlling Query Scope</h3><p>Running a 4-hour dedup-and-aggregate over all events every hour would be expensive. The key insight: once a campaign stops receiving events, recalculating its aggregates is wasted work. We added a subquery to filter only the brands and campaigns that had events in the last hour:</p><pre>WHERE (advertiserId, campaignId) IN (<br>    SELECT advertiserId, campaignId<br>    FROM   ad_impressions<br>    WHERE  event_timestamp &gt;= (now() - toIntervalHour(1))<br>    GROUP BY advertiserId, campaignId<br>)<br>AND event_timestamp &gt;= toStartOfFiveMinute(now() - toIntervalHour(4))</pre><p>This scopes the expensive in-memory dedup step to only active campaigns, cutting query load dramatically during off-peak hours.</p><h3>The Full Aggregation Query</h3><p>Here’s an excerpt of the query that ties it all together:</p><pre>INSERT INTO advertiser_analytics<br>SELECT<br>    advertiserId, campaignId, bidType,<br>    inventoryId, searchTerm, deviceType,<br>    cityName, page, widget</pre><pre>    sum(coalesce(impressions, 0))        AS impressions,<br>    sum(coalesce(clicks, 0))             AS clicks,<br>    toStartOfFiveMinute(event_timestamp) AS created_at,<br>    now()                                AS _timestamp<br> ( /* deduplicated raw events subquery */ )<br>GROUP BY<br>    advertiserId, campaignId, bidType,<br>    inventoryId, searchTerm, deviceType,<br>    cityName, page, widget, created_at</pre><h3>Handling Re-aggregation Without Bloat</h3><p>Every hourly run recalculates windows in the last 4 hours. If we simply append, the analytics table accumulates multiple rows for the same window — daily or monthly rollups would double-count. We solved this with ReplacingMergeTree on the analytics table, using the combination of dimension columns as the primary key and _timestamp (time of aggregation) to track which row is latest.</p><blockquote><strong><em>Gotcha:</em></strong><em> </em><em>ReplacingMergeTree only deduplicates during part merges, which aren&#39;t instant. We expose the analytics data through a view that applies the same </em><em>argMax(_timestamp) pattern at read time — guaranteeing callers always see the latest, non-duplicated aggregations.</em></blockquote><h3>Phase 3 — Scaling to a Billion Events</h3><p>With new advertisers onboarding steadily, event volume kept climbing until we crossed 1 billion events a day. We had already upgraded the cluster to 32 CPU / 90 GB RAM / 3 TB storage per node, tightened the dedupe window to 4 hours, and dropped the job to hourly. It wasn’t enough.</p><p>Going back to the whiteboard, we identified two distinct bottlenecks.</p><h3>Bottleneck 1 — Aggregating All Advertisers Together</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*da5tsbTYo6-xI_x8.png" /></figure><p>The hourly aggregation job was running dedup across the entire raw events table for all active advertisers in a single query. The dedup logic — grouping, sorting, and argMax over millions of rows — is inherently memory-intensive. Running it for every advertiser in one shot meant memory pressure grew with the total event volume, not with any individual advertiser&#39;s share.</p><p>The fix was to make aggregation <strong>event-driven and per-advertiser</strong>.</p><p>Instead of one monolithic query, the job now does two things:</p><ol><li><strong>Compute the work list.</strong> At the start of each cycle, a lightweight query identifies every brand that received events in the last 4 hours.</li><li><strong>Enqueue per-brand tasks.</strong> Each brand (or small batch of brands) becomes an independent task published to a Kafka topic. A pool of worker processes consumes these tasks and executes the aggregation query scoped to that brand alone.</li></ol><p>This matters because query cost doesn’t scale linearly with row count — memory and time grow faster. Processing smaller slices independently is strictly cheaper in aggregate: if cost is proportional to <em>n²</em>, then 10² &gt; 5² + 5². Beyond the math, it also gave us a natural parallelism knob — we can scale workers up or down independently of the ClickHouse cluster, and a slow brand doesn&#39;t block every other advertiser.</p><h3>Bottleneck 2 — The Ever-Growing 5-Minute Table</h3><p>The 5-minute aggregation table had become a problem for advertiser-facing queries. It holds multiple high-cardinality dimension columns — device, keyword, city, category, placement, and more — at 5-minute granularity. When an advertiser queries a quarter’s worth of data, the engine still needs to scan and aggregate millions of rows, even though the advertiser only needs a handful of rolled-up views.</p><p>The insight: <strong>advertisers don’t need all dimension combinations.</strong> They don’t need device × keyword data. What they need is day-level analytics sliced by city, by keyword, or by page — independently.</p><p>We built a set of <strong>low-cardinality day-level tables</strong> tailored to the actual query patterns advertisers use. The 5-minute table still exists but is now reserved for internal platform analytics, which has lower throughput requirements and no tight latency SLA. Advertiser-facing dashboards hit the pre-rolled day-level tables instead — dramatically smaller, dramatically faster.</p><h3>Bottleneck 3 — Budget Enforcement Can’t Wait an Hour</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*idLVE9CyM4QAb7A2.png" /></figure><p>Moving to an hourly aggregation job introduced a new problem: a campaign burning through its daily budget could overspend for up to an hour before we caught it. The analytics job and the budget enforcement job had very different latency requirements, and we had been conflating them.</p><p>We separated them into two independent jobs:</p><p><strong>Job 1 — Analytics (hourly).</strong> Builds the full multi-dimensional, multi-metric aggregations for advertiser dashboards and platform insights. This is the expensive job and it’s fine running hourly — advertisers checking yesterday’s keyword performance don’t need sub-minute freshness.</p><p><strong>Job 2 — Daily spend (frequent, lightweight).</strong> Calculates only the cumulative spend for each campaign for the current day. No dimensions, no secondary metrics. Critically, it only reads the event type that actually drives charges for that campaign — clicks for CPC campaigns, impressions for CPM campaigns. Everything else in the raw events table is ignored entirely.</p><p>This selectivity is what makes the job fast enough to run frequently. By reading only the billing-relevant event type, the raw events scanned per campaign drop dramatically. Combined with scoping execution to campaigns that had events in the last 3 hours, the query is lightweight enough to run at a cadence where overspend risk is acceptably small.</p><p>Like the analytics job, this one is broken down per-advertiser and enqueued as async tasks — so it inherits the same parallelism and isolation benefits without adding load to the main aggregation pipeline. The job runs every 3 minutes and completes in around 1 minute, meaning spend data is never more than a few minutes stale — a far tighter safety net than the hour-level window we had before.</p><h3>Further Tuning</h3><p>Beyond the two structural changes, a few targeted optimizations made a meaningful difference:</p><p><strong>Resource isolation per use case.</strong> High-throughput advertiser queries were competing with the background worker jobs for CPU and memory on the same ClickHouse nodes. We created separate ClickHouse users for each workload — advertiser queries, aggregation workers, and internal analytics — with explicit CPU and memory limits per user. A runaway advertiser query can no longer starve the aggregation pipeline.</p><p><strong>Spilling to disk for heavy queries.</strong> For queries that genuinely need to process large working sets, we configured max_bytes_before_external_group_by and max_bytes_before_external_sort. When in-memory usage crosses the threshold, ClickHouse spills intermediate state to disk rather than OOM-killing the query. It&#39;s slower than in-memory, but it&#39;s controlled and predictable — which is far better than an outage.</p><p><strong>Tiered storage — hot local, cold S3.</strong> Raw events and 5-minute aggregations are queried heavily for recent data and rarely for anything older than a few weeks. We set up partition-level archival policies to move older partitions to S3. Because we had partitioned by date from the start, the archival configuration was straightforward — ClickHouse’s toDate(created_at) partitions map cleanly to an age-based policy. Local NVMe stays lean; cold data stays accessible.</p><h3>What We’d Tell Ourselves Earlier</h3><p><strong>1. Don’t aggregate away your raw data.</strong> Pre-aggregation is tempting for storage efficiency, but it makes deduplication, debugging, and rule changes impossible. Store raw, aggregate at query time.</p><p><strong>2. Design for late events from day one.</strong> Network delays, retries, and replays are the norm, not the exception. A 4-hour late-event window combined with event-time bucketing absorbed all the edge cases we encountered.</p><p><strong>3. Partition and order keys are your most consequential schema decisions.</strong> We chose toDate(created_at) as partition key and (advertiserId, campaignId) as the leading order key. These two choices had more impact on performance than any query optimization we did afterward.</p><p><strong>4. Scope your aggregation jobs tightly.</strong> Only recalculating active campaigns (those with events in the last hour) cut our per-run compute significantly. Know what you can safely skip.</p><p><strong>5. </strong><strong>ReplacingMergeTree needs a view on top.</strong> Relying on merge-triggered dedup for reads will burn you. Always expose aggregated data through a dedup view that applies argMax inline.</p><p><em>The system today handles over a billion events per day, provides spend analytics to advertisers across dozens of dimensions, and powers the billing engine without overcharging or undercounting. The architecture is still evolving — but the core principles above have held across every iteration.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c28904bbf8de" width="1" height="1" alt=""><hr><p><a href="https://blog.zepto.com/a-billion-events-a-day-how-we-built-ads-analytics-that-actually-works-c28904bbf8de">A Billion Events a Day: How We Built Ads Analytics That Actually Works</a> was originally published in <a href="https://blog.zepto.com">Zepto TechXPress</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Field Guide to My Data Science Internship at Zepto]]></title>
            <link>https://blog.zepto.com/a-field-guide-to-my-data-science-internship-at-zepto-1cf0ea7ae928?source=rss-47a9eb8a04a5------2</link>
            <guid isPermaLink="false">https://medium.com/p/1cf0ea7ae928</guid>
            <category><![CDATA[internships]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[quick-commerce]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[pricing]]></category>
            <dc:creator><![CDATA[Zepto Tech]]></dc:creator>
            <pubDate>Wed, 22 Jul 2026 07:03:44 GMT</pubDate>
            <atom:updated>2026-07-22T07:03:44.579Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WhfNq7cfyVMByVb4.png" /></figure><blockquote><strong><em>🎙️ The Observer’s Log: Entry #01<br></em></strong><em>“We arrive at the bustling, hyper velocity habitat known as Zepto. Here, time is not measured in hours or days, but in ten minute increments. Today, a new species enters the environment: The Data Science Intern. Notice how they cautiously approach their workstation, armed with Python notebooks and an eagerness to understand the complex mathematical rhythms of quick commerce.”</em></blockquote><p>When I joined Zepto as a Data Science Intern on the Pricing team, I knew I was stepping into one of India’s fastest moving tech ecosystems. But reading about quick commerce at scale and actually building for it are two entirely different beasts.</p><p>Working alongside my manager, Deepak Dhankani, and my buddy, Abhijeet Anand, my mission for the season was clear: dive deep into the world of <strong>inventory based pricing</strong> and build models that could react to dynamic supply and demand in near real time.</p><p>Here is a look into my journey navigating the engineering, operations, and culture that make Zepto tick.</p><h3>Phase 1: Observing the Physical Backbone (The First 48 Hours)</h3><blockquote><strong><em>🎙️ The Observer’s Log: Entry #02<br></em></strong><em>“Before the intern can manipulate the digital data streams, they must first understand the physical realm. To build models for the wild, one must step into the wild. And so, the migration to the Dark Stores begins.”</em></blockquote><p>My onboarding during the first two days was far from a standard presentation slideshow. We were immersed straight into the philosophy and physical reality of the company.</p><p>We kicked things off with an interactive session with Co-founder Kaivalya Vohra (KV). What struck me most was his clarity and humility; he emphasized that fast experimentation and deep operational thinking are what truly power the company. We also sat down with CTO Nikhil Mittal, who broke down Zepto’s engineering philosophy: building systems that are fundamentally <strong>observable, scalable, and resilient</strong>.</p><p>The theoretical architecture came to life when we visited Zepto’s Super Stores and Dark Stores. Standing inside a Dark Store is like watching a highly optimized relay race, watching inventory being stocked, orders picked, and riders dispatched; all in real time where every second is accounted for<em>.</em></p><h3>Phase 2: No “Intern” Labels in the Wild</h3><blockquote><strong><em>🎙️ The Observer’s Log: Entry #03<br></em></strong><em>“In many corporate ecosystems, the intern is kept in protective captivity, working on simulated data and sandbox projects that never see the light of day. But at Zepto, the environment demands immediate integration. Survival means shipping to production.”</em></blockquote><p>One of the most refreshing aspects of my internship was that there was no artificial divide between “intern work” and “full time work.” From day one, I was entrusted with real ownership.</p><p>Working on inventory based pricing, I was tasked with developing a pricing model that optimizes inventory availability while ensuring alignment with Zepto’s Pricing strategy. Under the guidance of Amit, Deepak and Abhijeet, I went from ideation to deployment at lightning speed. Just <strong>two weeks into my internship, my model was deployed live on a pilot!</strong> To scale the model across Zepto’s massive footprint, I regularly sat in on strategy meetings with Product and Planning teams. Getting out of the data science silo taught me three core skills:</p><ul><li><strong>Translating Business to Tech:</strong> I learned how to dissect business requirement, ask the right questions about constraints, and frame it into a concrete mathematical optimization and predictive modeling problem.</li><li><strong>Experimentation Design:</strong> Structured A/B tests in a hyper local delivery environment; selecting control vs. test dark stores, accounting for demand spillover, and defining statistical significance for real-time metrics.</li><li><strong>Shipping to Production:</strong> Connected the model directly to live dark store inventory and pricing streams using ETL pipelines.</li></ul><h3>Phase 3: The Mega Scale of the Mother Hub</h3><blockquote><strong><em>🎙️ The Observer’s Log: Entry #04<br></em></strong><em>“We now venture into the epicenter of the supply chain: The Mother Hub. Here, the sheer volume of data we process on our screens manifests as towering aisles of physical goods moving in automated harmony.”</em></blockquote><p>Midway through the internship, our team took a field trip to a Zepto Mother Hub. If the Dark Stores are the sprinters of the ecosystem, the Mother Hub is the beating heart.</p><p>Witnessing the warehouse automation in action, a facility efficiently processing millions of products was mind-blowing. But what made me proudest as a data scientist was seeing in-house AI initiatives running live on the ground. A standout moment was seeing <strong>OdinEye</strong> (Zepto’s scalable computer vision face authentication system for attendance) seamlessly processing personnel in real time. Seeing machine learning solve tangible operational friction at scale reaffirmed why we build the way we do.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4yfj8E5M0D4EvwRy.png" /></figure><h3>Phase 4: Pack Bonding</h3><blockquote><strong><em>🎙️ The Observer’s Log: Entry #05<br></em></strong><em>“Even the most industrious pack must engage in ritualistic play to maintain social cohesion. Notice how the hierarchy dissolves on the turf, replaced by friendly competition and adrenaline.”</em></blockquote><p>It wasn’t all algorithms, pilot testing, and deployment metrics. The Data Science team made sure we took time to unwind and bond as a squad.</p><p>Our team outing was an absolute blast, featuring a high energy mix of football, go-karting, and an unforgettable game of laser tag. Moments like these, laughing, competing, and collaborating outside the office, made the team feel like a community.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fcw6BY2qjtj8KISc.png" /></figure><h3>Evolution of a Data Scientist</h3><blockquote><strong><em>🎙️ The Observer’s Log: Final Entry<br>“</em></strong><em>As the season concludes, we observe a transformed individual. The hesitation of day one is gone, replaced by the confidence of an engineer who has tested their hypotheses in the real world. The intern has successfully evolved.”</em></blockquote><p>Reflecting on my time at Zepto, I realise I’ve gained much more than just technical proficiency in pricing models and data pipelines. I’ve learned how to build with velocity, collaborate cross functionally with product and planning teams, and design data science models that generate business impact.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1cf0ea7ae928" width="1" height="1" alt=""><hr><p><a href="https://blog.zepto.com/a-field-guide-to-my-data-science-internship-at-zepto-1cf0ea7ae928">A Field Guide to My Data Science Internship at Zepto</a> was originally published in <a href="https://blog.zepto.com">Zepto TechXPress</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Developer Pipeline: My Zepto Engineering Internship]]></title>
            <link>https://blog.zepto.com/the-developer-pipeline-my-zepto-engineering-internship-c8096c07d6e6?source=rss-47a9eb8a04a5------2</link>
            <guid isPermaLink="false">https://medium.com/p/c8096c07d6e6</guid>
            <category><![CDATA[zepto-tech]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[internship-experience]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Zepto Tech]]></dc:creator>
            <pubDate>Thu, 16 Jul 2026 07:13:28 GMT</pubDate>
            <atom:updated>2026-07-16T07:13:28.868Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3G1_uz96NySKH9Ofjn58-A.png" /></figure><p>git init : <strong>Starting my Journey</strong></p><p>I was thrilled to land an internship offer from Zepto through our on-campus placements. Naturally, I spent the weeks leading up to the start date buzzing with excitement over the kinds of projects I would be tackling.</p><p>And before long, the joining date arrived. I felt an incredible surge of curiosity about what those two months would hold, and I remember being so genuinely eager to dive into the unknown and start learning.</p><p>git clone zepto : <strong>Onboarding at Zepto</strong></p><p>On the first day itself, we were given the opportunity to interact directly with Zepto’s senior leadership. The session with <strong>co-founder</strong> <strong>Kaivalya Vohra (KV)</strong> was particularly eye-opening. He shared insights into his personal journey and walked us through exactly how Zepto was built from the ground up. What stood out to me the most wasn’t just the scaling strategy, but his sheer conviction in the vision of Zepto and the passion driving it forward.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*rcap6HvmSMjo7rw9" /><figcaption>Session with KV</figcaption></figure><p>Next, Zepto’s leadership mapped out our internship roadmap and high-stakes ecosystem. <strong>CTO Nikhil Mittal</strong> emphasized how even seconds of engineering downtime cause massive losses, while <strong>Koushik Kottamasu (Sr. Director, Eng.)</strong> and <strong>CHRO Sneha Arora</strong> covered AI and career growth.</p><p>Following the leadership sessions, we stepped away from our desks to visit the actual backbone of Zepto: the <strong>dark stores</strong> and <strong>super stores</strong>. Walking in, the contrast to a traditional supermarket was immediately striking. Instead of finding similar items grouped together, a bar of soap sat right next to a bottle of moisturizer. While seemingly random, this calculated layout optimized every square inch to minimize picker movement and shave crucial seconds off delivery times.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*0hKcj-jytji019qF" /><figcaption>Dark Store Visit</figcaption></figure><p>After the visit, I was introduced to my team and given all the required access for my work as an intern.</p><p>git checkout -b monetization : <strong>Joining the Monetization Team</strong></p><p>During my time at Zepto, I had the opportunity to work with the Monetization team, the team responsible for all the advertisements you currently see on the platform. Knowing that ad revenue is poised to be one of Zepto’s major growth drivers moving forward, it was fascinating to peek behind the curtain and learn how the business logic of ad serving actually works at scale.</p><p>What immediately stood out to me was how well-organized the team was. We had daily standups and frequent discussions about both current and upcoming projects. I wasn’t just there to listen, either. The team actively expected me to be inquisitive, ask the tough questions, and bring my own ideas to the table during these discussions.</p><p>But the most amazing part of the entire experience? <strong>The absolute ownership I was given over my project.</strong> It is an open secret in the tech industry that a lot of companies are hesitant to let interns touch production. Zepto took the exact opposite approach. Instead of handing my code off to a senior engineer, I was expected to handle the deployments myself. I was the one pushing the code, actively monitoring the live traffic, and keeping a close eye on the logs during rollouts. That level of trust and hands-on responsibility taught me more about real-world software engineering than any classroom ever could.</p><p>The constant support I received from <strong>Shivam Singhal (Engineering Manager), Gaurav Gupta (Software Engineer III),</strong> and the entire <strong>Monetization team</strong> played an absolutely vital role during my time there.</p><p>What I appreciated most was that their mentorship extended far beyond just helping me ship code and hit my internship milestones. They genuinely invested in my growth as a professional, teaching me lifelong skills that I will carry forward in my career: how to structure my thoughts, frame my ideas effectively, and articulate complex technical approaches in a way that actually makes sense to others.</p><p>git push origin master: <strong>What I built</strong></p><p>The core problem was a major operational bottleneck: developers were constantly interrupted to investigate why specific brand advertisements were not showing up. While some causes were simple, like an exhausted budget, others involved complex backend ranking algorithms. Because diagnosing these required manual log inspection and deep tribal knowledge, non-technical teams had to escalate almost every query, delaying responses and draining critical engineering bandwidth.</p><p>To eliminate this dependency, I built a self-serve diagnostic system using a <strong>Model Context Protocol (MCP)</strong> connected to an LLM. Instead of forcing teams to parse raw server data, the MCP autonomously mimics a live ad-request path across our downstream microservices to fetch the underlying debug data. The connected LLM then analyzes this data to extract the relevant diagnostic signals, pinpointing exactly where and why an ad dropped out of the auction without exposing sensitive payloads or impacting production traffic.</p><p>I integrated this architecture directly into the team’s daily workflow via our internal Slack bot. Now, when a visibility query is raised, an AI agent orchestrates the MCP’s diagnostic trace and <strong>uses the LLM to translate those extracted signals into a clear, plain-English explanation.</strong> This empowers operations teams to resolve most blockers instantly on their own, reserving developers only for genuinely complex escalations.</p><p>time.sleep() : <strong>Life Outside the Code</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6I0z74aaCD4VtcaY.png" /><figcaption>Compiling…☕️</figcaption></figure><p>Of course, my time at Zepto wasn’t strictly about architectures and deployments; the Monetization team knew exactly how to unwind. On a day-to-day basis, our evenings were punctuated by casual coffee breaks and some highly competitive, friendly banter around the office foosball table.</p><p>We also went on an incredible team outing that served as the perfect, refreshing break from our fast-paced environment. From playing multiple games and jumping into the pool, to sharing late-night conversations over a ton of amazing food, we had an absolute blast. It was these everyday office moments and offsite adventures away from the laptops that truly helped us bond and made the overall internship experience so memorable.</p><p>Some takeaways that will stay with me:</p><ul><li><strong>Scale demands resilience</strong> — in a hyper-growth environment, even a few seconds of downtime have massive business consequences.</li><li><strong>Code needs business context</strong> — understanding the <em>why</em> behind the product is what makes engineering truly impactful.</li><li><strong>Build to empower others</strong> — using smart architectures like AI to solve operational bottlenecks scales the whole team, not just the codebase.</li><li><strong>Framing ideas is a technical skill</strong> — writing the code is only half the battle; explaining your approach clearly to others is the rest.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*Xln4WqIXZhoTyxCc.gif" /><figcaption>Less debugging, more spiking</figcaption></figure><p>Looking back, my time at Zepto was exactly what I hoped it would be: fast, challenging, and incredibly rewarding. It taught me how to think like a product engineer and build systems that actually matter. No matter where my career takes me next, the lessons I picked up during these months will be the foundation I continue to build on.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c8096c07d6e6" width="1" height="1" alt=""><hr><p><a href="https://blog.zepto.com/the-developer-pipeline-my-zepto-engineering-internship-c8096c07d6e6">The Developer Pipeline: My Zepto Engineering Internship</a> was originally published in <a href="https://blog.zepto.com">Zepto TechXPress</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Build at Zepto Speed: How We Cut Frontend Webpack Build Times by 95%]]></title>
            <link>https://blog.zepto.com/build-at-zepto-speed-how-we-cut-frontend-webpack-build-times-by-95-676440fd7963?source=rss-47a9eb8a04a5------2</link>
            <guid isPermaLink="false">https://medium.com/p/676440fd7963</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[web-performance]]></category>
            <category><![CDATA[front-end-development]]></category>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[webpack]]></category>
            <dc:creator><![CDATA[Zepto Tech]]></dc:creator>
            <pubDate>Tue, 07 Jul 2026 07:25:21 GMT</pubDate>
            <atom:updated>2026-07-07T07:25:21.079Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NhLgssc-Pv6nI-jd.png" /></figure><p>When frontend build times start climbing, the industry playbook is usually the same: migrate to a faster bundler.</p><p>We faced the same pressure. Build times across our Micro-Frontend platform had grown beyond 20 minutes, and Webpack seemed like the obvious culprit.</p><p>Before committing to a migration, we asked a simple question:<strong>Where is the build pipeline actually spending its time?</strong></p><p>The answer challenged several assumptions and ultimately helped us reduce build times from 20 minutes to 2 minutes — without replacing Webpack.</p><p>While there is already extensive coverage of bundler migrations and build optimisation, this article focuses on a large-scale MFE platform running on Docker-based infrastructure, where build performance depends on much more than the bundler alone. Alongside production profiling, we also consolidated publicly available benchmarks for loaders, plugins, minifiers, and package managers into a single analysis, providing a practical reference for evaluating build-performance tradeoffs.</p><h3>Why Build Times Became a Problem</h3><p>In a typical month, over 600 pull requests from 10+ engineers generated more than 5,000 CI runs across our frontend platform. As engineering velocity increased, 20-minute build times evolved from a developer inconvenience into a platform-wide bottleneck — slowing feedback loops, creating CI queue contention, and delaying production fixes. At this scale, the build pipeline consumed more than <strong>1,600 hours of CI time every month</strong>, making build performance a critical engineering productivity concern.</p><p>Our frontend monorepo compounded the challenge at scale:</p><pre>20 production apps: React and Next.js applications<br>7 shared libraries<br>1 centralized E2E test suite<br>~3,950 TSX/JSX files and 6,600+ TypeScript source files<br>~350 reusable component modules<br>144 npm dependencies (74 production, 70 development)<br>Workspace: single hoisted monorepo<br>Team: 10+ engineers, 600+ PRs/month<br>CI: 5,000+ build runs/month</pre><h3>Why We Chose Profiling Over Migration</h3><p>With build times exceeding 20 minutes, migrating to a faster bundler seemed like the obvious fix — but across 20 production apps, 7 shared libraries, and 144 dependencies, the risk was significant. A single regression in a shared library could cascade across the entire monorepo, and re-validating every build output could consume weeks with no guaranteed payoff.</p><p>Instead, we profiled the entire pipeline first. The results revealed major bottlenecks <em>outside</em> the bundler — in type-checking, test orchestration, and dependency resolution — areas no bundler migration would have touched.</p><h3>Measure Before You Optimise</h3><p>One of the most common mistakes in performance engineering is optimising before measuring.</p><p>Modern frontend build systems are complex. A Webpack build passes through multiple stages before producing a deployable artifact:</p><ul><li>Module compilation</li><li>Loader execution</li><li>Chunk generation</li><li>Asset optimisation</li><li>Minification</li><li>Asset emission</li></ul><p>Without visibility into those phases, every optimisation becomes an educated guess.</p><p>Before changing a single loader, plugin, or dependency, we focused on understanding where the build pipeline was actually spending its time.</p><h3>Using Webpack’s ProgressPlugin</h3><p>We started with Webpack’s built-in ProgressPlugin.</p><pre>const webpack = require(&#39;webpack&#39;);</pre><pre>plugins: [<br>  new webpack.ProgressPlugin()<br>]</pre><p>The output quickly revealed something interesting:</p><pre>[webpack] 92% sealing &gt; asset processing<br>TerserPlugin took 825.31s</pre><p>More than <strong>13 minutes</strong> were being spent inside a single plugin.</p><p>To put that into perspective, the rest of the optimisation stage looked like this:</p><pre>copy-webpack-plugin      30ms<br>WriteIndexHtmlPlugin     22ms<br>RealContentHashPlugin    58ms<br>CompressionPlugin        70ms<br>LicenseWebpackPlugin     1.15s<br>TerserPlugin             825.31s</pre><p>At that moment, one thing became obvious:</p><p><strong>Webpack wasn’t the bottleneck.</strong></p><p>JavaScript minification was.</p><p>That discovery fundamentally changed the direction of the optimization effort.</p><h3>Going Deeper with Speed Measure Plugin</h3><p>The ProgressPlugin helped us identify slow build stages. Next, we wanted loader-level visibility to catch exactly what was holding back our compilation. To achieve that, we integrated <strong>Speed Measure Plugin (SMP)</strong>.</p><pre>const SpeedMeasurePlugin = require(&#39;speed-measure-webpack-plugin&#39;);<br>const smp = new SpeedMeasurePlugin();<br>module.exports = smp.wrap(config);</pre><p>By wrapping our configuration, SMP gave us a granular, step-by-step breakdown of how long each loader chain took to process modules. This allowed us to explicitly benchmark our pipeline <strong>Before</strong> and <strong>After</strong> introducing modern compilers like SWC.</p><p>However, the data also gave us an objective reality check:</p><ul><li><strong>CSS processing remains expensive:</strong> The chain containing mini-css-extract-plugin, css-loader, and postcss-loader actually ticked up from <strong>8.66 seconds</strong> to <strong>9.36 seconds</strong>.</li><li><strong>The Culprit Found:</strong> CSS and un-loaded modules accounted for nearly <strong>16 seconds</strong> of our remaining 20.78-second general output time.</li></ul><p>Instead of debating tooling preferences blindly, we now had hard data pointing exactly to where we needed to slice out latency next.</p><h3>Defining the Optimisation Strategy</h3><p>Before implementing changes, we established a set of engineering objectives.</p><h3>Build Performance</h3><ul><li>Reduce cold build latency</li><li>Improve incremental build performance</li><li>Reduce transpilation overhead</li><li>Reduce minification overhead</li><li>Improve CPU utilisation</li></ul><h3>CI Efficiency</h3><ul><li>Maximize Docker cache reuse</li><li>Reduce dependency installation time</li><li>Improve pipeline determinism</li></ul><h3>Platform Stability</h3><ul><li>Maintain enterprise-grade tooling stability</li><li>Avoid high-risk migrations</li><li>Preserve ecosystem compatibility</li><li>Balance performance against long-term maintainability</li></ul><p>These objectives shaped every technology decision that followed.</p><h3>Why We Didn’t Migrate to Vite</h3><p>We’d be lying if we said we didn’t look at Vite. Everyone looks at Vite.</p><p>It’s genuinely excellent software. For greenfield projects or simpler setups, it’s often the right call. But we did something the hype cycle rarely encourages: we benchmarked it against our <em>actual</em> pipeline, not the synthetic toy examples in most migration guides.</p><p>The finding was uncomfortable. The 13+ minutes of Terser time we’d just discovered? It would have followed us to Vite. Minification is minification regardless of bundler. We’d have spent four weeks migrating and arrived at roughly the same problem, just with a different config syntax and a busted plugin ecosystem to untangle.</p><p>We noted Vite in our decision log — <em>revisit if bottlenecks shift to compilation</em> — and moved on.</p><p>We looked at <strong>Rspack</strong> too, a Rust-based bundler with Webpack-compatible config that’s genuinely exciting. But recent security disclosures and an ecosystem that’s still finding its feet made us nervous.<br>Rspack remains on our radar and will be reevaluated as the ecosystem matures.</p><h3>Modernising the Webpack Pipeline</h3><p>Once the strategic decisions were made, the next step was replacing the slowest parts of the pipeline.</p><h3>Replacing Babel with SWC</h3><p>JavaScript and TypeScript transpilation represented one of the largest optimisation opportunities.</p><p>We replaced Babel with SWC (Speedy Web Compiler), a Rust-based compiler built for high-throughput JS/TS transformation.</p><pre>{<br>  test: /\.(jsx?|tsx?)$/,<br>  use: [<br>    &#39;thread-loader&#39;,<br>    {<br>      loader: &#39;swc-loader&#39;<br>    }<br>  ]<br>}</pre><p>Public benchmarks demonstrate the scale of the improvement.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9PesGt_q3sZR9AbH.png" /></figure><p>The difference is difficult to ignore.</p><p>Benefits included:</p><ul><li>Dramatically faster transpilation</li><li>Parallel compilation through thread-loader</li><li>Lower CPU blocking</li><li>Faster CI execution</li></ul><h3>Eliminating the Biggest Bottleneck with esbuild</h3><p>The profiling exercise had already identified the primary offender:</p><p>Terser.</p><p>To address this, we adopted esbuild for JavaScript minification.</p><pre>new EsbuildPlugin({<br>  target: &#39;es2015&#39;,<br>  minify: true<br>})</pre><p>Before standardising on <strong>esbuild</strong>, we leveraged the <strong>minification benchmarks from the </strong><a href="https://github.com/privatenumber/minification-benchmarks"><strong><em>privatenumber/minification-benchmarks</em></strong></a><strong> project</strong> — which compares esbuild, terser, swc, uglify-js, and other minifiers across <strong>minified size, gzipped size, and minification time</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fB0NTkL8xz_u-9Ue.png" /></figure><p>The results showed:</p><ul><li><strong>esbuild delivered competitive minified and gzipped sizes</strong> — slightly larger than the best-in-class but within ∼5–8 % of smallest results.</li><li><strong>Minification time for esbuild (~295 ms)</strong> was dramatically faster than heavy tools like <strong>terser (~6.7 s)</strong>, which becomes costly at monorepo scale.</li><li>While tools like <strong>@swc/core and oxc-minify</strong> achieved smaller outputs, their relative time and ecosystem integration considerations influenced our choice.</li></ul><h3>Faster CSS Optimisation with LightningCSS</h3><p>For CSS optimisation, we adopted LightningCSS from the Parcel team.</p><pre>new CssMinimizerPlugin({<br>  minify: CssMinimizerPlugin.lightningCssMinify,<br>})</pre><p>We checked on official benchmark data published by <strong>Lightning CSS</strong> (Parcel team) comparing it against <strong>cssnano</strong> and <strong>esbuild (CSS minify)</strong>:</p><p>🔗 <a href="https://github.com/parcel-bundler/lightningcss#benchmarks">GitHub — parcel-bundler/lightningcss: An extremely fast CSS parser, transformer, bundler, and minifier written in Rust.</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pHlFiRhKawdW1Lcb.png" /></figure><p>Benchmark data showed three consistent advantages:</p><ul><li>⚡ <strong>Fastest across all benchmarks</strong></li><li>📦 <strong>Consistently smaller output size</strong></li><li>🚀 Significant gains on large CSS bundles (e.g., Tailwind)</li></ul><p>LightningCSS delivered both <strong>better compression and lower minification time</strong>, making it the optimal choice for CI-scale builds where CSS optimisation directly impacts total build latency.</p><p>The impact was substantial:</p><ul><li>JavaScript minification became roughly 10× faster</li><li>CSS optimisation improved by approximately 6×</li></ul><h3>Maximising CPU Utilisation</h3><p>Modern CI agents provide multiple cores, but many build pipelines fail to use them effectively.</p><p>To improve CPU utilisation, we introduced thread-loader:</p><pre>use: [&#39;thread-loader&#39;, &#39;swc-loader&#39;]</pre><p>This allowed expensive transformations to run across multiple CPU cores.</p><p>Benefits included:</p><ul><li>Reduced cold build bottlenecks</li><li>Better resource utilisation</li><li>Higher CI throughput</li></ul><h3>Optimising Dependency Management with pnpm</h3><p>Compilation wasn’t the only bottleneck.</p><p>Dependency installation was also consuming valuable CI time.</p><p>To reduce dependency installation overhead in our monorepo, we evaluated multiple package managers (npm, yarn, pnpm, bun) using publicly available benchmarks comparing <strong>install speed, disk usage, and dependency resolution models</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gUI2Ue5GcEJ6Nmz_.png" /></figure><p>Although some newer runtimes like <strong>Bun</strong> show strong performance in synthetic benchmarks, we selected <strong>pnpm</strong> due to its <strong>mature ecosystem support, Node.js compatibility, and stable adoption in large production systems</strong>.</p><p>Unlike npm’s copy-based installation model, <strong>pnpm uses a content-addressable global store</strong> where packages are downloaded once and <strong>hard-linked into project </strong>node_modules. This eliminates duplication across workspaces and enables efficient dependency reuse.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rCNWATwkho7KWT9y.png" /></figure><h3>Benefits</h3><ul><li><strong>Reduced install time</strong> — dependencies are downloaded once and reused across projects.</li><li><strong>Lower disk usage</strong> — no duplicated packages across multiple node_modules.</li><li><strong>Strict dependency resolution</strong> — prevents phantom dependencies and improves build determinism.</li><li><strong>Better Docker layer caching</strong></li></ul><h3>Impact</h3><ul><li><strong>Reduced CI install time</strong> through global dependency reuse</li><li><strong>Lower disk usage</strong> by eliminating duplicate node_modules packages</li><li><strong>Improved Docker build caching</strong> via pnpm store reuse</li><li><strong>More deterministic builds</strong> with strict dependency resolution</li></ul><pre>RUN --mount=type=cache,target=/root/.local/share/pnpm/store \<br>    pnpm install --frozen-lockfile</pre><h3>The Hidden Bottleneck: Jenkins Authentication</h3><p>One of the most interesting optimisations had nothing to do with frontend tooling.</p><p>AWS CodeArtifact authorisation tokens expire every 12 hours.</p><p>That caused:</p><ul><li>Cache invalidation</li><li>Re-authentication overhead</li><li>Docker layer busting</li></ul><p>To address this, we generated tokens once per Jenkins job and reused them throughout the pipeline.</p><pre>export CODEARTIFACT_AUTH_TOKEN=$(aws codeartifact get-authorization-token \<br>  --domain &lt;domain&gt; \<br>  --domain-owner &lt;owner&gt; \<br>  --query authorizationToken \<br>  --output text)</pre><p>The result:</p><ul><li>Better Docker layer reuse</li><li>Fewer redundant installs</li><li>More stable pipelines</li></ul><h3>Docker Build Optimisation</h3><p>Finally, we restructured our Dockerfiles into deterministic layers:</p><ol><li>Base runtime layer</li><li>Dependency installation layer</li><li>Source copy layer</li><li>Webpack build layer</li></ol><p>Combined with:</p><ul><li>— mount=type=cache</li><li>— mount=type=secret</li></ul><p>this significantly increased cache reuse across builds.</p><h3>Results</h3><p>The combined effect of these optimisations was substantial.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-F9D5IXTDKFfFbHU.png" /></figure><h3>Engineering Takeaways</h3><p>Several lessons emerged from this initiative:</p><ul><li>Build performance is a systems problem, not just a bundler problem.</li><li>Profiling is more valuable than assumptions — measure before you migrate.</li><li>Modern Rust-based tools such as SWC, esbuild, and LightningCSS deliver compounding performance gains.</li><li>pnpm and effective Docker caching improve both build speed and pipeline determinism.</li><li>Infrastructure decisions, from dependency management to token lifecycle handling, can have as much impact as frontend tooling.</li></ul><p>Most importantly:</p><p><strong>The biggest bottleneck is often not where everyone assumes it is.</strong></p><h3>Conclusion</h3><p>When build latency became a recurring bottleneck, replacing Webpack seemed like the obvious solution.</p><p>Profiling revealed a different reality. The largest costs were not in bundling, but in transpilation, minification, dependency management, caching, and container build workflows.</p><p>By modernising those components instead of replacing the bundler, we reduced build times from over 20 minutes to approximately 2 minutes while preserving ecosystem stability and avoiding a high-risk migration.</p><p><strong>Sometimes the fastest path forward isn’t a new platform — it’s understanding where your existing one spends its time.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=676440fd7963" width="1" height="1" alt=""><hr><p><a href="https://blog.zepto.com/build-at-zepto-speed-how-we-cut-frontend-webpack-build-times-by-95-676440fd7963">Build at Zepto Speed: How We Cut Frontend Webpack Build Times by 95%</a> was originally published in <a href="https://blog.zepto.com">Zepto TechXPress</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>