<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Raphael Montaud on Medium]]></title>
        <description><![CDATA[Stories by Raphael Montaud on Medium]]></description>
        <link>https://medium.com/@rmontaud?source=rss-7e93bd9fc8de------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*21Zc5TCKiaoelMCRC5_25A.jpeg</url>
            <title>Stories by Raphael Montaud on Medium</title>
            <link>https://medium.com/@rmontaud?source=rss-7e93bd9fc8de------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 02:10:07 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@rmontaud/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com" rel="hub"/>
        <item>
            <title><![CDATA[How we think about text classification in the LLM era]]></title>
            <link>https://medium.engineering/how-we-think-about-text-classification-in-the-llm-era-89a185f79b68?source=rss-7e93bd9fc8de------2</link>
            <guid isPermaLink="false">https://medium.com/p/89a185f79b68</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[text-classification]]></category>
            <category><![CDATA[recommendation-system]]></category>
            <dc:creator><![CDATA[Raphael Montaud]]></dc:creator>
            <pubDate>Wed, 19 Aug 2026 20:00:36 GMT</pubDate>
            <atom:updated>2026-08-19T20:00:36.635Z</atom:updated>
            <content:encoded><![CDATA[<h3>Why we think LLMs can be useful and why we will not replace all of our models with them</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MkYq_MYsJnyv418AXICeqw.jpeg" /></figure><h3>Context</h3><p>At Medium, we have many Machine Learning models that we use to label stories automatically. These affect what stories we recommend to readers.</p><p>Here’s some examples:</p><figure><img alt="Three examples of models that classify stories based on their content. An NSFW model and a Spam model each produce a single score, and a threshold decides the label — NSFW or SFW, Spam or Ham. A Topics model instead scores a long list of topics at once, and the ones scoring high enough are selected as the story’s topics." src="https://cdn-images-1.medium.com/max/1024/2884f65e559ed6527932f01aca16fc5fef2854ab94fd232084627e4aba64b1e3" /><figcaption>a few of our text classification models. All diagrams and charts made by the author</figcaption></figure><h3>Some Clarifications on our Machine Learning policy</h3><p>Before we go deep on this project, I just wanted to clarify a few things about how we stand regarding AI in general.</p><p>Medium has been training internal models with user and post data for a long time now. We train models with specific tasks. For example, models that power our recommendations algorithm, or text classification models like the ones presented in this story. All in the goal to improve our product. With the LLM approach I describe in this story, we ARE NOT sharing these models with other companies. And we ARE NOT allowing anyone to train on our users’ data and content. Here we used Snowflake LLM tools for <strong>inference only (no LLM training was done here) </strong>and they are actually hosting all of the models inside their own infrastructure and guarantee that they are not using any of this for training. Shoutout to the Snowflake team for making it so easy and safe to use LLMs on our data!</p><p>If you want to read more about Medium’s stance on AI, I definitely recommend giving these a read:</p><ul><li><a href="https://medium.com/blog/default-no-to-ai-training-on-your-stories-abb5b4589c8">Default No to AI Training on Your Stories</a></li><li><a href="https://medium.com/blog/finally-an-internet-standard-for-writers-rights-vs-ai-companies-07ffe09f15f8">Finally, an internet standard for writers’ rights vs. AI companies</a></li><li><a href="https://medium.com/blog/we-want-your-feedback-how-can-writers-use-ai-to-tell-human-stories-eb9dee926f2e">We want your feedback: How can writers use AI to tell human stories?</a></li></ul><h3>Problem</h3><p>During our roadmap planning we decided that our NSFW model was out of date and it was time to revamp it. This model labels stories as “Not Safe for Work” if they have sexually explicit content, lots of profanity, or basically anything you wouldn’t want to read on your big monitor in the middle of an open space!</p><p>As you can imagine it’s a pretty important model. We really need it to make sure our most “interesting” content only reaches our most “interested” users and ONLY them!</p><p>There’s actually a funny anecdote from 2021 that was shared internally back then. One of our employee was onboarding the WHITE HOUSE staff onto Medium so that they could start using the platform with the POTUS account. And the first thing that showed up on the homepage was a big picture of well… a butt… So that became of whole thing <strong><em>“Our algorithm is serving erotica to President Joe Biden!!”. </em></strong>That was a brand new account so that was our top story “by default”.</p><p>That’s embarrassing for us and for our users! This anecdote is actually what prompted the recommendations team to create the first version of our NSFW model (so, thank you Joe!).</p><p>We’re now in 2026 and this model is now pretty old. It’s not getting retrained regularly, the code runs on python versions that are extremely old. With turnover and changes in tools and file organization, we lost track of how it was trained and what kind of performance was measured at the time. So this is now an obscure part of the ML stack, in need of a good makeover.</p><h3>LLM based approach</h3><p>The typical way we’d build a binary classification model like this is to build a pipeline where:</p><ul><li>new data is regularly added to a training set by a human labeling team</li><li>and the model is regularly retrained</li></ul><figure><img alt="The typical machine learning setup, in two loops. In the training loop, humans label new data, those labels get added to the training set, the model is retrained on a daily or weekly cadence, and the retrained model is published. In the prediction loop, new data arrives — a story published on Medium — the latest published model makes a prediction on it, and the prediction is stored." src="https://cdn-images-1.medium.com/max/1024/cdd9ce532966bb51dd59598d4e47df84918fd8cbf98a392f729bc6aa0a726787" /></figure><p>So now, what if we replace all that with a really simple LLM based approach?</p><p>With LLMs it’s pretty easy to create whatever text classifier you want. You just need the proper prompt and have some sort of expected output that you can parse from the LLM’s response. And there’s no need for training at all.</p><figure><img alt="The LLM-based approach. Four pieces are assembled into a single prompt: instructions asking whether the post qualifies as NSFW and requesting a 0-to-100 score with an explanation, a set of NSFW guidelines, a handful of labelled examples, and the post’s content. The LLM returns a score and a written explanation. For example:score=12 expalanation=“nothing matches NSFW guidelines”" src="https://cdn-images-1.medium.com/max/1024/099039b8365f50f3ee0ac3e78082290871df1ddecaea50eb119ec432690c19bc" /><figcaption>We build a prompt with all the context needed for the LLM to make a decision. We get back a score between 0 (perfectly SFW) and 100 (undoubtedly NSFW) as well as a short explanation of the score.</figcaption></figure><p>We thought this NSFW model revamp was a great opportunity to give this a try. Are we able to build a NSFW classifier this way, that matches our performance expectations?</p><h3>How we built it</h3><p>The process was pretty simple</p><p><strong>First we built an evaluation dataset:</strong></p><ul><li>we sampled a random sample of stories published on Medium</li><li>then we had our curation team label the dataset (is it NSFW? yes/no)</li><li>as they did that, we also asked them to refine their definition of “NSFW”, and give concrete examples (this was very useful to build the prompt)</li></ul><p>Side note: when we picked the random sampling of stories, we used other signals to make sure we oversample NSFW stories. If we just sample at random, we’d have to sample thousands of stories just to get a handful of NSFW examples. Thanks to this, our evaluation had a nice 50/50 split on NSFW/SFW labels.</p><p><strong>Next step was to build the prompt for the LLM:</strong></p><ul><li>we leveraged all the clarifications that the curators added while labeling the dataset and summarized that into “NSFW guidelines”</li><li>we pass in the story’s contents</li><li>we ask the LLM to output a score from 0 → 100</li><li>and we also ask the LLM to give a short explanation for the score. For example: “graphic description of sexual intercourse”</li></ul><p><strong>Finally we need to evaluate the LLM approach and compare the different model:</strong></p><ul><li>have a few LLMs make their predictions on the evaluation set</li><li>have the legacy model make it’s predictions on the evaluation set too</li><li>and then compare the metrics</li></ul><figure><img alt="How the evaluation works. A dataset of story contents with a human-assigned yes/no NSFW label is run through each candidate model — the legacy model, Claude Haiku, Mixtral 8x7b and others — and each one gets a precision and recall score that can be compared side by side. Most land around 95% on both, while the smaller Claude Haiku shows high precision but much lower recall." src="https://cdn-images-1.medium.com/max/1024/ce45ef8c4c7aae3a67f28ea6ba05f1ad5297a8da2de87f55714c1babd0702b10" /><figcaption>Thanks to our new evaluation dataset, we can compare our legacy model with LLM models</figcaption></figure><p>Great news, after some testing, we found some LLM models that match or outperform the legacy model! We got parity with the legacy model on both false positives and false negatives. Some of the bigger LLMs even outperform the legacy model on both metrics. The mistral models performed really well for us, as well as some of the larger Claude models.</p><p>Performance is key here because there’s big downsides with both false negatives and false positives:</p><figure><img alt="A confusion matrix showing why both kinds of error hurt. When a story is actually SFW but the model flags it NSFW, that’s a false positive: the story doesn’t get distributed even though it should have, and the writer loses out. When a story is actually NSFW but the model flags it SFW, that’s a false negative: it might get distributed to users who don’t expect it — the Joe Biden problem. The other two quadrants, true positives and true negatives, are the correct outcomes." src="https://cdn-images-1.medium.com/max/1024/767b18e9b30b7a8cadf5a5718ad13c257a2b20def19c360e6bf391eea4d0faef" /></figure><p><strong>Now, we need to get an idea of the costs: is that LLM approach going to cost us an arm and leg?</strong></p><p>Something nice with LLMs is that the costs are easy to estimate. Each model has a cost per input token and a cost per output token. Using that and a little bit of back of the enveloppe math you can quickly get a really good cost estimate.</p><figure><img alt="A three-step simplification of the LLM cost equation. The full version — prompt size plus content size, times price per input token, plus output cost, all times the number of posts scored — gets reduced by crossing out the negligible parts. What’s left is prompt size × price per input token × number of posts. That worked out to about $250 a month with Mixtral 8x7b." src="https://cdn-images-1.medium.com/max/1024/1*Mbj9erwiOG3vnLdwWTvVdg.png" /></figure><p>We then picked the model that was the best fit for us in terms of costs and performance. For us that was Mixtral 8×7b (I promise that was a fair trial - nothing to do with the fact that I’m French!). The main way we kept the costs under control is by picking a relatively cheap model, and also by only scoring the stories that are eligible to get distributed in the first place (for example, no need to waste time on stories that have been flagged as spam).</p><p>For offline experimentation and for production classification, we simply used Snowflake. All of our data ends up in Snowflake and they make it super simple to run LLM inference directly inside a SQL query. There’s a complete function available that lets you call an LLM and pick your model as well as the expected response format.</p><p>Here’s the query I used during the evaluation process as an example:</p><pre>set modelName = &#39;claude-haiku-4-5&#39;;<br>set promptName = &#39;raph_test&#39;;<br>set promptVersion = &#39;1.3&#39;;<br><br>set responseFormat = &#39;{<br>  &quot;type&quot;: &quot;json&quot;,<br>  &quot;schema&quot;: {<br>    &quot;type&quot;: &quot;object&quot;,<br>    &quot;properties&quot;: {<br>      &quot;score&quot;:  {&quot;type&quot;: &quot;integer&quot;},<br>      &quot;reason&quot;: {&quot;type&quot;: &quot;string&quot;}<br>    },<br>    &quot;required&quot;: [&quot;score&quot;, &quot;reason&quot;]<br>  }<br>}&#39;;<br><br>with raw_llm_responses AS (SELECT post_id,<br>                                  title,<br>                                  text,<br>                                  SNOWFLAKE.CORTEX.COMPLETE(<br>                                    $modelName,<br>                                    -- build prompt: instructions and post information<br>                                    ARRAY_CONSTRUCT(<br>                                      OBJECT_CONSTRUCT(<br>                                        &#39;role&#39;,<br>                                        &#39;system&#39;,<br>                                        &#39;content&#39;,<br>                                        prompts.prompt<br>                                      ),<br>                                      OBJECT_CONSTRUCT(<br>                                        &#39;role&#39;,<br>                                        &#39;user&#39;,<br>                                        &#39;content&#39;,<br>                                        &#39;&lt;title&gt;&#39; || COALESCE(title, &#39;N/A&#39;) || &#39;&lt;/title&gt;&#39; || &#39;\n&#39; ||<br>                                        &#39;&lt;text&gt;&#39; || COALESCE(text, &#39;N/A&#39;) || &#39;&lt;/text&gt;&#39;<br>                                      )<br>                                    ),<br>                                    OBJECT_CONSTRUCT(<br>                                      &#39;temperature&#39;, 0,<br>                                      &#39;response_format&#39;, PARSE_JSON($responseFormat)<br>                                    )<br>                                  ) AS llm_raw_response<br>                           FROM posts_to_evaluate<br>                                  join medium.ml.nsfw_classifier_prompt as prompts<br>                           where prompts.name = $promptName<br>                             and prompts.version = $promptVersion),<br><br>     -- parse results from the returned JSON<br>     llm_responses_parsed AS (SELECT post_id,<br>                                     title,<br>                                     text,<br>                                     llm_raw_response,<br>                                     llm_raw_response:structured_output[0]:raw_message AS parsed_json,<br>                                     TRY_CAST(parsed_json:score::STRING AS INT)        AS score,<br>                                     parsed_json:reason::STRING                        AS score_reason,<br>                                     parsed_json::STRING                               AS response_as_string<br>                              FROM raw_llm_responses)<br><br>SELECT post_id,<br>       score              AS prediction,<br>       score_reason,<br>       response_as_string as raw_response,<br>       $modelName         AS model_name,<br>       $promptName        AS prompt_name,<br>       $promptVersion     AS prompt_version<br>FROM llm_responses_parsed;</pre><p>In our tech stack, it’s really easy to setup a simple job that runs a Snowflake query on a schedule. <strong>We just leveraged that and we were able to successfully ship that model very quickly.</strong></p><h3>And now repeat?</h3><p>Once that new model was up and running, we were really interested in applying that same formula to our other models. We first sat down to weigh the pros and cons of that new LLM based approach, here’s how we broke it down:</p><figure><img alt="A table comparing a custom ML model with an LLM. The custom model needs a training set, adapts automatically as new labels come in, and is much cheaper — but shipping a new version is hard and it offers no explainability. The LLM needs only an evaluation set, is easier to iterate on and somewhat explainable, but is more expensive and frozen in time, so the prompt and evaluation set need maintaining." src="https://cdn-images-1.medium.com/max/1024/fdb02663c14c90798aec78a6e0508f0aaf9a135219209a8f63f4ed2e7d3355e2" /></figure><p><strong>Generally the LLM approach is:</strong></p><ul><li>easier to setup</li><li>more expensive</li><li>less adaptive</li></ul><p>One thing to keep in mind too is that LLMs are “stuck at a point in time”. They have been trained at a certain date and they don’t know anything about what happened after that date. So they will not be adapting to new trends and performance is likely to drop over time. Depending on the use case, you might want to upgrade your model every now and then.</p><p><strong>The NSFW model was the perfect use case:</strong></p><ul><li>we don’t have a proper training set for NSFW content. And it would be costly to create and maintain one</li><li>we don’t feel like we need to be super reactive to new trends. I don’t really have data to support that, this is more a judgment call: we think that NSFW stories 6 months from now will be similar to the ones that are published today.</li><li>costs are low as long as we only score a portion of the stories published on Medium. Here we just need to score the stories that are eligible for distribution in the first place</li></ul><p><strong>All in all, that model revamp was a great success:</strong></p><ul><li>we have a brand new NSFW model</li><li>it performs really well</li><li>we don’t need to maintain a training set</li><li>it’s cheap to run</li><li>and it’s really simple in terms of engineering</li></ul><p>What about our other models then?</p><figure><img alt="Two notes on our other models. For the topic model, the dataset is free, and switching to an LLM would mean no longer adapting to new trends. For the spam model, an LLM would be expensive because there are so many posts to score, and a static LLM plus a fixed prompt is a problem in a cat-and-mouse game." src="https://cdn-images-1.medium.com/max/1024/16282e84bca7bea052a33fb776f18382b226fb4bbc9e34d621641cf372fd59af" /></figure><p>It turns out that the NSFW model is maybe our only model that’s a good use case for an LLM based approach:</p><ul><li>for our topic model, we get our dataset for free (we just train on the topics that users are adding to their stories). This means that we automatically adapt to new trends, which is really nice. Switching to an LLM based approach would feel like a downgrade</li></ul><figure><img alt="A Medium story with its topic tags highlighted at the top" src="https://cdn-images-1.medium.com/max/1024/096201a4b36fc95caa03b02ae32aec4b0a279c8c7131f8b725413db4a72f082b" /><figcaption>writers add topics to their stories and we leverage that to build a training dataset “for free”. The model automatically adapts to new trends as new stories get posted on Medium</figcaption></figure><ul><li>for the spam model, we also want to adapt to new trends <strong>quickly</strong>. Spammers are always trying new things and they are always trying to trick the platforms to get visibility. So in this case we are willing to pay the cost of maintaining a training set, augmented with new data every day. Also since we want to score ALL the stories posted on Medium, it becomes more complex and costly to use an LLM based approach</li></ul><figure><img alt="A Medium story titled “How to Keep a Glass Wine Cellar From Looking Like a Display Case,” tagged Architecture, Wine and Design. The writing is bland and generic, and the first sentence contains a link" src="https://cdn-images-1.medium.com/max/1024/83514975a67f2a08ae5f48828f8038b77efbca39368e1220ff98cbbbb7e4c311" /><figcaption>some freshly caught spam (SEO hacking)</figcaption></figure><h3>What about new use cases?</h3><p>Even though that doesn’t work really well for our other existing models. It is still very exciting because it makes it really easy to spin up new text classification models. What if we wanted to give users more control on their content, like “allow erotica” but do not allow “true crime”? We’re a text-based platform, so the possibilities are endless!</p><h3>Some side notes</h3><h4><strong>Why ask for scores and not just a yes/no value?</strong></h4><p>With scores, you are able to rank the predictions and it means that you can use metrics that quantify how well you separate the NSFW stuff from the SFW stuff (metrics like ROC-AUC or PR-AUC). It also means that you can set the threshold wherever you like and you can change that in the future too - letting you choose the balance of false positives and false negatives that suits you best.</p><figure><img alt="The same six stories ranked by NSFW score, shown with two different thresholds. At a threshold of 90, only the top story is flagged, so two genuinely NSFW stories slip through as false negatives. At a threshold of 30, almost everything is flagged, so a perfectly SFW story gets caught as a false positive and won’t be distributed. Scores let you move that line wherever you want." src="https://cdn-images-1.medium.com/max/1024/a28f14b2b086ae74b87b2405d7c313b3c4a0c2c0115d13d845f144f27385f158" /></figure><h4>Why ask for a short description?</h4><p>Our end goal is just to know if a story is SFW or not, we don’t actually need the LLM to explain it’s decision. And it does add a little to the inference costs (the output becomes longer). But in our use-case it was negligible in the overall costs.</p><p>I had to do a lot of spot checking and debugging on individual stories for this project. (I learned a lot of things 😳). And having the descriptions alongside the score was super useful for that. It’s also a great thing to have now that this is production, and it can also be used to break down NSFW stories into categories. That can be useful for the curation team or the trust and safety team</p><figure><img alt="A results table with three columns: story title, prediction, and reason. Four stories scored 90 or 95, each with a one-line explanation from the model — “explicit sexual content in one”, “selling illegal drugs online”, “erotic content” for the two other examples." src="https://cdn-images-1.medium.com/max/1024/32710f156963e1e8a59007feccfab42a09032e8160051ec8e1fb906f6b84abde" /><figcaption>spot checking and debugging is easier with a short explanation</figcaption></figure><figure><img alt="A bar chart, “Distribution of NSFW posts by high level category.” “Explicit sexual content” is by far the largest share at around 37%, followed by “violence and gore” at roughly 12% and “erotica” at about 10%. “Drugs”, “pornography” and “hate speech” each account for a few percent or less." src="https://cdn-images-1.medium.com/max/1024/ef1ab231c42f32208336c21319df4aecf123e9f920010b371cff322ea0f97bde" /><figcaption>asking for a description lets you break things down nicely</figcaption></figure><p><strong>To wrap this up, it was really fun to build this model and we’re excited to build new features with this new tool in our toolbox! We’re hoping this opens up new possibilities for where our product teams can take Medium next and helps us make sure readers find more stories they love, and writers get more readers that love their work!</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=89a185f79b68" width="1" height="1" alt=""><hr><p><a href="https://medium.engineering/how-we-think-about-text-classification-in-the-llm-era-89a185f79b68">How we think about text classification in the LLM era</a> was originally published in <a href="https://medium.engineering">Medium Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Engineering stories behind the Medium Daily Digest Algorithm: Part 4]]></title>
            <link>https://medium.engineering/engineering-stories-behind-the-medium-daily-digest-algorithm-part-4-ec7136f21acd?source=rss-7e93bd9fc8de------2</link>
            <guid isPermaLink="false">https://medium.com/p/ec7136f21acd</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[recommendation-system]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Raphael Montaud]]></dc:creator>
            <pubDate>Wed, 03 Sep 2025 11:31:38 GMT</pubDate>
            <atom:updated>2025-09-03T11:31:38.188Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OfHQPW0iVi3UVpfnk_fPqQ.jpeg" /></figure><h4>Cross-Digest diversification</h4><p>In this part 4, we’ll see how we went from investigating a few complaints from digest power users to improving our digest recommendations across the board.</p><blockquote><strong>Intro</strong>: This is a 4-part series breaking down improvements to the algorithm behind the Medium’s Daily Digest over the past year. When we started this work, the Digest was suboptimal — and since it’s a huge distribution surface, reaching millions of readers every day, we started working on incremental improvements.</blockquote><blockquote>By the end of these projects, the digest was 10% more likely to convert users to paying members, less expensive to run, more flexible and easier to maintain and it’s now providing higher quality recommendations for all our users, including our “power readers”.</blockquote><blockquote>This is told through the lens of our engineering team tackling a series of challenges one by one. Medium has a small team but we operate on a big scale. We’re working our way through some technical debt and at the same time, striving to provide the best experience for our readers. This is the source of many interesting challenges.</blockquote><blockquote>I hope this series helps you understand how the recommendations algorithm work and can help others who are facing similar technical challenges.</blockquote><h3>Some Concepts</h3><p>Here’s a little cheat sheet with some concepts you may need to follow along this story</p><figure><img alt="Hand-drawn cheat sheet explaining Medium’s platform and recommendation system. Shows how Medium curates content through a 3-stage process: Source (pulls stories from various sources), Filter (removes duplicates/already read), and Rank (scores stories to predict user interest). Includes 4 recommendation surfaces: Daily Digest email, Homepage feed, push notifications, and post-reading suggestions called “Recirc.”" src="https://cdn-images-1.medium.com/max/1024/1*9V1FpdLmgfQgEc8YDA9xFQ.png" /><figcaption>You may need this to understand the rest of this post</figcaption></figure><h3>User Complaints</h3><p>After we shipped all the changes mentioned in the previous installments of this series, we started seeing some support tickets coming in related to the digest:</p><figure><img alt="Two user complaint messages titled “User Complaints.” First from Daniel complains about daily newsletter “regurgitating the same articles over and over again” across various authors and content domains. Second from Omar notes recommendation quality has “fallen off dramatically” with repeated recommendations and asks if Medium changed their algorithm, requesting they revert to earlier version." src="https://cdn-images-1.medium.com/max/1024/1*GLhDqy_9lj6rgRafuU3SBg.png" /><figcaption>User complaints that started off our investigation</figcaption></figure><p>I think we should appreciate the level of thoughtfulness our users put in those support tickets. We review those carefully and we take pride in reading and answering every support tickets.</p><p>Those were forwarded to the recommendation team and we immediately thought that maybe we oversteered to much when we removed some of the filtering rules for the digest.</p><p>User digests were too repetitive and the issue was particularly noticeable for our digest power users. We started investigating the issue in search for a quick solution.</p><h3>The Problem</h3><p>As we’ve seen in <a href="https://medium.engineering/engineering-stories-behind-the-medium-daily-digest-algorithm-part-1-909a7ca5e807"><strong>Part 1: How we made our email story recommendations better</strong></a> there’s a few things that are in place to make the digest less repetitive:</p><figure><img alt="Diagram showing Medium digest story eligibility rules. “Digest Stories” box contains two items: “rank 1 story” with arrow pointing to “Never eligible again for the digest” and “rank 2 to 15 stories” with arrow pointing to “Not eligible for the next 4 days.” Shows how top story gets permanent exclusion while others get temporary 4-day exclusion." src="https://cdn-images-1.medium.com/max/1024/1*CrHn-hhmwjeTbiTSCAxs8g.png" /><figcaption>A digest contains 15 stories. The first one is used as the email subject, which is why it’s not eligible for the digest anymore</figcaption></figure><figure><img alt="Cross-digest diversification diagram with two sections. Left shows “Filtering Rules”: Digest Title Filter (excludes posts used in previous email subjects) and Digest Backoff Filter (excludes posts from last 4 days’ digests). Right shows “Organic Evolution of recs”: Newly Published Stories and User Profile Evolution explaining how changing content and evolving user preferences naturally diversify recommendations." src="https://cdn-images-1.medium.com/max/1024/1*BaQbRpLUWmtcbfvrhicYAA.png" /><figcaption>Filtering rules and organic evolution of recs is what makes every digest different</figcaption></figure><p>We dug into the data and found that for the vast majority of users the issue was minimal, barely noticeable. For the vast majority of users, digests are composed mostly of stories that have never been sent in a prior digest.</p><p>But for the users who were complaining, the issue was very visible. <strong>Some posts got sent up to 8 times to the same user!</strong> On average, for those users, posts in a given digest had been sent twice in a prior digest. So there was a massive issue with those users digests, seriously damaging their experience.</p><h3>The fix</h3><p>Fixing this is a little bit tricky. As we’ve seen in <a href="https://medium.engineering/engineering-stories-behind-the-medium-daily-digest-algorithm-part-1-909a7ca5e807"><strong>Part 1: How we made our email story recommendations better</strong></a> there’s no way to tell if a user has truly opened their daily digest email because of the Apple Mail Privacy Protection.</p><p>If we assume that all emails are “opened” and do some filtering based on that, we will undo all the gains we got in <a href="https://medium.engineering/engineering-stories-behind-the-medium-daily-digest-algorithm-part-1-909a7ca5e807"><strong>Part 1</strong></a>. That’s some big wins that we’re not ready to say goodbye to.</p><p>So how do we get out of this situation? There is one thing that we are able to track and that we haven’t leveraged yet. When a reader clicks on a story in their email digest, they navigate to Medium and we report the ID of the email they came from. We can use that to our advantage. Whenever a user clicks a story in a digest, we can be sure that they actually opened the email.</p><p>We decided to put some logic in place to keep track of which digest emails have at least one post that was clicked.</p><figure><img alt="Digest Database flowchart showing user action “User Reads a Post” leading to decision diamond “Is the referrer a digest email?” If yes, updates Digests Database (emailID, sentAt, hasPostRead columns) by setting hasPostRead=true. If no, leads to “Do Nothing.” Database shows sample entries with dates from July 2025 and true/NA values for hasPostRead field." src="https://cdn-images-1.medium.com/max/1024/1*SAdlXaO1jI32oIEh_pmi4g.png" /><figcaption>Links to posts in emails are engineered so that we can keep track of the emailID when the user navigated to Medium</figcaption></figure><p>So now we have a way to list some digests that the user has “definitely opened” with the posts that they contained.</p><p>With that new information we can set up a new filter in our recommendation stack. When building the feed, we fetch all of those “definitely opened” digests for this user and we count total occurrences of each post.</p><p>We can then filter out posts above a certain threshold. That rule would translate to something like:</p><ul><li>if we have sent this story in more than x digests that the user has “definitely opened”, then we will never send it to that user again</li></ul><figure><img alt="HasPostRead Digest Filter diagram showing recommendation algorithm querying Digests Database for “user_a with hasPostRead = true.” Database table shows userID, sentAt, hasPostRead, and postIDs columns. Algorithm builds counts for each post, then applies threshold filtering to exclude posts that appear frequently in opened digests." src="https://cdn-images-1.medium.com/max/1024/1*mo5lgwFYXOy32prr88H2hw.png" /><figcaption>When building the feed, we can look at the previously sent digests to create a filtering rule based on the <strong>hasPostRead</strong> attribute</figcaption></figure><p>In order to find the right threshold, we ran some tests to measure the impact on the user experience. For different user samples, we computed how many posts would be filtered out from their digest with this new rule, depending on the value of the threshold:</p><figure><img alt="Data table showing threshold experiment results with three threshold values (1, 2, 3) and three user groups. Shows average values for Random Sample Free Users (0.15, 0.03, 0.01), Random Sample Member Users (1.44, 0.47, 0.22), and Users who complained (4.6, 1.5, 1.1). Higher thresholds generally show lower values across all groups." src="https://cdn-images-1.medium.com/max/1024/1*jJ-hbsl2WZlcZy8TRi-iTg.png" /><figcaption>Number of posts filtered from the digest depending on the threshold</figcaption></figure><p><strong>Here’s the takeaways from those results:</strong></p><ul><li>free users digests are barely affected by this new filtering rule. Whatever the threshold, less than 1 story would be filtered out from their digests (a digest is 15 stories) if we started applying that filter today</li><li>members would be moderately affected with a threshold of 1. With a higher threshold they would barely be affected</li><li>complaining users would be greatly affected by this change with a threshold of 1, with almost 5 stories filtered out from each of their digests. It seems like that would introduce enough diversification to fix their issues with repetitive digests</li></ul><p>So that test helped us decide that a threshold of 1 is what we’re looking for and it gave us confidence that this a solution worth exploring:</p><ul><li>it seems to fix things for users who had an issue</li><li>it has a marginal impact on the majority of users</li></ul><p>Also, we’re leveraging new, reliable data in our recs algorithm so there’s chances that this will improve recommendations overall.</p><p>We decided to put it to the trial in an A/B test</p><h3>Results</h3><p>Results for this A/B test were really good, as <strong>we observed a statistically significant increase in reading time</strong> for the users in the experiment. So it seems like this had a positive impact on the experience of all our users overall, not just the small percentage of users that were the most impacted. Members had a bigger increase in reading time which makes sense as they were more impacted according to our tests from the previous section.</p><p>We didn’t see any complaints on digest diversification since we shipped this and we’re pretty happy about the new state of things. We love that our users are so open and ready to share their feedback with us — this helped us build a better product for all Medium users.</p><p>This is a new “hard filter” which is not necessarily the best approach here. It’s likely that turning this into a soft filter (CF <a href="https://medium.engineering/engineering-stories-behind-the-medium-daily-digest-algorithm-part-3-6fbf1512e6e6"><strong>Part 3: Hard vs Soft Filtering and how this applies to Medium’s Recommendation System</strong></a>) would yield better results.</p><p>Another possible element to explore is the position of stories in the digest. It’s important because the top stories are the first to be displayed, and it’s possible that a user didn’t scroll the entire email and so didn’t get a chance to see the posts at the bottom of the digest. We could imagine some new rules leveraging the position in the email to make this filter more fine-grained.</p><p><strong>Thank for you for following this series till the end, I hope this helped you understand a few concepts.</strong></p><p>Overall these projects helped our user read more and made them more likely to convert to paying members. Our algorithm is also less expensive to run, more flexible and easier to maintain and it’s now providing higher quality recommendations for all our users, including our “power readers”.</p><p>I hope this helped understand a few concepts as well as our recommendation systems and how we try to improve it step by step. Hopefully that gave you some ideas to improve you own systems too!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ec7136f21acd" width="1" height="1" alt=""><hr><p><a href="https://medium.engineering/engineering-stories-behind-the-medium-daily-digest-algorithm-part-4-ec7136f21acd">Engineering stories behind the Medium Daily Digest Algorithm: Part 4</a> was originally published in <a href="https://medium.engineering">Medium Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Engineering stories behind the Medium Daily Digest Algorithm: Part 3]]></title>
            <link>https://medium.engineering/engineering-stories-behind-the-medium-daily-digest-algorithm-part-3-6fbf1512e6e6?source=rss-7e93bd9fc8de------2</link>
            <guid isPermaLink="false">https://medium.com/p/6fbf1512e6e6</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[recommendation-system]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Raphael Montaud]]></dc:creator>
            <pubDate>Mon, 01 Sep 2025 11:31:38 GMT</pubDate>
            <atom:updated>2025-09-03T17:02:02.359Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yOR2ymyOFAsIzWJbUT508g.jpeg" /></figure><h4>Hard vs Soft Filtering and how this applies to Medium’s Recommendation System</h4><p>In this part 3 we’ll see how we modified one of our hard filtering rules and attempted to turn it into a machine learning based “soft filter”.</p><blockquote><strong>Intro</strong>: This is a 4-part series breaking down improvements to the algorithm behind the Medium’s Daily Digest over the past year. When we started this work, the Digest was suboptimal — and since it’s a huge distribution surface, reaching millions of readers every day, we started working on incremental improvements.</blockquote><blockquote>By the end of these projects, the digest was 10% more likely to convert users to paying members, less expensive to run, more flexible and easier to maintain and it’s now providing higher quality recommendations for all our users, including our “power readers”.</blockquote><blockquote>This is told through the lens of our engineering team tackling a series of challenges one by one. Medium has a small team but we operate on a big scale. We’re working our way through some technical debt and at the same time, striving to provide the best experience for our readers. This is the source of many interesting challenges.</blockquote><blockquote>I hope this series helps you understand how the recommendations algorithm work and can help others who are facing similar technical challenges.</blockquote><h3>Some Concepts</h3><p>Here’s a little cheat sheet with some concepts you may need to follow along this story</p><figure><img alt="Hand-drawn cheat sheet explaining Medium’s platform and recommendation system. Shows how Medium curates content through a 3-stage process: Source (pulls stories from various sources), Filter (removes duplicates/already read), and Rank (scores stories to predict user interest). Includes 4 recommendation surfaces: Daily Digest email, Homepage feed, push notifications, and post-reading suggestions called “Recire.”" src="https://cdn-images-1.medium.com/max/1024/1*9V1FpdLmgfQgEc8YDA9xFQ.png" /><figcaption>You may need this to understand the rest of this post</figcaption></figure><h3>Conference time</h3><p>Back in 2024, Leigh and I were in Bari, Italy for the annual RecSys conference, an international conference around Recommender Systems. In between some panzerottis and capuccinos in the old town patios we managed to go to a talk or two.</p><figure><img alt="Two-panel photo showing hands holding coffee items. Left panel shows hands holding a small white coffee cup. Right panel shows hands holding a white coffee cup from above, revealing a very short dark espresso" src="https://cdn-images-1.medium.com/max/1024/1*ViWxObDb3i3SfYtBBrr1FQ.png" /><figcaption>surprisingly small cups of coffee. And surprisingly small amounts of water</figcaption></figure><figure><img alt="Conference venue entrance with blue banner reading “18th ACM Conference on Recommender Systems” in Bari, Italy, October 14–18 2024. Modern building interior visible with attendees in background. Small fluffy dog sits in foreground outside the venue entrance." src="https://cdn-images-1.medium.com/max/1024/1*Q-QN4kylcNrKHPC3K3bKVg.png" /><figcaption>Nala (my dog) attending the conference</figcaption></figure><figure><img alt="Small fluffy cream-colored dog sitting at outdoor café table sniffing a plate of pasta on the table." src="https://cdn-images-1.medium.com/max/1024/1*BJZEaUdUULNzDk6w5SNNtA.png" /><figcaption>Nala realizing she’s been given her own plate of orecchiette alla bolognese</figcaption></figure><p>It had been a while since we went to a conference and it was a great way to get our heads out of the day-to-day grind and focus on some big picture stuff.</p><p>There were many interesting talks but one that stood out for us was Chris Johnson’s. He gave some insights on the recommendations algorithm at Indeed and mentioned something about hard vs soft filtering that resonated with us.</p><h3>Hard Filtering</h3><p>In recommendations systems, you want to make sure you filter out some items from a user’s recommendation for many different reasons.</p><p><strong>For example:</strong></p><ul><li>a Youtube video that you have already watched</li><li>a Medium story from a writer that you have muted or blocked</li><li>a Job posting that doesn’t match your requirements</li></ul><p>Those filtering rules are hard filters. It’s an “all or nothing” scenario. If the (user, item) pair passes a certain condition, then it will be filtered out.</p><p>A failure in one of those filters would be considered a bug and users would probably report it.</p><p>But there are some filtering rules that are not associated to a “feature”. They are just rules that are in place because we think they make the recommendations better.</p><p><strong>For example at Medium:</strong></p><ul><li>any story that we’ve presented to you 3 times or more in a feed is not eligible to be recommended to you anymore (we call that our “Presentation Filter”)</li><li>in certain feeds like the “Trending” feed, stories past a certain “age” are completely removed (we call that our “Old Filter”)</li></ul><figure><img alt="Table comparing filters with “Is it a feature?” column. Lists 5 filters: Read Filter (filters already read posts) — Yes, Muted Filter (filters muted writers’ posts) — Yes, Presentation Filter (filters posts shown 3+ times) — No, Old Filter (filters posts over x months old) — No, Digest Title Filter (filters posts used in previous email subjects) — Yes." src="https://cdn-images-1.medium.com/max/1024/1*_jQb8-1gljUJTgydN3sUKQ.png" /></figure><p>Chris Johnson’s point (at least my understanding of it) is that, as much as possible, the <strong>hard filters should be associated with a “feature”</strong> (ie a user expectation or a product specification). The other filters should be transformed into soft filters.</p><h3>So what’s a soft filter?</h3><p>Instead of a yes or no rule, a soft filter applies a rule in a more continuous way.</p><p>To illustrate that, let’s say we want to do a “trending” feed showing all the stories that are trending on Medium. We want this feed to show recently published stories that are popular. <strong>How can we create that fresh and trending experience for our users?</strong></p><p><strong>To build that feed we need two things:</strong></p><ul><li>pick a pool of stories that are eligible to show up in the feed</li><li>find a way to rank them so that we can select the “top 10” that will be displayed to the user</li></ul><p><strong>A. Trending feed with a hard age filter:</strong></p><p>One way to go about it is to say:</p><ul><li>we want only recent stories so we’ll select only stories published in the last 7 days<br> → NB: this is a hard filter: the story is either recent enough or too old</li><li>we want the most popular stories so we’ll rank them by the total number of claps they have received, and select the top 10</li></ul><p><strong>Now you have 10 recent, popular stories on Medium which will create a “trending feed” experience for the user.</strong></p><p>This works nicely but it has a few blind spots:</p><ul><li>we will prefer a story with 700 claps that’s 6 days old from a story that’s 2 days old but has 600 claps — which is not great because the one with 600 claps is clearly more promising and arguably more “trending”</li><li>a story that has 2k claps and that’s 8 days old will be excluded by our hard filter, although we could argue this one is very “trending”.</li></ul><p>So with this hard filter rule there’s a big bias when we build our feed. It’s not fair to stories that were published very recently, and it’s not fair for stories that are just above the threshold.</p><p><strong>B. Trending feed with a soft age filter:</strong></p><p>To counter those blind spots we can implement a soft filter on the age of posts. The idea is to include the age of posts in our ranking formula in a continuous way.</p><p>Previously we were ranking just based on score = number_of_claps. Let’s see what happens if we decide to rank stories based on score = number_of_claps - 100 * age_of_story_in_days instead.</p><figure><img alt="Hard vs Soft Filtering comparison showing three columns: original post list (Posts A-D with claps/age), Hard Filter that removes Post C for being “too old” (crossed out), and Soft Filter that ranks posts by “claps minus 100 times age” formula, reordering them by calculated scores (1200, 400, 100, 50) while keeping all posts." src="https://cdn-images-1.medium.com/max/1024/1*_v7xRpR9GE-Db5Lex9kuMg.png" /></figure><p>For the same number of claps, a story’s score will go down as it ages. It will therefore start ranking lower and lower compared to other “newer” stories. Eventually, even a recently published story with zero claps will outperform it. That makes sense, users come to see the “trending” feed to see recent stories.</p><p>In the ranking A, it’s a yes or no function: it’s either young enough or too old: <strong>that’s what we call a hard filter</strong>. In the ranking B it’s slowly going down in the ranks as it ages, <strong>that’s what we call a soft filter</strong>.</p><p>In practice we don’t always have to design formulas like that. We can use a machine learning algorithm to predict a score for each (user, post) pair and have this score act as a soft filter. Let’s see how this works in the next section.</p><h3>Machine Learning model as a soft filter</h3><p>Let’s dig in a little bit on machine learning models and how we can use them as soft filters.</p><p><strong>Our Machine Learning ranking algorithms can be summarized like so:</strong></p><ul><li>they have access to information on the user and the post</li><li>their goal is to predict the likelihood of certain events (eg: likelihood that you will click on the story preview, likelihood that you will clap the story, etc)</li><li>they rely on historical data to train and adjust their predictions</li></ul><figure><img alt="Machine Learning Ranking diagram showing post data (700 claps, python/ML tags, 17 days old) and user data (follows python/design, 6 months old) feeding into ML Model. Model outputs likelihoods (Click: 35%, Clap: 5%, Dislike: 1%) multiplied by weights (Click: 1, Clap: 30, Dislike: -100) to produce final Score: 0.85." src="https://cdn-images-1.medium.com/max/1024/1*HSmzx-RPL5hMOmZQjljk2g.png" /><figcaption>We trained our model so that it can be fed user and post information and send back some event likelihoods. We associate a weight to each event. The weight is the translation of “how bad we want this to happen”. A click is good, but a clap is much better. A “dislike” is really bad.</figcaption></figure><p>So we don’t have to filter out all the stories past a certain age, or to create handmade rules to prioritize more recent stories. We can rely on our Machine Learning model to learn the patterns and interactions between the different variables. The model should be able to sort out the impact of the post age by learning from the historical data.</p><p>And it should then be able to predict how that affects the different likelihoods. All things being equal, increasing the “age” value of the post should decrease the score. This means that the Machine Learning ranking step can act as a soft filter.</p><figure><img alt="Machine Learning soft filtering comparison showing two identical posts (700 claps, python/ML tags) with same user (follows python/design, 6 months old). Left post is 17 days old with positive Score: 0.85 (green). Right post is 780 days old with negative Score: -4.6 (red), demonstrating how ML ranking acts as soft filter by scoring rather than removing content." src="https://cdn-images-1.medium.com/max/1024/1*a4xT4we8LV_Kf8CXxfX8DA.png" /><figcaption>All things being equal, increasing the age of a post should decrease the score. The machine learning model is therefore acting as a soft filter.</figcaption></figure><p>This “soft filter” approach has many advantages in theory. There are more stories available at the ranking step, which means that the ranker has more leeway. It might be able to find “old gems” that wouldn’t make it into user feeds otherwise. The model has more possibilities and more information and so it should be able to produce better recommendations.</p><p>If we go back to our previous “Trending feed”, we had excluded a potentially great story with 2k claps just because it was one day older than the limit, this wouldn’t happen with a soft filter. Also we don’t have to choose and tune a maximum age limit (is 7 days the best limit? should we do 3 days, 15 days??). With a machine learning ranking we can just let the model prioritize recommendations based on the metrics we care about.</p><figure><img alt="Hard Filter diagram showing ranked post list with Post C (2000 claps, 8 days old) crossed out in red with annotation “Post C is too old.” Three remaining posts shown: Post A (700 claps, 6 days), Post B (600 claps, 2 days), Post D (150 claps, 1 day). Bottom text reads “Ranked by number of claps, no more than 7 days old.”" src="https://cdn-images-1.medium.com/max/889/1*qgwHGGsm-DArNkYhUl6Jnw.png" /><figcaption>post C is potentially a great recommendation but was excluded because of the hard filter</figcaption></figure><h3>Which hard filters can we convert into a soft filter?</h3><p>Considering all this, we went back to go through our different rules, trying to see which ones of those hard filters could be turned into a soft filters. We thought our “Presentation Filter” was a good candidate for that. This filter is a hard rule to remove any posts that have been presented three times prior to that same user.</p><figure><img alt="Diagram showing Medium feed presentation tracking. Left side shows user feed with three articles (resumes, travel, reading topics) labeled as post_a, post_b, post_d. Arrows show each presentation increments a counter (+1). Right side shows “Presentation Counter” tracking counts for each user-post pair, connected to recommendation algorithm applying “Presentation Filter” to exclude posts shown 3+ times." src="https://cdn-images-1.medium.com/max/1024/1*Ap1enCqRVv1ofHoXziOq6Q.png" /><figcaption>We keep track of how many times posts were presented to a user in a feed. When we build a new feed we make sure to exclude stories that were already presented several times</figcaption></figure><p>So the idea is that we should remove that hard rule and delegate the decision to the machine learning algorithm. Like your friend that still can’t believe you haven’t watched Friends yet, it may keep recommending the same thing over and over again, if it thinks it’s a really good recommendation. Or maybe not, and it will know after a single presentation that it’s time to stop.</p><h3>The experiment</h3><p>To test our theory we performed an A/B/C test. This was particularly simple because our Machine Learning ranking model already has access to the right information (the previous number of presentations). So all we needed was to test different values for our hard filter threshold:</p><ul><li>control kept the threshold at 3 previous presentations</li><li>experiment group A got a threshold at 5. More leeway, but keeps a hard limit</li><li>experiment group B got a threshold at 10. That’s almost infinite leeway for the machine learning model. We just kept the hard rule at 10 to prevent any edge case behaviour that could be particularly annoying for the end user.</li></ul><p>Results came in and they were …. disappointingly flat.</p><ul><li>users average reading time stayed mostly flat</li><li>conversions to paying members also stayed mostly flat</li></ul><p>We did see an increase in the average number of “prior presentations”. That means that we had more “repeat” recommendations, ie the model does think that it’s worth it to present the same stories several times to the same user. But that didn’t have a big enough impact on the users to show in the metrics.</p><p>So unfortunately we were not able to conclude further than this.</p><p>But that did leave us free to decide with what felt best from a product perspective. We decided to ship a threshold at 5 presentations of the same story to a reader.</p><p><strong>This means that:</strong></p><ul><li>we still apply a hard filtering rule</li><li>but it does give more leeway to the model (between 0 and 5 presentations, the model is free to decide which stories are worth recommending again to a user). Although it didn’t show up in the metrics in this experiment, future iterations of the model may take advantage of that extra degree of freedom.</li></ul><p>A threshold at 10 is what should give the best results in theory. But we thought it was too high and it doesn’t feel right to let the algorithm recommend the same story to a user more than 5 times. We could argue that it’s a “feature” that we don’t show the same recommendation to a user above that threshold. (At some point you need to give up and accept that your friend will never watch Friends and sadly miss out on the best show ever.)</p><figure><img alt="Table comparing presentation filter thresholds with trade-offs. Shows three options: 3 presentations (control) with minimum freedom/annoyance, 5 presentations (experiment A) with average freedom/annoyance that “feels right”, and 10 presentations (experiment B) with maximum freedom but maximum risk of being annoying." src="https://cdn-images-1.medium.com/max/1024/1*kWZm8-pEucWuAVsj4ihHFA.png" /></figure><p>Even though we didn’t get a big win here, the overall reflection on our filtering rules was super interesting and will stick with us. In future work we’ll make sure to have the hard vs soft discussion when we think about implementing new filtering rules.</p><p>A next step might be to re-evaluate our hard filtering on the age of stories. We are still using this on some recommendation surfaces. Although everything is in place for the soft filtering to apply on the age of posts, our model often gets it wrong and recommends old stories that are not relevant anymore. Our readers are (rightfully) very vocal about those out-of-date recommendations and so we need to maintain that hard filtering on certain recommendation surfaces.</p><p>Youtube for example is fantastically good at recommending “good old stuff”, sometimes surprising me with 7 or 10 years old videos, that gives a good objective for our team. It would be amazing to get better at this and be able to dig out all of the amazing stories that were published on Medium over the years.</p><p><strong>Thanks for reading this series, you can stay tuned for the next installments of this series by following the </strong><a href="https://medium.engineering/"><strong>Medium Eng Blog</strong></a><strong>. In the </strong><a href="https://medium.engineering/engineering-stories-behind-the-medium-daily-digest-algorithm-part-4-ec7136f21acd"><strong>Part 4</strong></a><strong> we’ll explore how we made digests more engaging by diversifying the daily emails</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6fbf1512e6e6" width="1" height="1" alt=""><hr><p><a href="https://medium.engineering/engineering-stories-behind-the-medium-daily-digest-algorithm-part-3-6fbf1512e6e6">Engineering stories behind the Medium Daily Digest Algorithm: Part 3</a> was originally published in <a href="https://medium.engineering">Medium Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Engineering stories behind the Medium Daily Digest Algorithm: Part 2]]></title>
            <link>https://medium.engineering/engineering-stories-behind-the-medium-daily-digest-algorithm-part-2-c977ad0b134f?source=rss-7e93bd9fc8de------2</link>
            <guid isPermaLink="false">https://medium.com/p/c977ad0b134f</guid>
            <category><![CDATA[database]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[recommendation-system]]></category>
            <category><![CDATA[bloom-filter]]></category>
            <category><![CDATA[dynamodb]]></category>
            <dc:creator><![CDATA[Raphael Montaud]]></dc:creator>
            <pubDate>Thu, 28 Aug 2025 11:31:36 GMT</pubDate>
            <atom:updated>2025-09-09T15:43:13.996Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mpEOt9tgqfJVgyNYZFT3vg.jpeg" /></figure><h4>How we made our filtering 10x cheaper by removing our Bloom Filters</h4><p>Bloom Filters are great tools to make fast and cheap filtering. They also come with plenty of problems and can easily get expensive and cumbersome. We switched to user-based direct database queries, which made our filtering cheaper and easy to maintain. Here’s the full breakdown of that migration.</p><blockquote><strong>Intro</strong>: This is a 4-part series breaking down improvements to the algorithm behind the Medium’s Daily Digest over the past year. When we started this work, the Digest was suboptimal — and since it’s a huge distribution surface, reaching millions of readers every day, we started working on incremental improvements.</blockquote><blockquote>By the end of these projects, the digest was 10% more likely to convert users to paying members, less expensive to run, more flexible and easier to maintain and it’s now providing higher quality recommendations for all our users, including our “power readers”.</blockquote><blockquote>This is told through the lens of our engineering team tackling a series of challenges one by one. Medium has a small team but we operate on a big scale. We’re working our way through some technical debt and at the same time, striving to provide the best experience for our readers. This is the source of many interesting challenges.</blockquote><blockquote>I hope this series helps you understand how the recommendations algorithm work and can help others who are facing similar technical challenges.</blockquote><p>This is probably the most technical story in the series, but I will keep it as simple as possible and hopefully this is interesting for non-technical readers too.</p><h3>Some Concepts</h3><p>Here’s a little cheat sheet with some concepts you may need to follow along with this story</p><figure><img alt="Hand-drawn cheat sheet explaining Medium’s platform and recommendation system. Shows how Medium curates content through a 3-stage process: Source (pulls stories from various sources), Filter (removes duplicates/already read), and Rank (scores stories to predict user interest). Includes 4 recommendation surfaces: Daily Digest email, Homepage feed, push notifications, and post-reading suggestions called “Recire.”" src="https://cdn-images-1.medium.com/max/1024/1*9V1FpdLmgfQgEc8YDA9xFQ.png" /><figcaption>You may need this to understand the rest of this post</figcaption></figure><h3>Bloom Filters at Medium</h3><p>A lot of the filters I mention in this series are backed by Bloom Filters (I’ve described some of those filtering rules in <a href="https://medium.engineering/engineering-stories-behind-the-medium-daily-digest-algorithm-part-1-909a7ca5e807">Part 1</a> if you haven’t read it already). We use Bloom filters to remove stories we think won’t interest readers from their feeds and other recommendations:</p><p><strong>For example:</strong></p><ul><li>our “muted” filter removes all stories from writers that you have muted</li><li>our “read” filter removes all stories that you have already read</li><li>our “presentation” filter removes all stories that have already been presented to you 3 times or more</li></ul><p>These all rely on Bloom Filters</p><h3>So what’s a Bloom Filter?</h3><p>I asked Claude to summarize Bloom filters in a really simple way and it went with a funny analogy I’m going to try here.</p><p>A Bloom filter is like a super-efficient bouncer at a club who has a really good memory but isn’t perfect. It lets you do two things:</p><ol><li>let someone into the club<br> → in code that would be an add(string) function</li><li>lets you ask if someone is in the club. There are two possible answers to this:<br> → “yes, probably”<br> → “no, definitely not”<br> → in code that would be a check(string) --&gt; bool function</li></ol><p>That doesn’t sound super useful like that but we’ll see next that it’s actually kinda well suited for recommendation systems.</p><p>At Medium we’re using it to store information such as “user a read story x” or “user a muted user b”. We add those to the “club” as strings, like read|user_x|story_y . Later on, when we want to know if user x has already read story y, we just ask our “bouncer”: is read|user_x|story_y in the club?</p><figure><img alt="Flow diagram showing Medium’s Bloom Filter system. Two user actions feed into the green Bloom Filter: “User reads a story” adds ‘read|user_a|post_x’ and “User mutes a writer” adds ‘muted|user_a|user_b’. The filter checks if actions are “in da club” and connects to recommendation algorithm for filtering user content." src="https://cdn-images-1.medium.com/max/1024/1*o_zsHWREij_C_OeehtOGNg.png" /><figcaption>How we use Bloom Filters to filter out muted writers and already read stories from user feeds</figcaption></figure><h3>The scale of the filtering</h3><p>What’s nice with Bloom filters is that they are able to store the information very efficiently and they are able to handle big amounts of requests per second. We don’t really need to know more about the inner workings of Bloom filters for this series but you can read more about it <a href="https://systemdesign.one/bloom-filters-explained/">here</a> (and it has some Excalidraw schemas too 🤌).</p><p>When we’re building a feed for a user (the digest for example), we’re sometimes sourcing up to 5000 stories as the initial “shortlist” of stories. This shortlist goes through many different steps and when we’re done, there’s only a handful of stories, ready to be sent in your digest 🙌</p><p>Ideally we should filter out stories as early as possible in the process. If we take the filters I’ve listed above, we’re asking ourselves 3 questions for every story in the short list:</p><ul><li>did the user read this story already?</li><li>is the writer of the story muted by the user?</li><li>was the user presented this story more than 3 times already?</li></ul><p>So that can add up to 15k questions in total for a single user feed, and we process thousands of feeds per second. So that’s more than 15M questions per second. You can see how that can get out of hand and become expensive very quickly, we’re going to need some solid infrastructure to handle this.</p><p>Fortunately our Bloom filter “bouncer” is able to answer an insane amount of questions per second for pretty cheap, which is exactly what we’re looking for.</p><h3>Bloom Filter downsides</h3><p>The biggest downside for us is that even though the Bloom filters are super memory efficient, we overused them so much by adding literally billions of items that they started getting really really big and expensive. Like I mentioned before, there are only two operations you can do on a Bloom filter: add or check. There’s no way to delete any data from the filter (remove people from the club, if we stick with the bouncer analogy). And so the club can only grow in size, there’s no way to perform routines cleanups for information that we don’t need anymore.</p><p>The only way to reduce the size of the club is to start a new club with a new bouncer and then retire the old club and bouncer.</p><p><strong>There are also a few more downsides:</strong></p><ul><li>you can’t list the items that are stored (you can’t get a list of people who are in the club…). So we can’t go and see what’s stored for a given user, this makes it really hard to debug issues with the filters</li><li>you can’t remove anyone from the club, which means that you can’t change your mind. For example if a user “unmutes” a writer, there’s no way to reflect that in the Bloom filter. That writer will be muted forever for that user. Yes, that’s very janky.</li><li>this is derived data. For example the user “mutes” are stored in a proper database, and anytime there is a mute action we need to forward that information to the bloom filter. This is inconvenient and introduces data drift issue and more complexity overall.</li></ul><figure><img alt="Diagram showing mute/unmute workflow with databases and Bloom Filter. Top: “User mutes a writer” updates database with timestamp and adds entry to Bloom Filter (shown as crosshatched “Black Box”). Bottom: “User unmutes a writer” updates database but shows “not possible” arrow to Bloom Filter, illustrating that Bloom Filters can’t remove entries once added." src="https://cdn-images-1.medium.com/max/1024/1*GDwjQOf0c7dHaVoFzo1y8Q.png" /><figcaption>Bloom filters are not databases. You can’t see what’s inside and you can’t remove items</figcaption></figure><p>All in all, there were too many downsides so we decided to explore a different approach.</p><h3>Replacing the “Muted Filter”</h3><p>When we build the feed for a given user, it happens in real time. It can be when you load the Medium homepage, or when we trigger the daily digest generation. We’re typically in the following situation:</p><ul><li>we have a shortlist of 1000 stories</li><li>we want to filter out the stories from writers that are muted by the user</li></ul><p><strong>There are two ways to go about this:</strong></p><p><strong>A. perform a lookup in the database table</strong> that keeps track of muted writers for the 1000 (userID, writerID) pairs and check the mutedAt attribute</p><figure><img alt="Database lookup diagram titled “Pairwise Lookups on Database” showing inefficiency of checking 1000 user-writer pairs individually. Database table shows userID, writerID, and mutedAt columns with sample data. Arrow shows recommendation algorithm requesting all pairs (user_a, writer_1) through (user_a, writer_1000) separately, demonstrating performance bottleneck." src="https://cdn-images-1.medium.com/max/1024/1*O21BbwJT2M1LZFePFSxHYA.png" /><figcaption>Approach A. For each pair we do a database lookup</figcaption></figure><p>This is extremely expensive which is why we have to introduce a Bloom filter, where lookups are faster and cheaper</p><figure><img alt="diagram titled “Pairwise Lookups with Bloom Filter”. Recommendation algorithm queries Bloom Filter with 1000 muted user-writer pairs like ‘muted|user_a|writer_1’ through ‘muted|user_a|writer_1000’ asking “Are those ‘in da club’?”" src="https://cdn-images-1.medium.com/max/1024/1*DfgSJkxOj5GoiuK4DHbHHw.png" /><figcaption>Approach A. but with a Bloom filter</figcaption></figure><p><strong>B. or fetch ALL of the muted writers for the current user</strong> and then cross-reference that with your shortlist of stories</p><figure><img alt="Database query diagram titled “With Direct Database query” showing average of ~1 database items read. Database table with userID, writerID, mutedAt columns. Arrow shows recommendation algorithm requesting “Get all writers muted by user_a” and database responding with “writer_1, writer_5”. Single efficient query instead of multiple lookups." src="https://cdn-images-1.medium.com/max/1024/1*ZlHYC_dAifZy8aj94CENRA.png" /><figcaption>Approach B. fetch all muted writers and then cross-reference with the shortlist</figcaption></figure><p>Approach B has a massive advantage: <strong>most users do not mute anyone</strong>. So on average, we’re reading a very small amount of data from the database. If we were to do that with dynamoDB, B is a Query that retrieves on average less than 1 item from the DB for each feed. <strong>This is very cheap and fast</strong>. For “power muters” — users who mute massive amounts of writers, in the thousands — we can still handle this in real time although with higher latencies.</p><p>So this approach immediately obliterates the need for a Bloom filter. We have a fast, cheap, reliable way to filter out muted writers. We can also use our “ground truth” database directly, no need for derived data with all the headaches this involves.</p><p>So that’s one less Bloom filter! Let’s move on to the next one.</p><h3>Replacing the “Presentation Filter”</h3><p>The question becomes a bit trickier when we look at the “presentation” filter. We’re typically in a situation where:</p><ul><li>we have a shortlist of 1000 stories</li><li>we need to remove all the stories that were presented to the user in a feed (in the past) 3 times or more</li></ul><figure><img alt="Diagram showing Medium feed presentation tracking. Left side shows user feed with three articles (resumes, travel, reading topics) labeled as post_a, post_b, post_d. Arrows show each presentation increments a counter (+1). Right side shows “Presentation Counter” tracking counts for each user-post pair, connected to recommendation algorithm applying “Presentation Filter” to exclude posts shown 3+ times." src="https://cdn-images-1.medium.com/max/1024/1*Ap1enCqRVv1ofHoXziOq6Q.png" /><figcaption>We keep track of how many times posts were presented to a user in a feed. When we build a new feed we make sure to exclude stories that were already presented several times</figcaption></figure><p>We’re currently doing that filtering using a Bloom filter. But you might wonder how we can even do that with Bloom filters. Remember there’s only two operations you can do with a bloom filter:</p><ul><li>add someone to the club</li><li>ask if someone is in the club</li></ul><p>Bloom filters were not built to maintain counters, only true / false information. So we have to hack our way around it by layering them. In terms of clubs and bouncers, it’s like we have a big festival with a bouncer. Inside the festival there’s a private club with another bouncer. And inside that private club there’s a VIP zone with another bouncer… Ultimately we only want to know if someone is inside the VIP zone, so we only need to ask the VIP bouncer. But we still need the two other bouncers to keep track of who’s eligible to get in the VIP zone…</p><p>Going back to Bloom filters: we use 3 layers of Bloom filters, on top of each other, with each one encoding the information “post was presented to user x times”</p><figure><img alt="Flowchart titled “Implementing a counter with Bloom Filters” showing how multiple Bloom Filters track presentation frequency. When post is presented to user, system checks three Bloom Filters: A (≥1 presentation), B (≥2 presentations), C (≥3 presentations). Flow shows: check Filter A, if yes check Filter B, if no add to Filter C. Each filter represents different presentation count thresholds." src="https://cdn-images-1.medium.com/max/1024/1*OAtMCmXPd0Me5A6Bd-gyRQ.png" /><figcaption>How we maintain counters with bloom filters. An event listener updates the bloom filters to maintain the counter when a post presentation happens. That’s just to count to 3. Imagine if you need to count to 10…</figcaption></figure><p>From the recommendations algorithm’s perspective it’s fairly simple, we just lookup the (user, post) pairs in the Bloom filter that encodes the “3” value of the counter (we just ask the VIP bouncer)</p><figure><img alt="Diagram titled “Presentation Filter with Bloom Filter” showing recommendation algorithm querying Bloom Filter C with user-post pairs (user_a, post_1) through (user_a, post_1000) asking “Are those ‘in da club’?” Bloom Filter C contains posts presented at least three times to the user, enabling efficient filtering during recommendation process." src="https://cdn-images-1.medium.com/max/1024/1*4UDl9CUje2EgrgGaM4pruw.png" /></figure><blockquote>NB: that’s the implementation in place at Medium. I don’t know what went into consideration when building it this way. But FIY there are other (probably better) ways to build counters with Bloom Filters. <a href="https://systemdesign.one/bloom-filters-explained/#counting-bloom-filter">It’s possible to use bit-arrays for example</a>.</blockquote><p><strong>So that’s for the Bloom filter implementation. Now how can we handle that differently?</strong></p><p>The user-based approach involves fetching all of the “presentation history” of a user (ie all of the posts that were presented to the user in a feed before) and then going over that list to compute presentation counts for each story.</p><figure><img alt="Direct Database Query diagram showing flow from database table (userID, postID, presentedAt columns) through query “get all presentations for user_a” to list of posts (post_1, post_4, post_13, post_3), then counting to create totals (post_1: 1, post_4: 2, post_3: 6), finally applying threshold to filter out posts that should be filtered." src="https://cdn-images-1.medium.com/max/1024/1*kiKSxLxxpjt9LOcrwKVBgA.png" /></figure><p>That’s a little bit more complex than the “muted filter”, because:</p><ul><li>a lot of users have really massive presentation histories, in the tens of thousands or even bigger. Those are readers who come to Medium every day and are exposed to many recommendations</li><li>on average, users have a lot of posts in their presentation histories. Typically less than 100 though. That’s because this includes many less engaged users who only came a handful of times on Medium, and they bring the average down.</li></ul><p>So the average cost is going to be higher than for the muted filter because we’re going to retrieve more data from the database <strong>AND</strong> some users have such big presentation histories that it’s not possible to fetch it in real time. So how can we tackle that?</p><p>Here, we’re saved by the fact that this filtering rule is not a hard requirement. Nothing says that “3 presentations is the absolute max” for a given (user, post). We built this rule to make feeds more diverse and less repetitive. We chose 3 as a reasonable default, but it’s okay if in some cases we reach higher counts.</p><p>So we can go with a “best effort” strategy here. We fetch the most recent 5k presentations for the user and simply act as if everything before that never existed. DynamoDB queries let us fetch thousands of items quickly, in the ~100ms. So that simple solution is very acceptable in terms of functionality and latencies. There are scenarios where the filter will not be doing its job properly, but they’re limited to certain edge cases with minimal impact on the user.</p><p>We tested this approach and found that the costs were reasonable. This solution gives us more flexibility, for instance we can now easily control and play with the maximum number of presentations (is 3 the right threshold? we’ll see that in <a href="https://medium.engineering/engineering-stories-behind-the-medium-daily-digest-algorithm-part-3-6fbf1512e6e6">Part 3</a>). Down the line that will lead to a better experience for users. Now that this filter has been replaced, let’s move on to the final filter, which was the most challenging to migrate.</p><h3>Replacing the “Read Filter”</h3><p>Here’s what we’re trying to do:</p><ul><li>we have a shortlist of 1000 stories</li><li>we need to remove all the stories that the user has already read</li></ul><p>Very similar to the “Presentation Filter”, the user-based approach for this filter involves fetching the entire reading history of the user. We’re in the same situation where most of the time the user’s reading history is relatively small, with less than 100 posts on average. But we do have some users with massive reading histories.</p><blockquote>Fun fact, one of or top readers is our very own Harris Sockel who was in charge of the Medium Newsletter, and had 22k stories in his reading history just for 2024!</blockquote><p>This time, filtering already read posts is a hard requirement of the recommendations algorithm, no way around it. We don’t want to send you an email or a push notification about a post you’ve already read.</p><p><strong>So what do we do?</strong> For this filter specifically we decided to do the filtering in two stages. Very early in the feed building process, we fetch the most recent posts in the user’s reading history (the last 5k posts you’ve read). For the vast majority of users, this will capture their entire reading history. And we use that to filter out already read posts from the feed (at this point, we can have up to 5k posts in the feed). But that’s not enough, we need to guarantee that there are no read posts in the final results for all users.</p><p>So to explain this further, we’ll need to go into our recommendation algorithm in a little bit more detail:</p><figure><img alt="Medium feed creation flowchart titled “Creating a Feed” showing process from multiple sources (followed writers, followed pubs, deep retrieval model) flowing through Source, Early Filter, Aggregate, Rank, and Final Filter stages. Side annotations show “Up to 5k posts” after Source, “Up to 1k posts” after Aggregate, and “About 10 posts” after Final Filter, demonstrating progressive narrowing." src="https://cdn-images-1.medium.com/max/1024/1*DvOB1kih8g6xfTl-7tQ1ZA.png" /><figcaption>A more detailed view of the recommendations algorithm</figcaption></figure><p>You can see that there are two different filtering steps. What we did is that we split out our “Read Filter” into two different implementations and we added each one at a different step:</p><figure><img alt="Two Stage Filtering diagram showing Early Filter and Final Filter stages. Early Filter shows post shortlist being filtered by “Fetch Already Read Posts” to create filtered shortlist, reducing from “Up to 5k posts.” Final Filter shows similar process with “Fetch User-Post Read State” filtering posts, ending with “About 10 posts.” Demonstrates efficient filtering at different stages." src="https://cdn-images-1.medium.com/max/1024/1*Vi7jAHinbo-yqrRq6uprMw.png" /></figure><p>We’re doing user-based queries in the early filtering step. This is nice because costs do not depend on the number of posts in the shortlist. Once we’re down to only a few posts, we can perform pairwise (user, post) lookups.</p><p>This solution works really well and doesn’t add too much complexity to the recs logic. This did require some gymnastics to maintain the requirements while keeping costs, recs performance and latencies under control. But this is much easier to control and debug than the previous implementation with Bloom Filters. It also allows to support things like clearing the reading history for a given user (you can do that from <a href="https://medium.com/me/lists/reading-history">this page</a> to get a “recs fresh start” on Medium). This is a functionality that didn’t work well with the Bloom Filter implementation.</p><h3>Conclusion</h3><p>All in all we were able to get rid of our Bloom filters entirely, resulting in big cost savings. <strong>The new implementation is ten times cheaper than what our Bloom filters were costing us.</strong></p><p>This isn’t necessarily a fair comparison since our Bloom filters had grown out of proportion. If we had migrated from scratch to a new Bloom instances, the Bloom filter implementation might have been cheaper. It was hard to evaluate it in our situation because of the way we built our Bloom filters. We used a single instance and string prefixes to manage all of the different filters.</p><p>If we go back to our club and bouncer analogy, you can see it that way: let’s say there’s different crowds that go to the club, techno lovers, jazz enthusiasts, disco heads…</p><p>A. you can build a single club that hosts everyone. A single bouncer has the responsibility to memorize all the people in the club</p><p>B. or you can build one club for each crowd, each one with a different bouncer</p><p>If one day you don’t care about jazz enthusiasts anymore, with B. you can just retire the jazz club and it’s bouncer. But with A. you still need the bouncer for the other crowds, so you can’t retire the bouncer and it’s still holding the information about all the jazz enthusiasts that entered the club. Down the line you might be paying for a club that’s bigger than you need with approach A. <strong>With approach B. you pay exactly for what you need!</strong></p><p><strong>With Bloom filters it looks like this:</strong></p><figure><img alt="Comparison of Single vs Multiple Bloom Instance architectures. Top shows one Bloom Filter receiving both read and muted data, queried by recommendation algorithm with a `read` or `muted` prefix. Bottom shows separate “Read” and “Muted” Bloom Filters. No need for string prefixes" src="https://cdn-images-1.medium.com/max/1024/1*AdT_565BROXqhN0LTVwhPA.png" /><figcaption>Decoupling filters into separate instances is probably much easier to maintain. When you stop using a filter you can just delete the instance. Our pattern of a single monolith Bloom instance forced us to keep paying until we had migrated all of the filters to newer instances (or to another implementation).</figcaption></figure><p>We were working with a single gigantic Bloom Filter, this meant that we weren’t able to retire anything and our Bloom Instance was ever-growing.</p><p>The bottom line is that this project helped us get our costs down and migrate to newer, more flexible implementations that are much easier to maintain. With our newer implementations:</p><ul><li>costs are easy to understand and are under control</li><li>filtering costs don’t depend on the initial size of the feed “shortlist”</li><li>there’s no need to maintain derived data</li></ul><p>This also laid the groundwork for experiments on our filtering rules that we’ll see in our <a href="https://medium.engineering/engineering-stories-behind-the-medium-daily-digest-algorithm-part-3-6fbf1512e6e6"><strong>Part 3: Hard vs Soft Filtering and how this applies to Medium’s Recommendation System</strong></a><strong>.</strong></p><p>Thank you for reading this serires, you can stay tuned for the next installments of this series by following the <a href="https://medium.engineering/">Medium Eng Blog</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c977ad0b134f" width="1" height="1" alt=""><hr><p><a href="https://medium.engineering/engineering-stories-behind-the-medium-daily-digest-algorithm-part-2-c977ad0b134f">Engineering stories behind the Medium Daily Digest Algorithm: Part 2</a> was originally published in <a href="https://medium.engineering">Medium Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Engineering stories behind the Medium Daily Digest Algorithm: Part 1]]></title>
            <link>https://medium.engineering/engineering-stories-behind-the-medium-daily-digest-algorithm-part-1-909a7ca5e807?source=rss-7e93bd9fc8de------2</link>
            <guid isPermaLink="false">https://medium.com/p/909a7ca5e807</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[recommendation-system]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Raphael Montaud]]></dc:creator>
            <pubDate>Tue, 26 Aug 2025 11:31:36 GMT</pubDate>
            <atom:updated>2025-09-04T17:39:21.452Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OJipelgT4x24BzI4gtjFIQ.jpeg" /></figure><h4>How we made our email story recommendations better</h4><p>In this Part 1, you’ll understand how we improved one of the main ways our users are exposed to our product and how that led to <strong>a massive 7% increase on the average reading time</strong> for the digest users.</p><blockquote><strong>Intro</strong>: This is a 4-part series breaking down improvements to the algorithm behind the Medium’s Daily Digest over the past year. When we started this work, the Digest was suboptimal — and since it’s a huge distribution surface, reaching millions of readers every day, we started working on incremental improvements.</blockquote><blockquote>By the end of these projects, the digest was 10% more likely to convert users to paying members, less expensive to run, more flexible and easier to maintain and it’s now providing higher quality recommendations for all our users, including our “power readers”.</blockquote><blockquote>This is told through the lens of our engineering team tackling a series of challenges one by one. Medium has a small team but we operate on a big scale. We’re working our way through some technical debt and at the same time, striving to provide the best experience for our readers. This is the source of many interesting challenges.</blockquote><blockquote>I hope this series helps you understand how the recommendations algorithm work and can help others who are facing similar technical challenges.</blockquote><h3>Some Concepts</h3><p>Here’s a little cheat sheet with some concepts you may need to follow along with this story</p><figure><img alt="Hand-drawn cheat sheet explaining Medium’s platform and recommendation system. Shows how Medium builds content recommendations through a 3-stage process: Source (pulls stories from various sources), Filter (removes duplicates/already read), and Rank (scores stories to predict user interest). Includes 4 recommendation surfaces: Daily Digest email, Homepage feed, push notifications, and post-reading suggestions called “Recire.”" src="https://cdn-images-1.medium.com/max/1024/1*9V1FpdLmgfQgEc8YDA9xFQ.png" /><figcaption>You may need this to understand the rest of this post</figcaption></figure><h3>The Discovery</h3><p>A little while back, Leigh, our Machine Learning Engineer and model training guru, started noticing something weird. The recommended stories in his digest were consistently not great matches for his reading profile. At the same time, the recommended stories in his homepage feed (what we call the “For You” feed) were consistently very well targeted.</p><figure><img alt="Side-by-side comparison of Leigh’s Medium feeds. Left shows “For You” feed with AI/recommendation systems stories (generative AI color schemes, Pinterest’s text-to-SQL, candidate ranking models, data team roles). Right shows Daily Digest email with broader data science topics (AI talent, workplace advice, Simpson’s Paradox, Nobel Prize decisions). Demonstrates personalized vs. curated content differences." src="https://cdn-images-1.medium.com/max/1024/1*tXfCC3EG_e14-cx-y3UzCg.png" /></figure><p>This was a bit puzzling and unexpected. Those two recommendation surfaces rely on exactly the same algorithm. We source stories the same way, and we rank them using the same model and features. The only big difference lies in the filtering step.</p><h3>The Investigation</h3><p>So at this point, we thought maybe this is a bug or maybe there’s something in the filtering step that’s amok specifically for the digest. We started digging into the different filters we use for the digest vs the ones we use for the homepage feed. We have many different filtering rules. For example the “Filter Read” simply filters out all the stories that the user has already read. This one is applied on most recommendation surfaces. But some other filters are specific to certain recommendation surfaces:</p><figure><img alt="Comparison of Medium’s filtering systems. Left side shows “Homepage Specific Filters” with one “Presentation Filter” that removes posts shown three times prior. Right side shows “Digest Specific Filters” with three filters: Digest Title Filter (excludes posts used in previous email subjects), Sent in Opened Digest Filter (excludes posts from opened emails), and Digest Backoff Filter (excludes posts from last 7 days’ digests)." src="https://cdn-images-1.medium.com/max/1024/1*3GOqIEAFNJd_ZBAVKaX2HA.png" /><figcaption>Filtering rules differ between the Homepage Feed and the Digest</figcaption></figure><p>One of the reasons why those two surfaces use different filtering is because the reporting does not work the same:</p><ul><li>on the homepage we can track exactly what posts were presented to the user</li><li>with the digest email we have access to less information:<br> → we know if the email was opened (thanks to a tracking pixel, explained below)<br> → we know what posts we sent in a given digest<br> → we know which posts were clicked in a given digest</li></ul><p>Looking at this, we started realizing that the filtering applied to the digest is probably too aggressive. Every day, the algorithm sources the best 15 stories on the entire platform for the reader. But as soon as we detect an email was opened then we will never send the posts it contains again to that reader. That can be a bummer in some situations, for example:</p><ul><li>the user opens the email but doesn’t scroll past the top 5 stories</li><li>all the stories in that digest will never be sent again, although 10 of them didn’t even get a chance to be presented to the reader</li></ul><figure><img alt="Diagram showing Medium’s digest story selection over time. Day 1: top-ranked stories (1–15) go to Day 1 Digest, with lower-ranked stories available. Day 4: top three story groups are crossed out (filtered), so Day 4 Digest uses stories ranked 45–60. Shows how filtering prevents repetition and pushes digests to use progressively lower-ranked content over consecutive days." src="https://cdn-images-1.medium.com/max/1024/1*cuK8iqlKupWijKcow5EtPQ.png" /><figcaption>Illustration to show how we quickly burn through the best stories for a given user if we detect that they open their daily digest emails. After a few weeks the recommendations are less and less relevant to the user as there are fewer and fewer eligible stories for their digest. This is a simplification, the recommendations are not stale and there’s new supply coming in every day, but not necessarily enough to compensate what we’re burning through</figcaption></figure><p>Having worked on email open tracking before, we also knew that there were other elements at play here and that are potentially making things even worse.</p><h3>Apple Mail Privacy Protection</h3><p>In 2021, Apple started Mail Privacy Protection. The idea was to better protect user data and to prevent email open tracking.</p><p>When we send the digest, we also send a pixel tracker in the email. That’s a tiny image URL that we send as part of the mail content and when the email is loaded, the email client has to fetch this image. The URL is engineered in a way that when it’s loaded, we know exactly which email loaded it and so we can record the email as “opened”.</p><figure><img alt="Email tracking diagram showing flow from sender to user. When user opens tracking email, a tiny 1x1 pixel tracking image loads from the tracking server, sending “email was opened” data back to sender. Illustrates how invisible tracking pixels monitor email engagement" src="https://cdn-images-1.medium.com/max/1024/1*orhtPFsZV-Cze_0VOUO8Vg.png" /><figcaption>Tracking pixels are used by email senders to know if a user has opened an email</figcaption></figure><p>Apple’s Mail Privacy Protection has completely destroyed this concept. They are now preloading all your emails directly from their servers, and your email client communicates only with the Apple servers. From our point of view, it means that pretty much all the emails from Apple users are marked as “opened”.</p><figure><img alt="Postmark diagram showing Apple Mail Privacy Protection. Email flows from sender to Apple’s proxy server, then to recipient. When recipient opens email, click and open data goes to Apple’s proxy instead of back to sender. Bottom text reads “Recipient activity is invisible to the sender,” illustrating how Apple blocks email tracking." src="https://cdn-images-1.medium.com/max/1024/1*Oist485PgypueAPxatOQFQ.png" /></figure><p>So a lot of our “email open events” are actually “remote servers opening your email for you so that no one can tell if you actually opened it or not” events. We’re still not sure exactly about the exact figures but several estimates suggest that 50 to 80% of our “email open events” are fake.</p><p>That means a lot of our email open events are “fake news”. And those 15 stories that we carefully selected for you will never be eligible for your digest again — even if a lot of user didn’t actually open the email.</p><p>So it does look like we’re filtering way too aggressively. At this point we were confident that this filtering rule was the culprit and that it’s responsible for degrading the quality of the recommendations in the digest. Now we just need to come up with a plan to fix it.</p><h3>The Plan</h3><p>We went over the filtering rules in place and decided we’d modify a few things:</p><figure><img alt="Before/after comparison of Medium’s digest filtering. Left shows three filters: Digest Title Filter (unchanged), Sent in Opened Digest Filter (red, removes posts from opened digests), and Digest Backoff Filter (filters posts from last 7 days). Right shows updated filters with red filter removed and backoff reduced from 7 to 4 days. Labels show “removed” and “modified.”" src="https://cdn-images-1.medium.com/max/1024/1*YKPx5kU-FQq14-Ee68KtPg.png" /><figcaption>We removed the “Sent in Opened Digest Filter” and scaled down the backoff logic from 7 to 4 days</figcaption></figure><p>This is now a much less aggressive setup. It means that there will be more repetition in digests, the hope is that the best stories are able to make their way to the user and that they don’t get “disqualified forever” too early. Several things are in place to ensure that the digests are not too repetitive and evolve over time:</p><figure><img alt="Cross-digest diversification diagram with two sections. Left shows “Filtering Rules”: Digest Title Filter (excludes posts used in previous email subjects) and Digest Backoff Filter (excludes posts from last 4 days’ digests). Right shows “Organic Evolution of recs”: Newly Published Stories and User Profile Evolution explaining how changing content and evolving user preferences naturally diversify recommendations." src="https://cdn-images-1.medium.com/max/1024/1*qYR-qNlFJ0nJ-3g_YT9yKA.png" /><figcaption>We expect those filtering rules and organic evolution of recs to bring diversification between the daily digests</figcaption></figure><figure><img alt="Diagram showing Medium digest story eligibility rules. “Digest Stories” box contains two items: “rank 1 story” with arrow pointing to “Never eligible again for the digest” and “rank 2 to 15 stories” with arrow pointing to “Not eligible for the next 4 days.” Shows how top story gets permanent exclusion while others get temporary 4-day exclusion." src="https://cdn-images-1.medium.com/max/1024/1*CrHn-hhmwjeTbiTSCAxs8g.png" /><figcaption>A digest contains 15 stories. The first one is used as the email subject, which is why it’s not eligible for the digest anymore</figcaption></figure><p>We were still a little bit wary about potential repetitiveness of digests. We wondered if users were going to notice and complain about that, so we gave a heads-up to the support team to be on the lookout for such complaints</p><p>We put that to the trial in an A/B test and got incredible results very quickly.</p><p><strong>Users in the experiment:</strong></p><ul><li>were <strong>10% more likely to convert</strong> to paying members</li><li><strong>read 7% more</strong> on Medium</li></ul><p><strong>A massive win for Medium!</strong></p><p>Readers didn’t seem to notice any repetitiveness in the digests, so we decided to ship without thinking too much about it.</p><p>We were so happy about the results that we immediately went on the hunt for more filtering rules that we could tweak.</p><p>We first had to dig into the implementation of our filters (CF <a href="https://medium.com/medium-eng/engineering-stories-behind-the-medium-daily-digest-algorithm-part-2-c977ad0b134f">Part 2: How we made our filtering 10x cheaper by removing our Bloom Filters</a>), before we could start making more changes (CF <a href="https://medium.engineering/engineering-stories-behind-the-medium-daily-digest-algorithm-part-3-6fbf1512e6e6">Part 3: “Hard vs Soft Filtering” and how this applies to Medium’s Recommendation System</a> and <a href="https://medium.engineering/engineering-stories-behind-the-medium-daily-digest-algorithm-part-4-ec7136f21acd">Part 4: Cross-Digest diversification</a>)</p><h3>Some Final Thoughts</h3><p>So it looks like the Apple Mail Privacy Program has been affecting our recommendation systems since 2021. Not all Apple Mail users were in the program initially and the number of users in the program has been scaling up over the years as well as the blast radius on Digest users. My recommendation to you, if you have any features that rely on email open tracking in your product, is to <strong>immediately audit the potential effects of those “fake email opens”</strong>.</p><blockquote>To be clear we are absolutely in favor of user privacy measures. The Apple privacy program has forced us to re-evaluate the way we build our recommendations in a less intrusive way, which is a good thing.</blockquote><p>We recently made the <a href="https://medium.com/blog/the-email-digest-is-how-millions-of-medium-readers-find-stories-now-weve-brought-it-to-the-app-c38f08463014">daily digest available in the Medium App</a>, this was a much requested feature by our users. On top of that this is an opportunity to get better tracking than what we do in the email clients. Down the line this could help us improve the recommendations we put in those digests.</p><p>Thank for reading this series, you can stay tuned for the next installments of this series by following the <a href="https://medium.engineering/">Medium Eng Blog</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=909a7ca5e807" width="1" height="1" alt=""><hr><p><a href="https://medium.engineering/engineering-stories-behind-the-medium-daily-digest-algorithm-part-1-909a7ca5e807">Engineering stories behind the Medium Daily Digest Algorithm: Part 1</a> was originally published in <a href="https://medium.engineering">Medium Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Learnings from optimising 22 of our most expensive Snowflake pipelines]]></title>
            <link>https://medium.engineering/learnings-from-optimising-22-of-our-most-expensive-snowflake-pipelines-5ea6fcf57356?source=rss-7e93bd9fc8de------2</link>
            <guid isPermaLink="false">https://medium.com/p/5ea6fcf57356</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[cloud-hosting-costs]]></category>
            <category><![CDATA[optimisation]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Raphael Montaud]]></dc:creator>
            <pubDate>Mon, 30 Sep 2024 10:06:01 GMT</pubDate>
            <atom:updated>2024-09-30T10:06:01.017Z</atom:updated>
            <content:encoded><![CDATA[<p>We recently spent a sprint focused on reducing our Snowflake costs. During this sprint, we investigated 22 of our most expensive pipelines (in terms of Snowflake costs), one by one. In total we merged 56 changes, and in this post we’ll be laying out the optimisations that worked best for us.</p><p>Most of these changes were just common sense and didn’t involve any advanced data engineering techniques. Still, we’re always making mistakes (and that’s okay!) and hopefully this post will help readers avoid a few of the pitfalls we encountered.</p><blockquote>⚠️ Medium is now 14 years old. Our team has inherited a tech stack that has a long history, some flaws and technical debt. Our approach to this problem was pragmatic; we’re not trying to suggest big pipeline re-designs to reach a perfect state, but rather consider our tech stack in it’s current state and figure out the best options to cut costs quickly. Our product evolves, and as we create new things we can remove some old ones, which is why we don’t need to spend too much time re-factoring old pipelines. We know we’ll rebuild those from scratch at some point with new requirements, better designs and more consideration for costs and scale.</blockquote><h3>Do we need this?</h3><p>In a legacy system, there often are some old services that just “sound deprecated.” For example, we have a pipeline called medium_opportunities , which I had never heard about in 3 years at Medium. After all, it was last modified in 2020… For each of those suspect pipelines we went through <strong>a few questions:</strong></p><ul><li><strong>Do we need this at all??</strong> Through our investigation, we did find a few pipelines that were costing us more than $1k/month and that were used by… nothing.</li><li>A lot of our Snowflake pipelines will simply run a Snowflake query and overwrite a Snowflake table with the results. For those, the question is: <strong>Do we need all of the columns?</strong> For pipelines we cannot delete, we identified the columns that were never used by downstream services and started removing them. In some cases, this removed the most expensive bottlenecks and cut the costs in a massive way.</li><li>If it turns out the expensive part of your pipeline is needed for some feature, you should question <strong>if that feature is really worth the cost or if there is a way to tradeoff some cost with downgrading the feature without impacting it too much</strong>. (Of course, there are situations where it’s just an expensive and necessary feature…)</li><li><strong>Is the pipeline schedule aligned with our needs?</strong> In our investigation we were able to save a bunch of money just by moving some pipelines from running hourly to daily.</li></ul><h4>An example:</h4><p>A common workflow among our pipelines involves computing analytics data in Snowflake and exporting it to transactional SQL databases on a schedule. One such pipeline was running on a daily schedule to support a feature of our internal admin tool. Specifically, it gave some statistics on <strong><em>every</em></strong> user’s reading interests (which we sometimes use when users complain about their recommendations).</p><p>It turns out this was quite wasteful since this feature wasn’t used daily by the small team who relies on it (maybe a couple times per week). So, we figured <strong>we could do away with the pipeline and the feature</strong>, and replace it with an on-demand dashboard in our data visualization tool. Then the data will be computed only when needed for a <strong><em>specific</em></strong> user. It might require the end user to wait a few minutes for the data, but it’s massively cheaper because we only pay when somebody triggers a query. It’s also less code to maintain and a data viz dashboard is much easier to update and adapt to our team’s needs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z9Zq7xqtOCw4az0wrXup_w.png" /><figcaption>Old vs new architecture for this example</figcaption></figure><h4>To conclude this section, here are a few takeaways that I think you can activate right away at your company:</h4><ul><li><strong>Make sure your analytics tool has a way to sync with Github</strong>. Our data scientist <a href="https://medium.com/u/d5c3ddbb2482">gustavo</a> set that up for us with Mode and it has been massively helpful to quickly identify if tables are used in our data visualisations.</li><li><strong>Make sure you document each pipeline</strong>. Just one or two lines can save hours for the engineers who will be looking at this in 4 years like it’s an ancient artifact. I can’t tell you the amount of old code we find every week with zero docs and no description or comments in the initial PR 🤦</li><li><strong>Deprecate things as soon as you can</strong>. If you migrate something, the follow-up PRs to remove the old code and pipelines should be part of the project planning from the start!</li><li><strong>Avoid </strong>select *<strong> statements as much as possible</strong>. Those make it hard to track which columns are still in-use and which ones can be removed without downstream effects.</li></ul><h3>Filtering is key</h3><p>By using <a href="https://docs.snowflake.com/en/user-guide/ui-query-profile">Snowflake Query Profile</a> we were able to drill down on each pipeline and find the expensive table scans in our queries. (We’ll publish another blog post about the tools we used for this project later on). Snowflake is extremely efficient at pruning queries and that’s something we had to leverage to keep our costs down. We’ve found many examples where the data was eventually filtered out from the query, but Snowflake was still scanning the entire table. So if we have one key piece of advice here, it’s that the <strong>filtering should be very explicit</strong> in order to make it easier for Snowflake to apply the pruning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/575/1*B4oEDmc1-HErmTicYipuMw.png" /><figcaption>Snowflake’s query profile tool</figcaption></figure><h4>Sometimes Snowflake needs a tip</h4><p><strong>Here’s an example:</strong> Let’s say that we want to get the top 30 posts published in the last 30 days that got the most views in the first 7 days after being published. Here’s a simple query that would do this:</p><pre>select post_id, count(*) as n_views<br>from events<br>       join posts using (post_id)<br>-- only look at view events<br>where event_name = &#39;post.clientViewed&#39;<br>  -- only look at views on the first seven days after publication<br>  and events.created_at between to_timestamp(posts.first_published_at, 3) and to_timestamp(posts.first_published_at, 3) + interval &#39;7 days&#39;<br>  -- only look at posts published in the last 30 days<br>  and to_timestamp(posts.first_published_at, 3) &gt; current_timestamp - interval &#39;30 days&#39;<br>group by post_id<br>order by n_views desc<br>limit 30</pre><p>If we look at the query profile we can see that 11% of the partitions from the events table were scanned. That’s more than expected. It seems like Snowflake didn’t figure out that it can filter out all the events that are older than 30 days.</p><p><strong>Let’s see what happens if we help Snowflake a little bit:</strong></p><p>Here I’m adding a mathematically redundant condition: events.created_at &gt; current_timestamp — interval ’30 days’ . Mathematically, we don’t need this condition because created_at ≥ published_at ≥ current_timestamp -interval ’30 days’ ⇒ created_at ≥ current_timestamp — interval ’30 days’ .</p><pre>select post_id, count(*) as n_views<br>from events<br>       join posts using (post_id)<br>-- only look at view events<br>where event_name = &#39;post.clientViewed&#39;<br>  -- only look at views on the first seven days after publication<br>  and events.created_at between to_timestamp(posts.first_published_at, 3) and to_timestamp(posts.first_published_at, 3) + interval &#39;7 days&#39;<br>  -- only look at posts published in the last 30 days<br>  and to_timestamp(posts.first_published_at, 3) &gt; current_timestamp - interval &#39;30 days&#39;<br>  -- mathematically doesn&#39;t change anything<br>  and events.created_at &gt; current_timestamp - interval &#39;30 days&#39;<br>group by post_id<br>order by n_views desc<br>limit 30</pre><p>Still, this helps Snowflake a bunch and we’re now only scanning 0.5% of our massive events table and the overall query is now 5 times faster to run!</p><h4>Simplify your predicates</h4><p>Here’s another example where you can help Snowflake optimise pruning.</p><p>If you have some complex predicates in your filtering rule, Snowflake may have to scan and evaluate all of the rows although that could be avoided with pruning.</p><p>The following query scans 100% of the partitions in our posts table:</p><pre>select *<br>from posts<br>-- only posts published in the last 7 days<br>-- (That&#39;s an odd way to write it, I know.<br>-- This is to illustrate how predicates can impact performance) <br>where datediff(&#39;hours&#39;, to_timestamp(published_at, 3), current_timestamp - interval &#39;7 days&#39;) &gt; 0</pre><p>If you simplify this just a little bit, Snowflake will be able to understand that partition pruning is possible:</p><pre>select *<br>from posts<br>-- only posts published in the last 7 days<br>where to_timestamp(published_at, 3) &gt; current_timestamp - interval &#39;7 days&#39;</pre><p>This query scanned only a single partition when I tested it!</p><p>In practice Snowflake will be able to prune entire partitions as long as you are using simple predicates. If you are comparing columns to results of subqueries, then Snowflake will not be able to perform any pruning (<a href="https://docs.snowflake.com/en/user-guide/tables-clustering-micropartitions#query-pruning">cf Snowflake docs</a>, and <a href="https://community.snowflake.com/s/article/Pruning-behavior-with-nondeterministic-predicates-on-clustered-tables">this other post</a> mentioning this). In that case you should store your subquery result in a variable and then use that variable in your predicate.</p><blockquote>💡 An even better version of this is to filter raw fields against constants. That is the best way to ensure that Snowflake will be able to perform optimal pruning in my opinion. This is my take on how this is being optimised under the hood, as I couldn’t find any sources confirming this, so take this with a grain of salt.</blockquote><blockquote>- Suppose we store a field called published_at which is a unix timestamp (e.g. 1466945833883)</blockquote><blockquote>- Snowflake stores min(published_at) and max(published_at) for each micro-partition</blockquote><blockquote>- If you have a predicate on to_timestamp(published_at) (e.g. where to_timestamp(published_at) &gt; current_timestamp() - interval &#39;7 days&#39;) then Snowflake must compute to_timestamp(min(published_at)) and to_timestamp(max(published_at)) for each partition.</blockquote><blockquote>- If, however, you have a predicate comparing the raw published_at value to a constant, then it&#39;s easier for Snowflake to prune partitions. For example, by setting sevenDaysAgoUnixMilliseconds = date_part(epoch_millisecond, current_timestamp() - interval &#39;7 days&#39;) , our filter becomes where published_at &gt; $sevenDaysAgoUnixMilliseconds. This requires no computation from Snowflake on the partition metadata.</blockquote><blockquote>In a more general case, Snowflake can only eliminate partitions if it knows that the transformation f you are applying to your raw field is growing or decreasing (published_at &gt; x =&gt; f(published_at) &gt; f(x)) only if f is strictly growing). It’s not always obvious what functions are growing or not. For instance, to_timestamp and startswith are growing functions. ilike and between are non-monotonic a priori.</blockquote><h3>Work with time windows</h3><p>Let’s say we are computing some stats on writers. We’ll scan some tables to get the total number of views, claps and highlights for each writer.</p><p>With the current state of a lot of our workflows, if we want to look at all time stats, we must scan the entire table on every pipeline run (that’s something we need to work on but that’s out of scope here). If our platform’s usage increases linearly, our views, claps and highlights tables will grow exponentially, causing our costs to grow exponentially as well due to scanning more and more data every time the pipeline executes. Theoretically, these costs would eventually surpass the revenue generated by a linearly growing user base.</p><p>We must move away from exponentially growing queries because they are highly inefficient and incur a lot of waste at scale. We can do this by migrating to queries based on sliding time windows. If we look at engagement received by writers only on the last 3 months, then our costs will grow linearly with our platform’s usage, which is much more acceptable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KYqkvHy-YDydLCJy4U0t_Q.png" /></figure><p><strong>But this can have some product implications:</strong></p><ul><li><strong>In our recommendations system</strong>: when looking for top writers to recommend to a user, this new guideline could potentially miss out on writers that are now inactive but were very successful in the past since we’ll be filtering for stats only for the past few months. But it turns out this is aligned with what we prefer for our recommendations; we would rather encourage users to follow writers that are still actively writing and getting engagement on their posts.</li><li><strong>In the features we implement</strong>: we used to have a “Top posts of all time” feed for each Medium tag. We have since removed this feature for unrelated reasons. In the future, I think that we would advise against features like this and prefer a time window approach (”Top posts this month”).</li><li><strong>In the stats we compute and display to our users:</strong> with this new guideline we may have weaker guarantees on some statistics. For example: there’s a pipeline where we look at recent engagement on Newsletter emails. For each new engagement we record, we look up the newsletter in our sent_emails table. Previously, we would scan the entirety of that massive table to retrieve engagements for all newsletters. But, for costs sake, we now look back on engagements for emails sent in the past 60 days. This means that engagement received on a Newsletter more than 60 days after it was sent will not be taken into account on the Newsletter stats page. This has negligible impact on the stats (&lt;1% change) but we wanted to be transparent about that with our writers. We added a disclaimer at the top of the newsletter page.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iOwjYc1wv0PQh6wgA0Av7Q.png" /><figcaption>The top of the Newsletter Stats page</figcaption></figure><p>Thanks to this small disclaimer we were able to cut costs by 1500$/month on this pipeline.</p><h3>Factorise the expensive logic:</h3><p>Modularity is a cornerstone of all things software, but sometimes it can get a bit murky when applied to datasets. Theoretically, it’s easy. In practice, we find that duplicate code in an existing legacy data model doesn’t necessarily live side by side — and it’s not always just a matter of code refactoring; it may require building intermediate tables and dealing with constraints on pipeline schedules.</p><p>However, we were able to identify some common logic and modularize these datasets by dedicating some time to dive deep into our pipelines. Even if it doesn’t seem feasible, slowly working through similar pipelines and documenting their logic is a good place to start. We would highly recommend putting an effort into this — it can really cut down compute costs.</p><h3>Play around with the Warehouses:</h3><p>Snowflake provides many different warehouses sizes. Our pipelines can be configured to use warehouses from size XS to XL. Each size is twice as powerful as the previous one, but also twice as expensive per minute. If the query is perfectly parallelisable, it should run twice as fast and therefore cost the same.</p><p>That’s not the case for most queries though and we’ve saved thousands by playing around with warehouse sizes. In many cases, we’ve found that down-scaling reduced our costs by a good factor. Of course we need to accept that the query may take longer to run.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WbsQ2sbu2DWWT-x3RGHHxQ.png" /><figcaption>With perfect parallelisation the query is faster as your increase power so the costs are constant. With imperfect parallelisation the gains on the query time are more and more marginal as you increase power, so your costs ( = time * power) increase</figcaption></figure><h3>What’s next?</h3><p>First off, we’ll be following up with a post laying out the different tools that helped us identify, prioritise and track our Snowflake cost reduction efforts. And we’ll be detailing that so that you can set those up at your company too.</p><h4>New tools, new rules</h4><p>We’ve built some new tools during this sprint and we’ll be using them to monitor cost increases and track down the guilty pipelines.</p><p>We’ll also make sure to enforce all the good practices we’ve outlined in this post and have a link to this live somewhere in our doc for future reference.</p><h4>Wait we’re underspending now?</h4><p>So apparently we went a bit too hard on those cost reduction efforts and we’re now spending less credits than what we committed for in our Snowflake contract. Nobody is necessarily <em>complaining</em> about this “issue”…but it’s nice to know we have some wiggle room to experiment with more advanced features that Snowflake has to offer. So, we are going to do just that.</p><p>One area that could use some love is our events system. The current state involves an hourly pipeline to batch load these events into Snowflake. But, we could (and most definitely should) do better than that. <a href="https://docs.snowflake.com/en/user-guide/data-load-snowpipe-streaming-overview">Snowpipe Streaming</a> offers a great solution for low-latency loading into Snowflake tables, and the <a href="https://docs.snowflake.com/en/user-guide/data-load-snowpipe-streaming-kafka">Snowflake Connector for Kafka</a> is an elegant abstraction to leverage the Streaming API under the hood instead of writing our own custom Java application. More to come on this in a future blog post!</p><h4>The 20/80 rule</h4><p>I think this applies to this project. There’s tons of other pipelines we should investigate and we can probably get some marginal savings on each of them. But it will probably take twice as much time for half the outcome… We’ll be evaluating our priorities but I already know that there’s other areas of our backend we can focus on that will yield some bigger and quicker wins.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*ShgrTkJWTIkW-j_weG8NMQ.png" /><figcaption>A poor attempt at illustrating the 20/80 rule</figcaption></figure><h4>Modularize datasets for re-use</h4><p>Although we put some effort into this already, there is certainly a lot more to do. Currently, all of our production tables live in the PUBLIC schema no matter if it’s a source or derived table, which doesn’t make discovering data very intuitive. We are exploring using the <a href="https://medium.com/@valentin.loghin/implementing-medallion-architecture-in-snowflake-4e1539d23c09">Medallion Architecture</a> pattern to apply to our Snowflake environment for better table organization and self-service discovery of existing data. Hopefully this will lay a better foundation for modularity!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5ea6fcf57356" width="1" height="1" alt=""><hr><p><a href="https://medium.engineering/learnings-from-optimising-22-of-our-most-expensive-snowflake-pipelines-5ea6fcf57356">Learnings from optimising 22 of our most expensive Snowflake pipelines</a> was originally published in <a href="https://medium.engineering">Medium Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to delete old data from DynamoDB without spending thousands]]></title>
            <link>https://medium.engineering/how-to-delete-old-data-from-dynamodb-without-spending-thousands-0ce3e48ef4a1?source=rss-7e93bd9fc8de------2</link>
            <guid isPermaLink="false">https://medium.com/p/0ce3e48ef4a1</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[hosting-cost]]></category>
            <category><![CDATA[dynamodb]]></category>
            <dc:creator><![CDATA[Raphael Montaud]]></dc:creator>
            <pubDate>Wed, 24 Jul 2024 17:48:10 GMT</pubDate>
            <atom:updated>2024-07-24T17:48:10.804Z</atom:updated>
            <content:encoded><![CDATA[<p>Like in many companies, you probably have some databases lying around that are starting to cost a lot in terms of storage. You are paying more and more every week; the more you wait, the more money will be spent for nothing. It’s time for a cleanup.</p><p>The basic approach is to write a script that will scan your DB and delete the items that aren’t useful anymore (e.g., really old data).</p><p><strong>Problem:</strong> that can be extremely expensive with Dynamo DB, so expensive that it will take years to pay off…</p><p><strong>Solution:</strong> One way to tackle this is to migrate all the data you need to a new table. From there, you can delete your old data altogether (which is free). This way, you pay only to migrate the data you need, which is usually a small percentage of your total table.</p><p>In this post I’ll do a small overview of how we did that at Medium and the tools we used to estimate the costs of the different scenarios.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3tiYNrUJQDbc7CwOWMkRXQ.png" /><figcaption>DynamoDB is AWS’s managed noSQL database</figcaption></figure><h3>First things first, clean up the incoming data</h3><p>What happened to your data exactly? Why did this table grow out of proportion? Storage costs of “real” data such as user / user interactions or user / content interactions are usually reasonable.</p><p><strong>In our case, we’ve been storing things that weren’t meant to be stored forever:</strong></p><ul><li><strong>feature logs:</strong> that’s data on the user and post state at the moment we recommend it to a user — Although this data is useful to retrain our algorithm, we don’t need to store this forever.</li><li><strong>impressions</strong>, i.e. posts we presented in a feed to a user — This is useful to make sure we don’t show a user the same recommendations over and over again, but we don’t need to store this for longer than a few months.</li></ul><blockquote>One of the mistakes we made with presentation data (meant to expire at some point) is that we stored this alongside persistent data in a UserPostRelations table. It’s not easy to maintain items that aren’t meant to have the same lifecycle in the same table. We should have stored this into two separate tables.</blockquote><ul><li>emails sent — We store email contents we send to users (eg: the Medium Daily Digest). We use those to make sure we don’t send the same stories several times to the same user, but again, we don’t need to store that forever.</li></ul><p>The first thing to do is to make sure that your new incoming items will be removed after some time. Luckily, that’s really easy to setup with DynamoDB: All you need to do is to configure the field DynamoDB should be using as the expiration date, and then you can simply set that attribute with the proper expiration date when you insert or update items. DynamoDB automatically deletes expired items and doesn’t charge you for those deletions.</p><p>Now that the incoming data is handled, we can look into ways to clean up the existing items in the table.</p><h3>Get rid of your bad items: Cost estimations</h3><p><strong>We have three possible paths forward here:</strong></p><ul><li>do nothing and let the table grow indefinitely (along with your costs!)</li><li>delete the “bad” items</li><li>migrate the “good” items to a new table and then delete the old table</li></ul><p>To understand these options more fully, let’s do some cost estimations using one of our Dynamo tables at Medium as an example. This table is pretty massive with 105 billion items and an average item size of 100 bytes. Since we’re concerned about short-term spend as well as long term, let’s estimate total expenses after 6 months and after 3 year.</p><p>For this table, 93% of the items are not useful anymore and can be deleted. We were able to estimate that because this table is exported to Snowflake for analytics purposes.</p><h3>Scenario 1: Do Nothing:</h3><p>In this scenario, your storage costs should remain more or less the same over time. Thanks to the automatic item expiration described above, new items are now “clean” and will expire after some time, but old items without an expiration date will stay forever.</p><p>To view the storage costs, we use the AWS cost explorer tool. You can get a view by month and group costs by “Usage Type.” We added tags from the AWS console to our Dynamo table; this way we can filter by Dynamo table in the Cost Explorer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uk_HBfeVhnE8iss6M4zJhw.png" /><figcaption>Screenshot for a cost explorer view</figcaption></figure><p>As you can see, we’re currently spending $7.2k every month in storage costs (”TimeStorage-ByteHrs”, which is regular storage, as well as “Point In Time Recovery” costs). In this scenario, after 6 months, we will have spent $43k. After 3 years, the amount spent jumps to $259k.</p><h3>Scenario 2: Delete “bad” items:</h3><p>This is the brute force method. You need to scan your database and delete each “bad” item, but be aware: Dynamo will charge you for each read and write in this case. To avoid scanning the whole database, you can either figure out the right global secondary index to set up on your table or you can use your analytics data (At Medium, most of our Dynamo tables are exported to Snowflake).</p><p>To estimate the costs, you can use <strong>AWS’ cost estimator:</strong> <a href="https://calculator.aws/#/addService">https://calculator.aws/#/addService</a></p><p>For this table, I found that it would cost around $2k to scan the table and $19k to delete the bad items. That’s taking into account all the best optimisations: The tables should be provisioned beforehand and the traffic should be stable throughout the migration.</p><p>In this scenario, we pay $21k up front — but then, since we deleted 93% of our table, we’ll be paying much less in storage costs after the cleanup. If we consider that the good and bad items are the same size then our new storage costs are now 7% of what we had previously. Accounting for everything, we would have spent $23k after 6 months. After 3 years, we will have spent $39k.</p><h3>Scenario 3: Migrate only the “good” items:</h3><p>This option is the most complex but is also the cheapest. In our use-case, we still have some incoming data and we can’t allow our services to be down at any time. This means we need to do a “hot” migration.</p><p><strong>We do this in 6 steps:</strong></p><ul><li>create a new table (let’s call them Table and TableV2)</li><li>change the logic so that all insertions and updates now update both Table and TableV2. We’re now in a double-write situation.</li><li>backfill all the good items from Table to TableV2. For that you’ll need to perform a scan on Table and an insertion in TableV2 for each good item</li><li>you can then switch all read traffic to TableV2</li><li>stop writing updates to Table</li><li>delete Table</li></ul><p>That option is really cheap, in our case we’d pay $2k for scanning Table and then $1.4k to insert the “good” items into TableV2.</p><p>After that, we continue to pay storage costs, but only for the good items (just like in Scenario 2).</p><h3>Conclusion:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/747/1*g9xzS69DUK1CjZHHs4sU7Q.png" /><figcaption>Pros and cons summary</figcaption></figure><p>For this particular table, the “delete bad items” option is viable. It’s a bit more costly in terms of hosting costs but requires less engineering time and is less risky. Again that really depends on your use-case. We’ve had use-cases where tables were used across all our codebase, with high traffic and any downtime would cause our product to be severely downgraded. This makes migrations much harder and riskier. In other use-cases the table is used only by an isolated admin tool and accidental downtime is more acceptable. In that example, migrating to a new table would be much easier.</p><p>We’ve had other examples where the “delete bad items” option was not viable, and we would’ve only seen a return on investment after two years. So the only viable option here was to perform a difficult migration to a new table.</p><p>You should try to reproduce those estimations with your own use case before you start any cleanup work. With the migrations we did, the cost estimations we made beforehand were exact and the savings perfectly matched what we had predicted.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0ce3e48ef4a1" width="1" height="1" alt=""><hr><p><a href="https://medium.engineering/how-to-delete-old-data-from-dynamodb-without-spending-thousands-0ce3e48ef4a1">How to delete old data from DynamoDB without spending thousands</a> was originally published in <a href="https://medium.engineering">Medium Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>