<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sunil Sandhu on Medium]]></title>
        <description><![CDATA[Stories by Sunil Sandhu on Medium]]></description>
        <link>https://medium.com/@sunilsandhu?source=rss-a7b125868703------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*YyGdeaexKWDHzwxXG9v9XQ.jpeg</url>
            <title>Stories by Sunil Sandhu on Medium</title>
            <link>https://medium.com/@sunilsandhu?source=rss-a7b125868703------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 01:54:35 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sunilsandhu/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com" rel="hub"/>
        <item>
            <title><![CDATA[Ten years in software taught me it was never about the code]]></title>
            <link>https://sunilsandhu.medium.com/ten-years-in-software-taught-me-it-was-never-about-the-code-67407b2c5c87?source=rss-a7b125868703------2</link>
            <guid isPermaLink="false">https://medium.com/p/67407b2c5c87</guid>
            <category><![CDATA[business]]></category>
            <category><![CDATA[work]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[software]]></category>
            <dc:creator><![CDATA[Sunil Sandhu]]></dc:creator>
            <pubDate>Tue, 20 Oct 2026 10:43:35 GMT</pubDate>
            <atom:updated>2026-10-20T10:43:35.655Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1vkJoE2iQNd4xBtV4SK-YQ.png" /></figure><p>Ten years ago I had a dead end office job, an unused Law degree, and a desperate desire to build a different future for myself. So I proceeded to spend any free time I had learning how to code. The deeper I went into programming, so did the waters around me.</p><p>When I did eventually force my way into tech, I was well and truly drowning. I didn’t know algorithms, I wasn’t “trained” in software engineering, and I didn’t know any “real” programming languages like C++. I was a person who enjoyed to party and burn the candle at both ends. I assumed every other developer was a savant who spoke in binary. I remember one time a colleague cracked a joke about NP complexity. The joke went over my head but the imposter in me wasn’t going to let that slip out, so I smiled as if I had understood. Naturally, I proceeded to spend the following lunch break googling “NP”.</p><p>In hindsight, I guess you could say the joke was NP-hard to get…</p><p>Today, after a decade of professionally working in the software industry, my relationship with software has entirely inverted. I’ve gone from being consciously incompetent to realizing that a lot of what I used to agonize over doesn’t matter anymore.</p><p>These days, I hardly read the code I push. I don’t need to have taste. I don’t need to write my own tests. I barely even need to produce schemas and types. All of that gets delegated to the LLM. I come up some semblance of a plan of action and press enter. We umbrella this entire approach with the term <em>vibe coding</em>. Software Engineers like to call it: <em>Agentic Engineering</em>. If the spectrum of experience was a line that started with vibe coding and ended with agentic engineering, where you place yourself on that line is probably all that matters these days.</p><h3>The era of disposable code</h3><p>In recent years, my role has shifted from engineering to orchestrating. I am no longer deep in syntactic weeds. I just need to understand the realities of shipping: security concerns, database efficiency, caching, hosting costs, and token spend. That’s it.</p><p>About ten years ago, I hand-coded a Python bot. Every morning before work, I’d turn my computer on and run the script. It would launch a browser, log into my Instagram account, search for coding-related hashtags, and leave pre-written comments on all the posts it found. I even had to write a secondary script just to shut my computer down after 30 minutes.</p><p>At the time, this felt like a little feat of engineering. There were all sorts of moving parts to figure out. The browser automation, the login, the searching, the commenting, and the scheduling. I had to think through how to get the whole thing working.</p><p>Fast forward to today: I have about 15 automations up in the cloud running daily, silently handling different parts of my job.</p><p>Each one of those automations was vibe-coded and probably took me less time than it did to code that Python bot all those years ago. The difference isn’t just that I can get the same task done faster. It’s that the economics of deciding whether something is worth automating have changed completely.</p><p>Ten years ago, I might have spent a morning writing a script to save myself a few minutes every day. Today, I can build a tool for a task I’ve only encountered once or twice because the cost of creating it is so low.</p><p>This has fundamentally changed how I think about software.</p><p>One of my automations sends me a daily report for all of my businesses. It sends the report by email. Some days I receive just the one email as expected. Other days I receive eight. Such is the nature of non-deterministic systems.</p><p>I should probably tell the automation to just send one email. But then again, sometimes it starts the process by sending a test email to check that it’s working.</p><p>Rather than trying to diagnose the issue, I update the automation by adding these two lines to the end of the prompt:</p><p>- You do not need to send any test emails. — You should only send one email with the full report.</p><p>That’ll fix it!</p><p>Code is no longer built for robustness and stability. More lines of code are being produced than ever before. Got an idea for an app? You don’t agonize over the architecture, you just prompt it into existence. Was the idea shit? Just forget it ever existed. Projects are disposable. The code is ephemeral. The sheer volume of new databases being deployed is insane. Hosting costs are higher than ever, fuelled by the vibes.</p><p>The bottleneck used to be the code and the supply of engineers. Now it’s which models does your Premium Ultra Pro Plus plan offer and how many tokens do you have left.</p><h3>Old paradigms die hard</h3><p>In 2018, I wrote an article titled <em>“I created the exact same app in React and Vue”</em>. It went viral and became a staple format because developers were obsessed with comparing syntax and framework philosophies.</p><p>That kind of content became a recognisable part of the developer education landscape. You could learn React, compare it with Angular, explore a new JavaScript library, and follow along as someone built an application from scratch.</p><p>In 2021, I started Stackademic to help people learn how to code and make software development more accessible. I still believe that was a worthwhile thing to do. Learning to build software opens up enormous opportunities, and I wanted to help more people access them.</p><p>But if I were starting a coding education company today, I would approach it very differently.</p><p>Recently, I saw a video that followed the same format as my React versus Vue article. The creator had built the same application using Claude, Codex, and Cursor. The framework wasn’t the point anymore. The models and the tools were.</p><p>We used to care deeply about the latest library releases. Updates, breaking changes, bug-fixes and new features were the talk of the town. Now, all we care about are the latest model releases.</p><p>That feels representative of a much bigger shift. The interesting question is no longer necessarily how to implement something in a particular programming language. It’s what you can build when you have access to an AI agent that can implement things for you.</p><p>And yet, a lot of coding education still starts from the assumption that you need to learn programming in much the same way you did ten years ago. You learn syntax, loops, algorithms, and the fundamentals of a language before you eventually get around to building something useful.</p><p>I’m increasingly unconvinced that this should be the default path for everyone who wants to make software. The journey should no longer be focused around code as the bottleneck for zero-to-MVP. That timeframe has been compressed to hours.</p><p>I’m not saying algorithms are useless or that absolutely nobody needs to understand programming fundamentals. If you want to work on compilers, operating systems, performance-critical infrastructure, or complex production systems, deep technical knowledge is advantageous. But even that will dissipate with time.</p><p>Generally most people don’t want to become specialists in computer science. They want to build things. They have an idea for an application, a tool that would make their lives easier, or a business they want to start.</p><p>For those people, I think we should be asking a different question: why are we making them spend months learning the mechanics of implementation before letting them explore what they can create?</p><p>If I were to start a bootcamp today, I’d probably begin by introducing people to a few different AI coding tools, such as Claude Code, Codex, and Cursor. I’d let them try all three — along with any others that emerge — because the new hills to die on are which models and tools you prefer. I’d show them how to go from an idea to a working application, how to get that application online, and how to keep experimenting until they had something they were proud of.</p><p>Then I’d largely get out of their way.</p><p>Sure, I’d touch on design, product thinking, marketing, and finding users. Those things matter when you’re trying to turn a project into something people actually use. But I wouldn’t want to turn the whole thing into another business course with a little bit of AI sprinkled on top.</p><p>The main objective would be to teach people how to use the tools, develop their curiosity, and give them the confidence to see an idea through. Once they understood what was possible, I’d want them to spend most of their time building whatever they could imagine.</p><p>The old paradigm of a coding bootcamp would last three to six months. The new paradigm can be completed within a couple of weeks.</p><p>The goal wouldn’t be to produce a cohort of people who all know the same set of frameworks. It would be to produce people who have discovered that they can build almost anything they can think of, and who have enough curiosity to keep pushing at the boundaries. I suspect the most important skill we’d teach wouldn’t be programming at all.</p><h3>The billion dollar mistake</h3><p>I’ve made plenty of questionable career decisions over the years. Quitting jobs, going solo, hiring staff, firing staff, shrinking back down, and growing again. Some of those decisions worked out better than others, but I’ve generally managed to keep things moving in the right direction.</p><p>I’ve also had plenty of ideas for software that went absolutely nowhere.</p><p>Towards the end of 2022, I spent months working on a product that would let you describe an application in a chat interface and have AI build and deploy it for you. You’d type what you wanted, the system would generate the code, and you’d see the end result in the UI. You could even inspect the code and make changes yourself if you fancied it.</p><p>Sound familiar?</p><p>I eventually abandoned the idea. Part of the problem was that the LLMs available at the time struggled to follow the sheer volume of instructions required to make something like this work reliably. But I also wasn’t particularly confident in my ability to pull the whole thing off. I didn’t have the co-founders I thought I’d need, and I wasn’t sure I could take it where it needed to go on my own.</p><p>So I stopped.</p><p>A few months later, Lovable, Bolt, Base44, Replit, and several other platforms started taking over the programming landscape. Suddenly everyone was talking about building applications by describing what they wanted in plain English.</p><p>I remember thinking about the product I’d abandoned and wondering what might have happened if I’d kept going.</p><p>Of course, it’s easy to look backwards and convince yourself you were one decision away from building a billion-dollar company. Maybe the idea was good but my execution would have been terrible. Maybe I’d have run out of money, failed to find users, or spent another year building something nobody wanted. Having an idea before everyone else doesn’t count for much if you can’t turn it into a business.</p><p>AI has made me reconsider how much confidence you actually need before you start building something. When implementation was expensive, complicated, and dependent on specialist expertise, it made sense to spend a long time evaluating an idea before committing to it.</p><p>Today, you can get much further before you need to make those commitments.</p><p>You can test the idea. You can build the first version. You can put it in front of people. You can discover nobody wants it. And you can move on without having spent months assembling a team and building an elaborate technical foundation. And because it turned out to be a terrible idea, you haven’t necessarily lost much beyond a few afternoons and some tokens.</p><p>I still make bad bets. I still abandon projects. But the cost of finding out whether an idea has potential is lower than it has ever been.</p><p>I sometimes wonder how many ideas I’ve talked myself out of over the years because I wasn’t sure I was capable of building them. If we had today’s technology back then, I could have spent that time building a rough version and finding out whether they were any good. Instead, I spent it worrying about whether I could build them.</p><p>That’s a pretty expensive way to do things.</p><h3>Building the software was never the hard part</h3><p>There’s something else I’ve come to appreciate as I’ve moved away from client work and spent more time building my own products.</p><p>When I was doing client and agency work, the problem was already solved, at least in one important sense. Someone had decided they wanted a website or an application. They had a problem, they had a budget, and they’d hired me to build a solution.</p><p>I didn’t need to convince them that the problem existed. I didn’t need to figure out where to find customers. I just needed to deliver the software.</p><p>But when you step into the arena as a founder, you realize that building the product is only 10% of the job. You are entirely responsible for the marketing and the distribution.</p><p>Ten years ago, being a good developer gave you a meaningful advantage. If you could build something technically impressive that solved a real problem, you had something that most people couldn’t easily replicate. The time, expertise, and resources required to build software acted as a barrier to entry. But for a growing category of software, the cost of building a functional first version has plummeted.</p><p>Your competitor can probably build something that looks good, works reasonably well, and solves the same basic problem. So can the next person. And the next.</p><p>The novelty wears off pretty quickly when everyone has access to the same tools.</p><p>You can’t assume people will discover your product just because you’ve built it well. You can’t assume you’ll attract millions of users because the application feels magical. And you can’t assume that being the best developer in the room will make you the most successful founder.</p><p>Distribution matters. Reputation matters. Understanding your audience matters. The relationships you’ve built over the years can matter enormously.</p><p>I’ve spent much of my career building publications, communities, and audiences around technology. For years, these were things I worked on alongside the actual business of building software. I wanted to publish articles, help people learn, and create places where developers could share what they knew.</p><p>Looking back, those things have become some of my most valuable assets as a founder. It’s funny, because I spent so much of my career trying to prove myself by building software, partly because the community-building work I was good at never quite made me feel like a proper engineer. The thing I spent years trying to prove I could do has turned out to be less valuable than the thing I was already good at. And the thing I used to push back against has become my superpower.</p><p>When I want to launch something new, I have an existing network of publications and audiences that can help me get it in front of people. I don’t have to start entirely from scratch every time I have an idea.</p><p>That’s a rather different kind of advantage from being able to write better code than the next person. And it’s one that becomes more valuable when everybody can build.</p><p>You can have the best product in the world, but if nobody knows it exists, you’re not going to get very far. You can spend a weekend building something incredible and then spend the next six months trying to get anyone to use it.</p><h3>So, what makes a good developer now?</h3><p>When I look back at the past ten years, I find it quite funny how much time I spent worrying about things that barely feature in my working day anymore.</p><p>I used to think I needed to know more programming languages, understand more algorithms, and write better code. I worried about whether I was technically good enough to call myself a software engineer.</p><p>These days, I spend a lot of my time telling AI agents what I want them to build, reviewing the results, and figuring out what to work on next.</p><p>I’m more productive than ever, and I feel less constrained by my own technical ability.</p><p>That doesn’t mean I’ve stopped caring about how software works or that technical knowledge has become worthless. If you’re building critical systems, you still need to understand what you’re shipping. If you’re responsible for security, reliability, and performance, you can’t just cross your fingers and hope the agent got everything right.</p><p>But it does mean that technical abilities (ie programming skill) is no longer the barrier to entry. Anyone who wants to build software no longer needs to start by becoming an expert in the mechanics of programming.</p><p>For a long time, that was simply the price of admission. If you wanted to build an application, you had to learn how to code it. You had to understand the tools, the languages, and the frameworks before you could get very far.</p><p>But if your ambition is to build things, I think we’re entering a period where curiosity can take you further than ever before.</p><p>A good developer, in my eyes, is increasingly someone who is curious enough to experiment with new tools, persistent enough to work through problems, and imaginative enough to see possibilities that others haven’t explored. They know how to describe what they want, evaluate what they’ve been given, and keep pushing until the result is good enough.</p><p>They understand that getting something working is only the beginning.</p><p>And if you want to build a business, there’s another set of skills you’ll need. You have to understand the people you’re building for, figure out how to reach them, and give them a reason to choose your product over everything else available to them.</p><p>As I’ve already discovered, getting the software built is the easy part. Getting people to care about it is another story.</p><p>Ten years ago, I thought I needed to become a different kind of person to belong in software. I had a law degree I wasn’t using, a mountain of imposter syndrome, and a pretty clear idea of what I thought a software engineer was supposed to look like. They were the people who knew all the algorithms, understood the intricacies of their programming languages, and could write elegant code without having to look anything up.</p><p>I thought I needed to become one of those people.</p><p>Before writing code, I wanted to be a writer. I spent a lot of time journaling, writing articles, short stories, poetry, and thinking about all the ways I could turn words and sentences into elegant prose. I was interested in people, their experiences, and the things that made them tick. Somewhere along the way, I started treating those interests as separate from my work in software. My curiosity about the human experience took a back seat to my curiosity about Big O notation.</p><p>I guess I thought I had to choose.</p><p>Looking back, I find that quite funny. The things I was interested in outside of programming have turned out to be some of the things that matter most in building products and businesses. Understanding people, communicating ideas, and being curious about the world are pretty useful skills when you’re trying to make something people actually want.</p><p>Maybe it just took ten years for me to figure that out.</p><p>The tools have changed, and so have I. I can spend a morning building something that would once have taken weeks, then spend the rest of the day thinking about what else might be possible. Sometimes I’ll have an idea for something completely unrelated to my work and find myself building a rough version of it just to see what happens. I recently wanted to work on improving my drawing ability, specifically anatomical objects. Within 30 minutes of prompting, I had an app that showed me three-dimensional human head that I could view from any angle and then use that as a drawing reference.</p><p>What a time to be alive!</p><p>You don’t have to spend years becoming the kind of person you think you’re supposed to be before you start making things. You can start with what interests you, learn what you need along the way, and see where it takes you.</p><p>Curiosity is the currency. Go make something with it.</p><p>Long live coding in plain English!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=67407b2c5c87" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[SMTP vs Email APIs: Which Should You Use in 2026?]]></title>
            <link>https://sunilsandhu.medium.com/smtp-vs-email-apis-which-should-you-use-in-2026-9da7440e839a?source=rss-a7b125868703------2</link>
            <guid isPermaLink="false">https://medium.com/p/9da7440e839a</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[api]]></category>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[coding]]></category>
            <dc:creator><![CDATA[Sunil Sandhu]]></dc:creator>
            <pubDate>Mon, 05 Oct 2026 13:21:52 GMT</pubDate>
            <atom:updated>2026-10-05T13:21:52.062Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6LlGcT-Q_-olIAVc-IKOqg.png" /><figcaption>SMTP vs Notify’s Email API</figcaption></figure><p>I still see new SaaS projects reach for Nodemailer on day one. Not because they’ve evaluated SMTP carefully — because every tutorial from 2018 does it that way.</p><p>SMTP isn’t dead. It’s how mail servers talk to each other and always will be. But there’s a difference between “SMTP exists in the email ecosystem” and “my Express app should open a connection to port 587.” For most new apps, that second part is a trap.</p><h3>Two different layers</h3><p>When people say “SMTP vs email API,” they’re usually comparing two ways your <em>application</em> talks to an email provider — not whether SMTP exists at all.</p><p><strong>SMTP</strong> is the protocol mail transfer agents use to hand messages off to each other. Gmail’s servers speak SMTP to Outlook’s servers. Your provider speaks SMTP to Gmail. That layer isn’t going anywhere.</p><p><strong>An email API</strong> is an HTTPS endpoint. Your backend sends JSON, authenticates with an API key, and the provider handles queuing, retries, and delivery. Your app never touches port 25.</p><p>Think of it as two hops. <strong>Your code → your provider</strong> is where you choose: HTTPS API (what I’d recommend) or an SMTP relay if you’re stuck with legacy tooling. <strong>Your provider → Gmail, Outlook, etc.</strong> is SMTP between mail servers — Notify, Resend, SES, all of them handle that part for you. You don’t touch it.</p><p>The confusion shows up when someone debugs ECONNREFUSED on port 25 from a Vercel function. The host blocked outbound SMTP years ago. The fix isn&#39;t tweaking Nodemailer config — it&#39;s using HTTPS instead.</p><p>I wrote more about why this shift happened in <a href="https://notify.cx/blog/why-smtp-is-no-longer-the-default">why SMTP is no longer the default</a>.</p><h3>Why app-level SMTP hurts on modern infra</h3><p>The failure modes are predictable once you’ve hit them once:</p><p>Port 25, 465, and 587 get blocked or rate-limited on PaaS and serverless platforms. What worked on your laptop dies in production. Shared VPS IPs carry reputation baggage — your password resets land in spam because someone else’s cron job got the IP blacklisted. There’s no bounce pipeline unless you build one. Credentials end up on every server that sends mail, and one leak turns into an open relay.</p><p>Serverless makes it worse. Cold starts plus SMTP handshakes equals flaky timeouts. I’ve watched teams spend days tuning relay settings on a platform that was never going to reliably support outbound SMTP anyway.</p><p>Provider SMTP relays — where Nodemailer talks to smtp.sendgrid.net instead of directly to Gmail — are a reasonable middle ground for legacy apps. Raw &quot;send from the droplet&quot; SMTP is technical debt that accrues deliverability interest.</p><h3>When SMTP still makes sense</h3><p>I’m not saying burn it all down.</p><p>On-prem software that only exposes SMTP settings in a config panel? Use the provider’s SMTP relay. You’re migrating off a fragile VPS setup but need something working this week? Relay now, HTTP later. You operate your own MTA for fun? Sure, SMTP is your whole job.</p><p>Even then: authenticate your domain, watch bounces, and don’t send marketing blasts from the same identity you use for password resets. Transactional reputation and marketing reputation should not share a subdomain.</p><h3>Where HTTPS APIs win</h3><p>Password resets, magic links, receipts, failed-payment notices — these are request/response jobs. Something happened in your app, send an email, move on. HTTPS fits that model cleanly.</p><p>It works anywhere fetch works: Node, Deno, Cloudflare Workers, Edge functions. Auth is an API key header instead of username/password on the wire. Errors come back as HTTP status codes with JSON bodies you can actually log. Local dev and production run the same call — no &quot;works on my machine because my ISP doesn&#39;t block port 587.&quot;</p><p>Observability is the underrated win. With an API provider you get per-message logs and webhooks for bounces and complaints without building a pipeline yourself. When support asks “did the reset email go out?” you have an answer in thirty seconds.</p><h3>The Nodemailer trap</h3><p>Nodemailer is a solid library. I’ve used it. The trap is treating “we use Nodemailer” as an architecture decision when it’s really just a transport choice.</p><p>Nodemailer still needs somewhere to send mail. That destination is either blocked SMTP, a fragile relay, or — ironically — a provider you could have called over HTTPS in the first place.</p><p>For new Node code, I’d reach for a transactional API and skip the SMTP client entirely. I wrote a walkthrough for that: <a href="https://notify.cx/blog/how-to-send-transactional-email-nodejs-without-nodemailer">sending transactional email in Node.js without Nodemailer</a>.</p><h3>What an API doesn’t replace</h3><p>Picking an API doesn’t mean you skip DNS. You still need SPF and DKIM on your sending domain — the API replaces the app-to-provider hop, not authentication. I wrote a <a href="https://notify.cx/blog/spf-dkim-dmarc-explained-for-developers">developer-friendly explainer on SPF, DKIM, and DMARC</a> if that’s new territory.</p><p>You still want transactional and marketing mail separated. You still need a plan for bounces — start with logs, add webhooks when bad addresses start costing you money.</p><p><a href="https://notify.cx/docs/domain-verification">Domain verification</a> in Notify (or any provider) walks you through the DNS records. This will either take you 5 minutes or 50 minutes depending on how comfortable you are with modifying DNS records.</p><h3>How I’d decide</h3><p>New SaaS on App Router, serverless, or Workers? Email API, no question. Migrating off a VPS that’s been sending mail directly? Same answer. Vendor tool that only accepts SMTP credentials in a settings form? Fine, use their relay.</p><p>Running your own mail infrastructure because you enjoy pain? SMTP is your life now.</p><p>“We’ll fix deliverability later” while sending from port 25 on a shared host? Please don’t. Ship with an API, fix DNS, and sleep better.</p><h3>What I actually use</h3><p>For transactional mail on side projects and early-stage apps, I use <a href="https://notify.cx/docs/quick-start-guide">Notify</a>. One endpoint, four fields:</p><pre>await fetch(&#39;https://notify.cx/api/email/send&#39;, {<br>  method: &#39;POST&#39;,<br>  headers: {<br>    &#39;Content-Type&#39;: &#39;application/json&#39;,<br>    &#39;x-api-key&#39;: process.env.NOTIFY_API_KEY!<br>  },<br>  body: JSON.stringify({<br>    from: &#39;noreply@your-verified-domain.com&#39;,<br>    to: user.email,<br>    subject: &#39;Reset your password&#39;,<br>    message: html<br>  })<br>});</pre><p>to, from, subject, message. That&#39;s the whole API. You write the HTML (or plain text), Notify delivers it. <a href="https://notify.cx/docs/email-logs-and-tracking">Logs</a> on every plan, <a href="https://notify.cx/docs/webhooks-and-notifications">webhooks</a> on Pro and Scale. Free tier is 1,000 emails/month; Pro is $10 for 10,000 (<a href="https://notify.cx/pricing">pricing</a>).</p><p>No template studio, no newsletter product, no MCP layer you have to install. It’s deliberately small — transactional email for apps, not a marketing suite bolted onto your DNS records. (Yes, there’s a page about <a href="https://notify.cx/mcp">why MCP is overkill for sending email</a>.)</p><p>If you need Resend’s React Email ecosystem, Postmark’s deliverability reputation, or SES unit economics at massive scale inside AWS, use those. They’re good at different things. <a href="https://notify.cx/blog/resend-vs-postmark-vs-ses-vs-notify">This comparison</a> lays out the tradeoffs without pretending one size fits all.</p><p>Migrating off SMTP you already have running? <a href="https://notify.cx/docs/migrate-from-smtp">This guide</a> maps the steps.</p><h3>After you pick</h3><p>Verify your domain. Write one server-side send helper. Ship verification email, password reset, and a receipt. Read the logs for a week before you worry about webhooks.</p><p>That’s the whole stack for most startups. <a href="https://notify.cx/blog/minimal-transactional-email-stack-for-startups">More detail here</a> if you want it spelled out, or just start with the <a href="https://notify.cx/docs/quick-start-guide">quick start</a>.</p><p>SMTP still delivers most of the world’s email. Your app doesn’t need to be part of that conversation.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9da7440e839a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Obsurfable just made all of their private LLM visibility data freely available to the public]]></title>
            <link>https://ai.plainenglish.io/obsurfable-just-made-all-of-their-private-llm-visibility-data-freely-available-to-the-public-9909ae86dc4d?source=rss-a7b125868703------2</link>
            <guid isPermaLink="false">https://medium.com/p/9909ae86dc4d</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[marketing]]></category>
            <category><![CDATA[seo]]></category>
            <category><![CDATA[aeo]]></category>
            <dc:creator><![CDATA[Sunil Sandhu]]></dc:creator>
            <pubDate>Wed, 02 Sep 2026 09:06:05 GMT</pubDate>
            <atom:updated>2026-09-02T09:07:47.475Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Search is changing faster than most marketing teams can adapt.</strong> People are not clicking ten blue links anymore. They are asking ChatGPT, Gemini, and other AI assistants a question and trusting the single answer they get back. If your brand is missing from that answer — or worse, misrepresented — you will never see the traffic you lost in Google Analytics. There is no click to measure. There is only absence.</p><p>For the past year, a growing number of tools have sprung up to help companies track how they appear in AI-generated answers. Answer Engine Optimization (AEO). Generative Engine Optimization (GEO). AI visibility monitoring. The pitch is usually the same: pay us to run prompts on your behalf, store the responses, and tell you whether you were mentioned, recommended, or ignored.</p><p>That is a reasonable business. It is also, in many cases, a business built around selling you back information that is largely public in the first place.</p><p>Anyone can open ChatGPT and ask <em>“What are the best project management tools for remote teams?”</em> Anyone can ask Gemini the same question tomorrow and read the answer. The response is not secret. It is not proprietary intelligence locked inside a model vendor’s black box in any meaningful sense — it is an answer that was generated in response to a question anyone could ask.</p><p><strong>Obsurfable has decided to treat it that way.</strong></p><p>They have opened our full observation corpus to the public at <a href="https://explorer.obsurfable.com">explorer.obsurfable.com</a>. No signup required to browse. No paywall on the latest answers. No tier that gates basic visibility into how AI describes your category.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E3tth1p7s69DW9swBFTZLA.png" /></figure><h3>What is actually in the corpus?</h3><p>They call each recorded AI response an <strong>observation</strong>. Every observation captures:</p><ul><li>The <strong>prompt</strong> — usually a buyer-style question, like <em>“Which CRM is best for small B2B sales teams?”</em></li><li>The <strong>model</strong> — ChatGPT, Gemini, Claude, Grok, and others as they expand coverage</li><li>A <strong>timestamp</strong> — when the observation was recorded</li><li>The <strong>full response</strong> — what the model actually said</li><li><strong>Brands mentioned</strong> — which companies appeared in the answer, and how</li><li><strong>Citations</strong> — which sources the model relied on, when available</li></ul><p>Those observations are not stored in a spreadsheet behind a login screen. They are the raw material for a public research site. You can browse prompts, search companies, walk category landscapes, and see which brands AI recommends in a given vertical — without creating an account.</p><p>Company pages show mention profiles built from the corpus. Category pages show recommendation landscapes: who shows up, who does not, and which domains get cited most often. Prompt pages show the latest observation and, for contributors, historical runs over time.</p><p>This is not a demo dataset. It is the same class of data that visibility monitoring products charge monthly subscriptions to collect on your behalf.</p><h3>Why give it away?</h3><p>Because the alternative feels wrong.</p><p>Most of what Obsurfable records is already <strong>publicly obtainable</strong>. You could run the same prompts yourself. You could screenshot the answers. You could build a spreadsheet. What you probably will not do — because it is tedious — is run hundreds of buyer prompts across dozens of categories, store every response, extract every brand mention, track citations, and keep it updated as models change.</p><p>That is the work. That is the value of aggregation and consistency. But gating the <em>output</em> of that work behind a subscription, when the underlying answers are not secret, creates a strange incentive: charge brands to see how they are described in systems anyone can query for free.</p><p>They would rather publish the corpus openly.</p><p><strong>A level playing field matters.</strong> A solo founder researching their category should see the same evidence as a Fortune 500 competitive intelligence team. A journalist writing about AI recommendations should not need an enterprise contract to cite real observations. A researcher studying how LLMs favor certain brands should have access to structured data, not just anecdotes.</p><p>Making the corpus public does not mean there is nothing left to build. It means the baseline — <em>what did the model say, about whom, when</em> — should not be a paid unlock.</p><h3>What this means if you care about AI visibility</h3><p>If you are a marketer, founder, or SEO practitioner trying to understand AEO, the first question is not <em>“How do I optimize?”</em> It is <em>“What is actually happening today?”</em></p><p>Go look.</p><p>Search your company. Search your competitors. Browse the category you compete in. Read the prompts buyers actually ask. See whether you appear, whether you are described accurately, and which brands AI recommends instead of you.</p><p>That clarity is free now.</p><p>You do not need to trust a vendor’s proprietary score. You do not need to wonder whether a dashboard is showing you a cherry-picked sample. You can read the observations yourself and form your own view.</p><p>If you want to go deeper — run your own prompts, explore full history, contribute observations back to the shared dataset — you can create a free account on the Explorer. But the bar to entry for basic research is zero.</p><h3>The shift is already happening</h3><p>AI systems are becoming the interface to the internet. Discovery is increasingly conversational. Rankings are increasingly irrelevant to the question <em>“Did the AI mention us?”</em></p><p>That shift is already underway. The brands that understand it early will adapt. The ones that do not will wonder why their traffic slowly erodes while their Google rankings look fine.</p><p>In their words: <em>“We built Obsurfable because we think this visibility should be </em><strong><em>measurable, shared, and accessible</em></strong><em> — not locked behind tiers that recreate the same information asymmetry search engines used to sell.”</em></p><p>The Explorer is live. The corpus is open. They will keep adding prompts, models, and categories as the landscape evolves.</p><p><strong>See for yourself:</strong> <a href="https://explorer.obsurfable.com">explorer.obsurfable.com</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9909ae86dc4d" width="1" height="1" alt=""><hr><p><a href="https://ai.plainenglish.io/obsurfable-just-made-all-of-their-private-llm-visibility-data-freely-available-to-the-public-9909ae86dc4d">Obsurfable just made all of their private LLM visibility data freely available to the public</a> was originally published in <a href="https://ai.plainenglish.io">Artificial Intelligence in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Extract Contact Information from LinkedIn for Market Research]]></title>
            <link>https://javascript.plainenglish.io/how-to-extract-contact-information-from-linkedin-for-market-research-9e5fa5ae540d?source=rss-a7b125868703------2</link>
            <guid isPermaLink="false">https://medium.com/p/9e5fa5ae540d</guid>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[linkedin]]></category>
            <category><![CDATA[bright-data]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Sunil Sandhu]]></dc:creator>
            <pubDate>Sat, 08 Aug 2026 09:39:56 GMT</pubDate>
            <atom:updated>2026-08-08T09:39:56.549Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jNjADrrDbhm5J-fKPxbbag.png" /></figure><p><strong>Quick answer:</strong> Extracting contact information from LinkedIn for market research means collecting <em>public</em> profile and company data — name, title, employer, location, industry, company domain — through a scraper API or pre-built dataset that doesn’t require your own LinkedIn login, then layering on approved enrichment for verified business emails or phone numbers where your use case calls for it. It does <strong>not</strong> mean pulling private emails or phone numbers directly off profile pages at scale; LinkedIn doesn’t expose that data publicly, and LinkedIn’s User Agreement prohibits automated scraping outright.</p><p>The most reliable technical approach for market research teams is a no-login scraper API or dataset — such as<a href="https://brightdata.com/products/web-scraper/linkedin"> <strong>Bright Data’s LinkedIn Scraper API</strong></a> — paired with a validation and compliance step before any record is used. The workflow below covers both the technical steps and the legal ground you need to understand before scaling it.</p><h3>Key Takeaways</h3><ul><li>“Contact information” from LinkedIn scraping is almost always public professional context (name, title, company, location) rather than direct emails or phone numbers — those typically require a separate, approved enrichment step.</li><li>Tools that collect public data via API or dataset — without logging into your own LinkedIn account — carry meaningfully lower account-ban risk than automation that acts from your own session.</li><li>LinkedIn’s User Agreement bars automated scraping outright, and enforcement is active and current: LinkedIn sued LinkedIn-data vendor Proxycurl in January 2025, and the company shut down by mid-2025 despite an estimated $10M in annual recurring revenue.</li><li>The <em>hiQ Labs v. LinkedIn</em> litigation established that scraping publicly available data generally doesn’t violate the U.S. Computer Fraud and Abuse Act — but hiQ still lost on a separate breach-of-contract theory and was permanently enjoined from scraping LinkedIn in December 2022.</li><li>Using scraped LinkedIn data for outreach brings GDPR, CAN-SPAM, and CASL into play simultaneously if your prospects span jurisdictions — most B2B cold outreach relies on GDPR’s “legitimate interest” basis, which requires documentation, not just public availability.</li><li>A defined data schema, deduplication, and a confidence-scoring pass before any record reaches a CRM or outreach tool consistently separates workable market-research data from unusable noise.</li></ul><h3>What Does It Mean to Extract Contact Information from LinkedIn?</h3><p>Extracting contact information from LinkedIn usually does <strong>not</strong> mean pulling private emails or phone numbers directly from a profile at scale. In most real-world market research workflows, it means collecting publicly visible professional data such as:</p><ul><li>Full name</li><li>Job title</li><li>Company name</li><li>Location</li><li>Industry</li><li>Profile URL</li><li>Company website</li><li>Seniority or department signals</li></ul><p>That data supports market research use cases including:</p><ul><li>Competitive and market mapping</li><li>Industry hiring and workforce trend analysis</li><li>Account-based research</li><li>Partnership and vendor landscape research</li><li>CRM and sales-intelligence enrichment</li><li>Recruiting pipeline research</li></ul><p>In practice, the “contact information” a market research team ends up with is a combination of three layers: <strong>public LinkedIn profile data</strong>, <strong>company-level contact paths</strong> such as corporate domains, and <strong>enriched business contact data</strong> sourced separately and lawfully. A LinkedIn scraper is strongest at the first layer — structured public professional information. Whether you can lawfully derive or append emails or phone numbers depends on your jurisdiction, your data source, and your purpose for processing.</p><h3>How to Extract Contact Information Using a LinkedIn Scraper, Step by Step</h3><p>The most reliable approach breaks the job into ordered stages: define what you need, collect public profile data, structure it, enrich it where appropriate, validate the output, and stay compliant throughout.</p><h3>Step 1: Define the Data You Actually Need</h3><p>Start by narrowing your target fields before scraping anything. A useful minimum dataset for market research typically includes: first and last name, LinkedIn profile URL, current company, current role, company website or domain, location, and a public summary or headline. Defining this schema up front prevents over-collection — which creates storage, compliance, and data-quality problems — and gives legal or security teams a clear, auditable scope to review.</p><h3>Step 2: Choose a LinkedIn Scraper for the Job</h3><p>The best LinkedIn scraper for market research reliably collects public profile and company data at scale, handles LinkedIn’s dynamic page structures, and fits into a broader enrichment workflow. Look for: support for profile and company page extraction, structured JSON/CSV/API output, anti-blocking infrastructure (proxy rotation, browser automation), batch or pagination handling, and clear documentation of rate limits.</p><p><a href="https://brightdata.com/"><strong>Bright Data</strong></a> is a widely used option for this kind of workflow. Its<a href="https://brightdata.com/products/web-scraper/linkedin"> <strong>LinkedIn Scraper API</strong></a> returns structured data on profiles, companies, jobs, and posts without requiring your own LinkedIn credentials — priced from roughly $1.50 per 1,000 records pay-as-you-go, dropping to $0.75–$0.98 per 1,000 records on a subscription, with a free tier for evaluation. For bulk historical research, <a href="https://brightdata.com/products/datasets/linkedin"><strong>LinkedIn Datasets</strong></a> offer pre-collected profile and company records filtered by geography, industry, or title. Both sit on Bright Data’s broader proxy infrastructure and carry GDPR, CCPA, ISO 27001, and SOC 2 certification, documented at its <a href="https://brightdata.com/trustcenter"><strong>Trust Center</strong></a>. Apify and Oxylabs are commonly cited alternatives for teams that want to build more custom extraction logic.</p><p>A dedicated platform matters here because LinkedIn pages are dynamic, its anti-bot defenses are sophisticated, and maintaining your own proxy rotation, session handling, and fingerprint management can become its own engineering project. That overhead is a poor trade for a market research team whose real goal is analysis, not scraping infrastructure.</p><h3>Step 3: Collect Public Profile Data Safely and Consistently</h3><p>Once you have a tool, collect records from one of three inputs: a list of profile URLs, a search result set by role/industry/geography, or a list of companies followed by employee discovery. URL-based collection is generally the cleaner path since it produces more stable, record-level extraction than search-result scraping.</p><p>At minimum, capture name, headline, current role, employer, location, profile URL, public summary text, and — where relevant to the research question — seniority and department signals (manager, director, VP, founder; sales, marketing, engineering, HR). Improve output quality by standardizing profile URLs, deduplicating early, normalizing company names, separating current from past roles, and storing an extraction timestamp — stale B2B data remains one of the biggest causes of wasted outreach and unreliable market analysis in 2026.</p><h3>Step 4: Turn Profile Data into Usable Contact Information</h3><p>This is where many teams misjudge the workflow. A LinkedIn scraper delivers strong professional identity data; turning that into usable contact information requires <strong>enrichment</strong>, not just scraping. Think of it in three layers:</p><ul><li><strong>Identity and role data from LinkedIn</strong> — who the person is, where they work, what they do, and how senior they appear to be. This layer tells you whether a record is relevant.</li><li><strong>Company contact path</strong> — from the employer name, identify the company website, corporate domain, and official contact or leadership pages. General business emails, press contacts, or departmental forms are often more appropriate for market research outreach than a personal inbox.</li><li><strong>Approved business-contact enrichment</strong> — where your use case allows it, append business contact data from sources that are lawfully obtained and documented for professional outreach. Personal work emails or phone numbers typically come from this layer, not from LinkedIn itself.</li></ul><p>Don’t assume every LinkedIn profile maps to a valid work email, that public visibility equals permission for unrestricted marketing use, or that a single source stays accurate indefinitely. The strongest workflows treat LinkedIn as a high-value source of professional context, not a guaranteed database of direct contact details.</p><h3>Step 5: Clean, Validate, and Verify Extracted Records</h3><p>Raw scraped data is rarely ready for research or outreach. Run a validation pass that checks for duplicate profiles and people, outdated employer information, broken or malformed URLs, missing key fields, inconsistent formatting, and domains that no longer resolve. For enriched emails or phone numbers, add separate verification logic. A simple confidence-scoring model — high (role, domain, and contact method all confirmed), medium (role and company matched, contact unverified), low (partial data, weak match) — lets downstream teams prioritize the strongest records and route lower-confidence ones to manual review.</p><h3>Step 6: Staying Compliant When Collecting LinkedIn Data</h3><p>The single biggest factor in your legal exposure is whether the collection method logs into an account at all. Courts have consistently drawn a sharp line between the two.</p><p><strong>No-login, public-data collection is the defensible side of that line.</strong> In <em>hiQ Labs v. LinkedIn</em>, the Ninth Circuit held (2019, reaffirmed 2022) that scraping publicly available data — nothing behind a login — generally doesn’t violate the Computer Fraud and Abuse Act, since there’s no “unauthorized access” to data anyone can already see. Meta v. Bright Data (Jan. 2024) reinforced this from the other side: the court found Bright Data’s “logged-off” scraping of Facebook/Instagram wasn’t bound by Meta’s Terms of Service, since it wasn’t acting as a logged-in “user” when it collected the data. This is also how Bright Data’s LinkedIn tooling is built — its Scraper API requires no LinkedIn credentials and returns structured public data (profiles, companies, jobs, posts) via API call rather than through an authenticated session.</p><p><strong>Logged-in or fake-account collection is where enforcement actually lands.</strong> hiQ’s CFAA win didn’t save it: a district court separately found hiQ breached LinkedIn’s User Agreement, and in December 2022 hiQ entered a $500K consent judgment, admitted liability for trespass to chattels and misappropriation, and shut down. The Proxycurl case follows the same fact pattern: LinkedIn and Microsoft sued in January 2025 over hundreds of thousands of fake accounts used to scrape millions of profiles; Proxycurl (then ~$10M ARR) settled, shut down, and deleted its LinkedIn data by mid-2025. <em>Facebook v. Power Ventures</em> is the earlier version of the same story. In each case, the exposure came from acting as an authenticated (or fraudulently authenticated) user bound by the platform’s terms — not from touching public data itself.</p><p>Two caveats worth keeping in view. First, LinkedIn’s User Agreement prohibits scraping outright regardless of method — the no-login approach reduces legal exposure, it doesn’t eliminate it, and LinkedIn has taken other action even against no-scraping-litigation targets (it removed Apollo.io’s and Seamless.ai’s Company Pages in 2025 over ToS concerns). Second, if the data feeds outreach, GDPR, CAN-SPAM, and CASL apply regardless of collection method — GDPR’s Article 6(1)(f) “legitimate interest” basis requires a documented three-part test (necessity, balancing, role-relevance), not just public availability; CAN-SPAM requires honest sender ID and a working opt-out (~$53K per violation); CASL requires consent before first send (up to $10M per violation).</p><p><strong>Practical filter for vendor selection:</strong> does the tool collect via API/dataset with no LinkedIn login required (lower exposure, matches the hiQ/Meta reasoning), or does it operate through your logged-in session or purchased/automated accounts (higher exposure, matches the hiQ-breach/Proxycurl pattern)? When in doubt, loop in legal counsel or a privacy specialist before scaling.</p><p><strong>Lower-risk market research uses</strong> include aggregate company and industry research, talent and hiring-trend mapping, account-based planning using role and organization data, and building lists for manual review before any contact is made. <strong>Higher-risk uses</strong> include credential-based automation, unsolicited high-volume outreach without a documented lawful basis, and reselling personal data. When in doubt, involve legal counsel or a privacy specialist before scaling the workflow.</p><h3>Step 7: Automate Without Losing Control</h3><p>Once the workflow works for a small batch, a practical pipeline typically follows this order: import target profile URLs or company names, run extraction, normalize names/companies/titles, match companies to domains, run approved enrichment, validate records, and push only qualified records into a CRM or research repository. Keep guardrails in place as you scale: batch-size limits, logging and audit trails, error-rate monitoring, human review for sensitive cases, and scheduled refreshes instead of constant recrawling.</p><h3>Live API vs. Pre-Built Dataset vs. Account-Based Automation</h3><p>The three technical approaches to LinkedIn data collection carry very different risk and freshness profiles:</p><ul><li><strong>Live scraper API (no login required)</strong> — collects fresh, targeted records on demand; account-ban risk to you personally is low since it doesn’t use your credentials; best for ongoing or targeted research where currency matters.</li><li><strong>Pre-built dataset (no login required)</strong> — bulk, pre-collected records filtered by geography, industry, or title; shifts collection-time compliance exposure toward the vendor; best for historical backfill or large-scale mapping where a slightly older snapshot is acceptable.</li><li><strong>Account-based automation (uses your LinkedIn login)</strong> — acts from your own session, which is exactly the fact pattern that has drawn enforcement in cases like <em>hiQ</em> and <em>Proxycurl</em>; carries the highest account-ban and legal-exposure risk of the three, and is generally the option to avoid for any recurring or scaled workflow.</li></ul><p>Many mature market research programs combine the first two: a dataset for cold backfill, plus a live API for what’s changed recently.</p><h3>Pros and Cons of Using a LinkedIn Scraper for Contact Extraction</h3><p><strong>Pros:</strong></p><ul><li><strong>Fast research at scale</strong> — collect professional context far faster than manual copying</li><li><strong>Better targeting</strong> — titles, seniority, and company data sharpen prospect or research-subject selection</li><li><strong>Stronger enrichment inputs</strong> — clean name-company-role combinations improve downstream match quality</li><li><strong>Fresh signals</strong> — profile changes can reveal job moves, promotions, or hiring activity relevant to market analysis</li><li><strong>Scalable across use cases</strong> — supports market mapping, competitive research, and account-based planning alike</li></ul><p><strong>Cons:</strong></p><ul><li><strong>Direct contact details are limited</strong> — LinkedIn itself isn’t a guaranteed source of emails or phone numbers</li><li><strong>Compliance complexity</strong> — personal data use requires careful, documented review</li><li><strong>Variable data quality</strong> — profiles can be outdated, incomplete, or inconsistently formatted</li><li><strong>Technical maintenance</strong> — dynamic pages and anti-bot systems create ongoing overhead for self-built scrapers</li><li><strong>Enforcement risk</strong> — LinkedIn has continued to pursue vendors and account-based automation tools through 2025 and into 2026</li></ul><h3>When Should You Use Bright Data for LinkedIn Scraping?</h3><p>Bright Data is a sensible option once a manual workflow is too slow and a self-built scraper is too fragile to maintain — typically when one or more of the following are true: you need LinkedIn data at operational scale, you want structured output instead of raw HTML, you need resilient collection infrastructure rather than a script that breaks on every layout change, or you want documented compliance certifications (GDPR, CCPA, ISO 27001, SOC 2) to support an internal audit. Because Bright Data is a broader web data platform rather than a single-purpose tool, it also fits research programs that need to combine LinkedIn signals with company-site data or other public sources over time.</p><h3>Common Mistakes to Avoid</h3><ol><li><strong>Treating LinkedIn as a complete contact database.</strong> It’s a professional identity source, not a substitute for a proper enrichment and verification process.</li><li><strong>Scraping more fields than you can justify.</strong> Over-collection creates storage, compliance, and operational problems.</li><li><strong>Skipping normalization.</strong> Job titles and company names vary widely; without normalization, segmentation and enrichment quality drop fast.</li><li><strong>Pushing raw records straight into outreach.</strong> Unverified data damages campaign performance and, over time, sender or organizational reputation.</li><li><strong>Skipping legal and privacy review.</strong> Even public-data workflows need documented purpose, retention limits, and allowable use — especially given how actively LinkedIn has enforced its Terms of Service through 2025 and 2026.</li></ol><h3>Frequently Asked Questions</h3><h4><strong>How can I extract contact information using a LinkedIn scraper?</strong></h4><p>Use a scraper API or pre-built dataset that collects publicly available LinkedIn profile and company information — such as names, job titles, employers, industries, and locations — without requiring your own LinkedIn login. Platforms such as Bright Data provide managed LinkedIn data collection and datasets that can reduce the operational complexity of maintaining your own scraping infrastructure. You can then layer on approved enrichment services to add verified business emails or phone numbers where appropriate. Before importing records into a CRM or research repository, validate and normalize the data, remove duplicates, and confirm your compliance basis, including LinkedIn’s Terms of Service and applicable privacy and marketing regulations such as GDPR, CAN-SPAM, and CASL.</p><h4><strong>Can a LinkedIn scraper extract email addresses directly?</strong></h4><p>Rarely at scale. A LinkedIn scraper primarily extracts public profile data — name, title, company, location. Direct, verified email addresses usually require a separate, approved enrichment and verification step rather than coming straight off the profile page.</p><h4><strong>Is it legal to scrape LinkedIn data for market research?</strong></h4><p>It’s legally nuanced rather than simply legal or illegal. U.S. case law (<em>hiQ Labs v. LinkedIn</em>) established that scraping publicly available data generally doesn’t violate the Computer Fraud and Abuse Act, but LinkedIn’s Terms of Service explicitly prohibit scraping regardless, and LinkedIn has actively enforced this — including a 2025 federal lawsuit against LinkedIn-data vendor Proxycurl that ended in the company’s shutdown. This isn’t legal advice; consult counsel for your specific use case.</p><h4><strong>Does using a scraper without my LinkedIn login reduce risk?</strong></h4><p>Yes, meaningfully. Tools that collect only public data via API or dataset — without your credentials — don’t put your personal LinkedIn account at risk of a ban, unlike automation that acts from your own logged-in session. The underlying Terms of Service question is separate, but the account-level and legal-exposure risk is measurably lower.</p><h4><strong>Can I use LinkedIn-scraped data for market research outreach?</strong></h4><p>Possibly, but only after reviewing applicable privacy and marketing rules for your regions. Most B2B cold outreach under GDPR relies on a documented “legitimate interest” basis rather than public availability alone, and CAN-SPAM/CASL impose their own separate requirements if US or Canadian recipients are involved.</p><h4><strong>What’s the difference between a LinkedIn dataset and live scraping?</strong></h4><p>A dataset is pre-collected and licensed in bulk, which shifts collection-time compliance exposure toward the vendor and suits historical backfill. Live scraping collects fresh data on demand but keeps more of that exposure with whoever runs the collection. Many research programs use both.</p><h4><strong>Is Bright Data suitable for this workflow?</strong></h4><p>Yes, particularly for teams that need a production-grade setup rather than an ad hoc script — Bright Data offers no-login LinkedIn scraping tools, supporting proxy infrastructure, and documented GDPR/CCPA/ISO 27001/SOC 2 compliance, which together can move a team from manual collection to a more scalable, auditable process.</p><h4>Further Reading</h4><ul><li><a href="https://brightdata.com/products/web-scraper/linkedin">Bright Data LinkedIn Scraper API</a> — supported fields, request methods, and pricing.</li><li><a href="https://docs.brightdata.com/datasets/scrapers/linkedin/introduction">Bright Data LinkedIn technical documentation</a> — setup, inputs, and outputs.</li><li><a href="https://brightdata.com/trustcenter">Bright Data Trust Center</a> — compliance certifications and data-handling practices.</li></ul><h3>Practical Takeaway</h3><p>A LinkedIn scraper can absolutely help with market research, but the highest-value output is <strong>structured professional context</strong> — not a guaranteed list of direct personal contact details. The strongest process combines no-login public profile extraction, company-domain matching, approved enrichment, validation, and an explicit compliance review that accounts for both LinkedIn’s Terms of Service and whatever outreach regulations apply to your recipients. If you need a scalable way to do that, Bright Data is worth evaluating because it combines LinkedIn-specific scraping capabilities with broader public web data infrastructure and documented compliance certifications in one platform. This article is for informational purposes and isn’t legal advice — consult counsel before scaling any workflow involving personal data.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9e5fa5ae540d" width="1" height="1" alt=""><hr><p><a href="https://javascript.plainenglish.io/how-to-extract-contact-information-from-linkedin-for-market-research-9e5fa5ae540d">How to Extract Contact Information from LinkedIn for Market Research</a> was originally published in <a href="https://javascript.plainenglish.io">JavaScript in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Open Web Is Closing. What Scraper Developers Need to Do Before It Does]]></title>
            <link>https://ai.plainenglish.io/the-open-web-is-closing-what-scraper-developers-need-to-do-before-it-does-f594fb31f4a0?source=rss-a7b125868703------2</link>
            <guid isPermaLink="false">https://medium.com/p/f594fb31f4a0</guid>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[internet]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[business]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Sunil Sandhu]]></dc:creator>
            <pubDate>Mon, 03 Aug 2026 08:38:35 GMT</pubDate>
            <atom:updated>2026-08-03T08:38:35.025Z</atom:updated>
            <content:encoded><![CDATA[<p>In this article, you will learn:</p><ul><li>Why the rise of agentic AI threatens publicly accessible web data at a structural level</li><li>How A2A communication is reducing the open-web surface area scrapers depend on</li><li>Which data categories are already moving behind agent-accessible access layers</li><li>Five concrete steps to future-proof your pipelines before access conditions tighten</li><li>How to position your data infrastructure to hold a competitive advantage as the web evolves</li></ul><p>Let’s dive in!</p><h3>The surface area problem</h3><p>The public web was designed to be read, not scraped. For two decades that distinction didn’t matter much because the browser was the access layer for humans and bots alike, and a developer patient enough to <a href="https://get.brightdata.com/bd-web-unlocker?utm_content=the_open_web_is_closing_what_scraper_developers_need_to_do_before_it_does">handle CAPTCHAs</a> could reach nearly anything.</p><p>That’s changing structurally. Not because of better bot detection, but because the web is being rebuilt around machine-to-machine communication. When AI agents retrieve data directly through APIs instead of navigating HTML, the open-web surface area that scrapers depend on starts to shrink.</p><p>This isn’t a prediction. The infrastructure is already being deployed.</p><p>Preparing your scraping infrastructure for the future? Create a new Bright Data Pay-As-You-Go account and receive 5,000 free credits every month, with no credit card required and a hard spending limit that prevents unexpected charges. It’s an easy way to test Web Unlocker API, SERP API, Scrapers, and Scraper Studio as you build more resilient web data pipelines. <a href="https://get.brightdata.com/bd7914?utm_content=the_open_web_is_closing_what_scraper_developers_need_to_do_before_it_does">Get started with Bright Data today.</a></p><p>Scrapers depend on a specific condition: the data you want is served in a format a browser can render. That condition holds when humans are the primary consumers of web content. It becomes unstable when the primary consumers are machines with direct API access.</p><p>Three categories of data are already moving behind agent-accessible layers:</p><ul><li>Search and SERP data. Search engines are serving AI-generated summaries directly in results pages, reducing the link surface area that scrapers monitor. Some SERP features render only for verified browsers, not headless clients.</li><li>E-commerce and pricing data. Major platforms restrict automated access through rate limiting, device fingerprinting, and bot scoring. As those platforms build agent-facing APIs for partners, they have less reason to maintain publicly accessible HTML pages for non-partner data consumers.</li><li>Social and community data. Platforms that built their businesses on open-web distribution are gating data access behind authentication, API quotas, and legal agreements.<a href="https://en.wikipedia.org/wiki/2023_Reddit_API_controversy"> Reddit locked down its API in 2023</a>, Twitter/X followed and LinkedIn has been litigating the question for years.</li></ul><p>None of these categories are fully closed, but the direction is consistent. Every year, the publicly accessible surface area of commercially valuable web data narrows.</p><h3>What Agentic AI Actually Changes</h3><p>Agentic AI refers to AI systems that act autonomously to complete multi-step tasks (browsing, clicking, form submission, data extraction) without human intervention. Current implementations are production-grade:</p><ul><li><a href="https://www.salesforce.com/agentforce/">Salesforce’s Agentforce platform</a> now has<a href="https://venturebeat.com/technology/while-everyone-talks-about-an-ai-bubble-salesforce-quietly-added-6-000-enterprise-customers-in-3-months/"> more than 18,500 enterprise customers</a> running agents across sales, service, and procurement workflows.</li><li>ServiceNow has<a href="https://www.servicenow.com/blogs/2025/meet-your-new-servicenow-ai-agents"> deployed AI agents across finance, HR, and supply chain operations</a>.</li><li>Klarna automated<a href="https://www.klarna.com/international/press/klarna-ai-assistant-handles-two-thirds-of-customer-service-chats-in-its-first-month/"> two-thirds of its customer service volume</a> with an AI agent, handling work equivalent to<a href="https://openai.com/index/klarna/"> 700 full-time employees</a>, with query resolution time dropping from 11 minutes to two. By 2025, Klarna reintroduced human agents for complex cases after satisfaction scores dropped on high-difficulty queries; the hybrid model is now the accurate picture.</li></ul><p>Agentic AI only cares about the data. Data providers are responding by building direct feeds for agents to consume, bypassing the browser stack entirely. Agent-to-agent (A2A) communication formalizes this: agents communicate directly with structured data sources through APIs, with no human-readable web layer in the loop.<a href="https://developers.googleblog.com/en/a2a-a-new-era-of-agent-interoperability/"> Google published an A2A specification in April 2025</a>. Anthropic’s<a href="https://www.anthropic.com/news/model-context-protocol"> Model Context Protocol (MCP)</a> is moving in the same direction.</p><p>When agents can retrieve data by talking to structured services directly, the incentive to maintain a publicly accessible HTML page for that data disappears.</p><h3>The Signals Developers Are Already Seeing</h3><p>The anxiety about the future of web scraping is no longer theoretical, as threads on r/webscraping have documented it in real time. Developers are reporting scrapers that ran cleanly for years suddenly failed without any changes to their own code. The targets changed the game by simply moving the detection layer down the stack.</p><p>The degradation shows up in production metrics.<a href="https://proxyway.com/research/web-scraping-api-report-2025"> Proxyway’s 2025 Web Scraping API Report</a>, which tested 11 commercial scraping APIs across 15 heavily protected e-commerce and search targets, found that only 4 of those 11 managed success rates above 80%. On the hardest targets, results were worse: Shein returned an average success rate of just 21.88% across all providers. These are purpose-built commercial APIs with proxy rotation, fingerprint spoofing, and CAPTCHA handling built in. Unmodified Playwright or Puppeteer instances fare significantly worse.</p><p>Unmodified<a href="https://playwright.dev/"> Playwright</a> and<a href="https://pptr.dev/"> Puppeteer</a> instances that ran cleanly against major e-commerce targets in 2023 now get blocked at the TLS handshake on the same targets today. Bot protection platforms like<a href="https://www.kasada.io/"> Kasada</a> and<a href="https://datadome.co/"> DataDome</a> fingerprint the browser runtime itself (canvas rendering, WebGL behavior, and timing signatures) before a single request fires. If your scraper is failing, the automation logic is probably fine. The browser is what gives you away.</p><p>JavaScript-heavy SPAs compound the problem. The DOM a scraper sees depends on session state, authentication history, and A/B test assignment, all of which vary per request. Two requests to the same URL from the same IP can return structurally different pages. Selectors that worked in staging break in production because the production target is serving a layout variant the developer never tested against.</p><p>Rate limiting has moved from static thresholds to behavioral pattern detection. Targets aren’t counting requests per minute anymore. They’re scoring request sequences (timing distribution, navigation patterns, referrer chains) and throttling when the sequence matches known bot signatures. A scraper that respects a 1-second delay but makes requests in perfectly uniform intervals is more detectable than one making requests at human-variable cadence.</p><p>Each mitigation a developer adds (rotating proxies, fingerprint spoofing, behavioral delay injection) extends a pipeline’s lifespan but doesn’t change the underlying dynamic. The mitigation stack compounds in maintenance cost. The window for building robust open-web pipelines is narrowing, and the cost of maintaining a fragile one keeps rising.</p><p>Looking for a more resilient approach to web data collection? New Bright Data Pay-As-You-Go accounts include 5,000 free credits every month, with no credit card required and a hard spending limit that prevents unexpected charges. Test Web Unlocker API, SERP API, Scrapers, and Scraper Studio to see how they can help improve the reliability of your scraping workflows. <a href="https://get.brightdata.com/bd7914?utm_content=the_open_web_is_closing_what_scraper_developers_need_to_do_before_it_does">Start with Bright Data today.</a></p><h3>What to Do Before the Window Closes</h3><p>The access restrictions tightening across high-value data categories are not reversible. Each step below addresses a different failure mode in how most scraping pipelines are currently built. None of them require abandoning direct web collection. They require building infrastructure that doesn’t depend on it exclusively.</p><h3>1. Shift from page scraping to dataset access where possible</h3><p>The fastest-degrading pipelines are the ones scraping individual pages in real time for data that doesn’t require real-time freshness. Product pricing checked hourly, news monitored daily, and company data updated weekly all need a reliable data feed, not necessarily a live scraper.</p><p>The failure mode is consistent: a pipeline built on direct page scraping hits a bot protection upgrade, goes dark, and the downstream system (a pricing model, a competitive intelligence dashboard, an AI training job) silently starts consuming stale data, and the scraper breaks quietly. The data quality problem surfaces later in a different system and gets attributed to the wrong cause.</p><p>For data types where freshness requirements don’t demand live collection,<a href="https://get.brightdata.com/bd-datasets?utm_content=the_open_web_is_closing_what_scraper_developers_need_to_do_before_it_does"> Bright Data’s Datasets</a> provides pre-collected, structured data across e-commerce, financial data, social media, and business intelligence. It removes the scraper maintenance burden and the access risk simultaneously.</p><p>For cases that genuinely require real-time collection,<a href="https://get.brightdata.com/bd-web-unlocker?utm_content=the_open_web_is_closing_what_scraper_developers_need_to_do_before_it_does"> Web Unlocker</a> handles fingerprinting, CAPTCHA solving, and proxy rotation at the infrastructure level. The same targets that block unmodified Playwright pass cleanly through it.</p><h3>2. Build compliance-first pipelines now, not after a legal event</h3><p>Most scraping pipelines have no data provenance documentation. The data arrives, gets processed, and flows downstream with no record of where it came from, whether the source permitted automated access, or which jurisdiction’s privacy law governs it. That’s workable until it isn’t: a cease-and-desist, a data audit, an AI training dataset that surfaces in litigation.</p><p>The legal environment is tightening on multiple fronts. The<a href="https://en.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn"> hiQ v. LinkedIn litigation</a> established that the<a href="https://www.law.cornell.edu/uscode/text/18/1030"> CFAA</a> cannot be used to block scraping of publicly accessible pages (a ruling that favored scrapers), but LinkedIn ultimately won on contract grounds, because hiQ had accepted LinkedIn’s terms of service. That outcome is the actual precedent to watch: platforms don’t need federal hacking law when their ToS creates an enforceable contract. New cases are testing those limits further. The<a href="https://artificialintelligenceact.eu/"> EU AI Act</a> imposes sourcing and documentation requirements on training data. U.S. state privacy laws add data residency requirements that vary by jurisdiction and change annually.</p><p>What compliance-first actually looks like in practice: every record in your pipeline should carry a provenance object that captures the source URL, collection timestamp, access method (direct scrape, licensed dataset, or API), the jurisdiction governing the source, whether a consent layer was present, and the downstream usage rights.</p><p>This isn’t a legal nicety. If your data feeds an AI training workflow, that documentation is already required under the EU AI Act. If it feeds a competitive intelligence product sold to enterprise customers, your buyers’ legal teams will ask for it. Building the logging infrastructure after a legal event means reconstructing it from incomplete server logs under time pressure.</p><p>If your pipeline feeds an AI training workflow, provenance documentation is already a production requirement. The practical question is whether you have it before you need it.<a href="https://get.brightdata.com/blog-web-unlocker-enables-better-fingerprinting-auto-unlocking-and-captcha-solving8276?utm_content=the_open_web_is_closing_what_scraper_developers_need_to_do_before_it_does"> Bright Data’s AI training data solutions</a> build provenance tracking, consent layer verification, and sourcing records into the data delivery rather than treating them as an audit-time reconstruction problem.</p><p>Want to evaluate Bright Data without any upfront commitment? Create a new Bright Data Pay-As-You-Go account and receive 5,000 free credits every month. With no credit card required and built-in spending protection, you can explore Bright Data’s scraping tools and infrastructure while assessing what works best for your data pipelines. <a href="https://get.brightdata.com/bd7914?utm_content=the_open_web_is_closing_what_scraper_developers_need_to_do_before_it_does">Create your Bright Data account today.</a></p><h3>3. Treat your data sources as volatile infrastructure and eliminate single points of failure</h3><p>These two failure modes are the same problem at different scales.</p><p>At the selector level: a site redesign silently invalidates every selector in a scraper without returning a single error. Requests succeed. Fields come back empty, or worse, populated with whatever the new layout put where the old layout had the data you actually wanted. By then the bad data has been in the system for days.</p><p>At the pipeline level: a single headless browser hitting a single target at volume with no fallback means that when that target tightens access (bot scoring upgrade, IP range block, authentication wall), the feed goes dark completely. No degraded mode, no partial data.</p><p>The fix is architecture, not better selectors. Version your selectors separately from your data models. Build monitoring that detects DOM structure drift and field disappearance, not just HTTP errors. Document which downstream systems depend on each data source so you know the blast radius within minutes of a breaking change.</p><p>Then map your data dependencies and mark every pipeline with no alternative access path. For each one: Is there a structured dataset that covers this data type? Is there a licensed data agreement that provides the same access at lower legal and operational risk? Is there an API that covers the data without requiring direct page access?</p><p><a href="https://get.brightdata.com/bd-products-web-scraper?utm_content=the_open_web_is_closing_what_scraper_developers_need_to_do_before_it_does">Bright Data’s Web Scraper IDE</a> has structural change alerting built into the scraper lifecycle. For teams running more than a handful of scrapers, catching a breaking change in 20 minutes versus discovering it after a production incident is the difference that justifies the tooling. For pipelines that audit as highest-risk,<a href="https://get.brightdata.com/bd-datasets?utm_content=the_open_web_is_closing_what_scraper_developers_need_to_do_before_it_does"> Bright Data’s Datasets</a> operates as a fallback layer independent of whether the target’s HTML surface stays open.</p><h3>4. Hedge against SERP dependency</h3><p><a href="https://blog.google/products/search/generative-ai-google-search-may-2024/">Google’s AI Overviews rollout in May 2024</a> broke many SERP scrapers that had run stably for years. The informational query results that competitive intelligence pipelines depended on restructured overnight: different DOM, different schema, different content in the positions that mattered. Pipelines built on CSS selectors for the classic ten-blue-links layout were scraping layout chrome instead of results.</p><p>SERP is the highest-churn scrape surface in web data. It’s high-value, heavily monitored, and Google treats layout as a product variable it changes without notice. Build redundancy before the next layout change forces the issue.</p><p><a href="https://get.brightdata.com/bd-serp-api?utm_content=the_open_web_is_closing_what_scraper_developers_need_to_do_before_it_does">Bright Data’s SERP API</a> delivers structured search result data through a stable API schema. When Google changes the organic results layout, the schema you’re consuming stays consistent. The API absorbs the structural churn.</p><h3>The access window is still open</h3><p>Most commercially valuable data on the public web is still reachable with the right infrastructure. But the direction of every trend described here is the same: higher friction, narrower surface area, more legal exposure.</p><p>The developers who take web scraping future-proofing seriously now, building across datasets, real-time collection, and compliance documentation, will hold a structural data advantage when access conditions tighten further. The ones who wait will build the same infrastructure under pressure, against harder targets, with less time.</p><p>Bright Data is built as an infrastructure layer that works regardless of which direction the web moves. If targets stay open, Web Unlocker and the Scraper IDE handle direct collection at scale. If they move behind agent-accessible APIs or licensed data agreements, Datasets provides structured access that operates independently of whether the HTML surface stays available. The question isn’t whether your current scraper works. It’s whether your data infrastructure is built for the web that’s coming, or the one that’s leaving.</p><p>The future of web data collection depends on building infrastructure that can adapt as access conditions evolve. Create a new Bright Data Pay-As-You-Go account and receive 5,000 free credits every month, with no credit card required and a hard spending limit that prevents unexpected charges. Whether you’re testing Web Unlocker API, SERP API, Scrapers, or Scraper Studio, you can start building resilient data pipelines at no upfront cost. <a href="https://get.brightdata.com/bd7914?utm_content=the_open_web_is_closing_what_scraper_developers_need_to_do_before_it_does">Create your Bright Data account and get started today.</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f594fb31f4a0" width="1" height="1" alt=""><hr><p><a href="https://ai.plainenglish.io/the-open-web-is-closing-what-scraper-developers-need-to-do-before-it-does-f594fb31f4a0">The Open Web Is Closing. What Scraper Developers Need to Do Before It Does</a> was originally published in <a href="https://ai.plainenglish.io">Artificial Intelligence in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Build Scrapers That Survive Anti-Bot Updates (2026)]]></title>
            <link>https://javascript.plainenglish.io/how-to-build-scrapers-that-survive-anti-bot-updates-2026-03efea0fc080?source=rss-a7b125868703------2</link>
            <guid isPermaLink="false">https://medium.com/p/03efea0fc080</guid>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[residential-proxies]]></category>
            <category><![CDATA[bright-data]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Sunil Sandhu]]></dc:creator>
            <pubDate>Sat, 01 Aug 2026 10:11:10 GMT</pubDate>
            <atom:updated>2026-08-01T10:11:10.532Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>AI-driven workflows are increasing the demand for web data in 2026. More scraping jobs are running across monitoring systems, data pipelines, and automation tasks. According to the<a href="https://www.imperva.com/resources/resource-library/reports/2025-bad-bot-report/"> Imperva Bad Bot Report</a>, nearly half of all web traffic comes from bots, which directly impacts how requests are evaluated.</p><p>As automated traffic increases, platforms now have stricter checks. You see more <a href="https://get.brightdata.com/bd-web-unlocker?utm_content=how_to_build_scrapers_that_survive_anti_bot_updates_2026">blocks, challenges, and validation layers</a>, especially when performing <a href="https://brightdata.com/blog/web-data/bypass-cloudflare?utm_content=how_to_build_scrapers_that_survive_anti_bot_updates_2026">Cloudflare web scraping</a>. So the question is simple: can you still scrape? Yes, you can access publicly available data. You cannot misuse it.</p><p>But running scrapers reliably is where most systems fail. When you run scrapers in production, failures usually happen during execution. Requests get blocked, sessions reset, and pipelines stop after small anti-bot updates.</p><p>That is why you need a setup that handles these failures by design, using retries, fallbacks, and stable request execution instead of relying on one-time scripts.</p><p>In this article, we will explore why scrapers break, what resilient web scraping infrastructure means, and how to build systems that survive anti-bot updates.</p><p><strong>Want to build resilient scrapers without any upfront commitment?</strong> Create a new <strong>Bright Data Pay-As-You-Go account</strong> and receive <strong>5,000 free credits every month</strong>, with <strong>no credit card required</strong> and a hard spending limit that prevents unexpected charges. It’s an easy way to test tools like Web Unlocker API, Scrapers, SERP API, and Scraper Studio before deploying them in production. <a href="https://get.brightdata.com/bd7914?utm_content=how_to_build_scrapers_that_survive_anti_bot_updates_2026"><strong>Get started with Bright Data today</strong></a><strong>.</strong></p><h3>What the Meta vs Bright Data Case Actually Changed</h3><p>The <a href="https://brightdata.com/blog/web-data/court-rules-in-favor-of-bright-data-in-meta-v-bright-data-case?utm_content=how_to_build_scrapers_that_survive_anti_bot_updates_2026">Meta vs Bright Data case</a> began in 2023, when Meta sued Bright Data for collecting data from Facebook and Instagram. The data in question came from public pages that anyone could access without logging in. Bright Data operated in a logged-out state and collected only publicly visible information.</p><p>Meta argued that this violated its terms of service. The court looked at how the data was accessed. Since no login was used, the access was treated the same as a normal public visitor opening the page in a browser.</p><p>The outcome favored Bright Data in the year 2024 and clarified a key point for scraping systems. Accessing public data is allowed, including through automated requests. The boundary applies to how the data is used after collection, not the access method itself. <a href="https://brightdata.com/blog/general/meta-dismisses-claim-against-bright-data?utm_content=how_to_build_scrapers_that_survive_anti_bot_updates_2026">Meta later dropped the remaining claims</a>, reinforcing this position.</p><p>From a development perspective, this removes ambiguity about access when working with public endpoints. You can design your pipeline without treating basic access as a legal constraint. What still needs attention is execution. Platforms continue to apply rate limits, fingerprint checks, and anti-bot controls. Your scraper needs to handle these conditions to run consistently.</p><h3>Scrapers Are Easy to Build. Hard to Keep Running.</h3><p>You can build a scraper quickly today. For example, you might use Playwright to extract product details from an e-commerce site. You test it locally, the HTML loads correctly, selectors work, and the data looks clean. You add it to a scheduled job, and for a few runs, everything works as expected.</p><p>After some time, the behavior starts to change. The same requests begin returning 403 responses or CAPTCHA pages. In some cases, the HTML is incomplete or different from what you saw earlier. The code has not changed, but the output has. This usually happens when the target site updates its anti-bot checks, which is common on platforms using Cloudflare.</p><p>This is where scraper maintenance starts to fail. The initial logic works, but it does not handle changes in request validation, session behavior, or detection patterns. LLMs help generate the scraping logic, but they do not account for how the scraper behaves in a live environment.</p><p>To fix this, you need to understand where the failure is happening in the request flow. In the next section, let’s understand why these blocks happen and what anti-bot systems actually check before allowing a request.</p><h3>Why Your Scraper Breaks After Every Update</h3><p>When a scraper stops working, the failure usually happens before your parsing logic even runs. The request gets evaluated at multiple levels, and small mismatches can lead to blocks. Even if your code stays the same, changes in anti-bot systems can affect how your requests are treated.</p><p>Below are a few common reasons why this happens:</p><ul><li><strong>IP reputation: </strong>The site checks where the request is coming from. Static or low-quality IPs get flagged quickly. Repeated requests from the same IP increase the risk of blocking.</li><li><strong>TLS fingerprint: </strong>Each request carries a network-level signature. If this does not match a real browser pattern, it signals automation.</li><li><strong>Browser behavior: </strong>Headless setups often miss real user signals like proper rendering, timing, or interactions. This makes the request easier to detect.</li><li><strong>Request patterns: </strong>High-frequency or uniform requests look unnatural. Lack of delays or variation can trigger rate limits and blocks.</li></ul><p>For a full breakdown of how Bright Data handles these detection layers, <a href="https://docs.brightdata.com/scraping-automation/concepts/how-bright-data-handles-blocking?utm_content=how_to_build_scrapers_that_survive_anti_bot_updates_2026">see the anti-bot bypass documentation</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BJzRtyx3v8lJEZVR.jpg" /></figure><h3>The Common Setup That Fails at Scale</h3><p>Most scraping setups look fine in local testing, but they can start breaking once you run them continuously. The issue usually isn’t your extraction logic. It comes from how the request pipeline is put together.</p><p>In many cases, you start with a single proxy or a very basic rotation. That works for a few requests, but over time, the IP gets flagged because the traffic pattern is too consistent. Along with that, a default headless browser setup sends predictable signals. These don’t fully match real user behavior, so detection systems pick them up quickly.</p><p>Retry logic often doesn’t help much either. If the same request is repeated with the same setup, it fails again. Static selectors add another layer of fragility. Even a small change in the page structure can break the extraction.</p><p>Session handling is also easy to overlook. Without maintaining cookies and headers, each request looks like a fresh session, which increases the chance of being blocked.</p><p>When these issues combine, the scraper doesn’t slow down or degrade. It just stops working, because there’s no fallback or recovery built into the pipeline.</p><h3>What Resilient Web Scraping Infrastructure Looks Like</h3><p>Now that access to public data is clearly defined, you need to focus on how to keep the scraper running over time. This is where resilient web scraping infrastructure becomes important.</p><p>It is a system designed to handle anti-bot changes, reduce failures, and improve overall web scraping reliability at scale.</p><p>Instead of relying on a single script, you can work with a layered pipeline where each part handles a specific responsibility:</p><ul><li>The proxy layer manages <a href="https://brightdata.com/faqs/ip-rotation/why-it-is-important?utm_content=how_to_build_scrapers_that_survive_anti_bot_updates_2026">IP rotation and distribution</a>, so requests do not come from a single source and trigger easy blocks.</li><li>The browser execution layer runs requests in a real browser environment, which helps handle dynamic content and stricter validation checks.</li><li>Fingerprint consistency ensures that network and browser signals match expected user behavior, reducing detection.</li><li>Retries and fallback logic allow the system to recover from failed requests, instead of stopping after a single error.</li><li>Session handling maintains cookies and headers across requests, so the interaction looks continuous and trusted.</li><li>Flexible extraction handles small DOM changes without breaking the entire scraper, which improves stability.</li></ul><p>You can understand this better with the flow diagram below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gMYtd8-7ebyTWkCb.jpg" /></figure><p><strong>Ready to test resilient web scraping infrastructure for yourself?</strong> New <strong>Bright Data Pay-As-You-Go accounts</strong> receive <strong>5,000 free credits every month</strong>, with <strong>no credit card required</strong> and built-in spending protection. Explore Web Unlocker API, Scrapers, SERP API, and Scraper Studio to see how they help your scrapers stay reliable as anti-bot systems evolve. <a href="https://get.brightdata.com/bd7914?utm_content=how_to_build_scrapers_that_survive_anti_bot_updates_2026"><strong>Start with Bright Data today</strong></a><strong>.</strong></p><h3>Build for Reliability: Retries, Fallbacks, and Sessions</h3><p>Once your scraper runs as a pipeline, failures are expected. Requests time out, get rate-limited, or return challenge pages. Reliability comes from how you handle these cases so the pipeline keeps moving.</p><h3>Retries with Backoff</h3><p>A retry should not repeat the same request immediately. That usually triggers another block. Use exponential backoff so each retry waits longer than the previous one. Also, retry only when it makes sense.</p><p>Example:</p><ul><li>A request returns 429 (rate limit) → wait and retry</li><li>A request times out → retry with delay</li><li>A request returns 403 (blocked) → retry only after changing IP or strategy</li></ul><p>This avoids sending identical requests that keep failing.</p><h3>Fallback Strategy</h3><p>Not every request needs the same execution path. Start with a lightweight request and move to heavier options only if needed.</p><p>HTTP → headless browser → full browser → unlocker</p><p>Example:</p><ul><li>Try a simple HTTP request</li><li>If content is missing (JS-rendered), switch to a browser</li><li>If blocked, escalate to a more advanced execution or bypass layer</li></ul><p>This keeps your pipeline efficient while still handling protected pages.</p><h3>Session Consistency</h3><p>Websites expect continuity. If every request looks new, it raises suspicion. Maintain cookies, headers, and session state across requests.</p><p>Example:</p><ul><li>First request sets cookies</li><li>Next requests reuse the same cookies and headers</li><li>Navigation appears consistent, not random</li></ul><p>This makes your resilient web scraping infrastructure behave more like a real session and improves the success rate.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RsN4-Y74oNbM33hK.jpg" /></figure><h3>Compliance Checklist for Web Scraping at Scale</h3><p>Once the legal aspects are clear, the next step is to translate them into how your scraper actually runs. At scale, compliance depends on how requests are sent, how sessions are maintained, and how the system reacts to blocks.</p><p>A few practical rules to follow:</p><ul><li><strong>Access only publicly available data: </strong>Target pages that do not require login or authentication. If the content is visible in a logged-out browser, it is a valid input source.</li><li><strong>Avoid authenticated or restricted endpoints: </strong>Do not interact with private APIs, user accounts, or paywalled content. This keeps the access layer clean and reduces risk.</li><li><strong>Control request rate and distribution: </strong>Use rate limiting, delays, and IP rotation to avoid burst traffic and repeated patterns that trigger blocking.</li><li><strong>Maintain session continuity: </strong>Reuse cookies and headers across requests so the interaction looks consistent instead of restarting on every call.</li><li><strong>Handle blocks and challenges correctly: </strong>Detect 403, 429, and CAPTCHA responses, and adjust the request strategy instead of retrying the same request.</li><li><strong>Log and monitor request behavior: </strong>Track failure rates, response changes, and block patterns to identify issues early in the pipeline.</li></ul><p>Applying these rules consistently across multiple pipelines is difficult. Each scraper needs coordinated handling for rate limits, sessions, retries, and anti-bot checks.</p><p>This is where dedicated scraping infrastructure helps, by enforcing these patterns at the request level instead of relying on manual fixes.</p><h3>Reduce Manual Work with Resilient Web Scraping Infrastructure</h3><p>A more practical approach to reduce your effort is to separate the extraction logic from the access layer. Instead of managing IP rotation, browser behavior, and anti-bot handling within your code, these can be handled by dedicated infrastructure.</p><p>Bright Data provides this as a set of components for your resilient web scraping infrastructure:</p><ul><li><a href="https://get.brightdata.com/bd-proxy-types?utm_content=how_to_build_scrapers_that_survive_anti_bot_updates_2026"><strong>Proxy Networks</strong></a> handle IP rotation, geo-distribution, and request routing, which helps reduce blocks at the network level.</li><li><a href="https://get.brightdata.com/bd-scraping-browser?utm_content=how_to_build_scrapers_that_survive_anti_bot_updates_2026"><strong>Scraping Browser</strong></a> runs requests in a real browser environment, managing JavaScript execution, fingerprint consistency, and session behavior.</li><li><a href="https://get.brightdata.com/bd-web-unlocker?utm_content=how_to_build_scrapers_that_survive_anti_bot_updates_2026"><strong>Web Unlocker</strong></a> handles anti-bot bypass by adapting requests, managing retries, and resolving challenges like CAPTCHAs when needed.</li><li><a href="https://get.brightdata.com/bd-products-web-scraper?utm_content=how_to_build_scrapers_that_survive_anti_bot_updates_2026"><strong>Web Scraper API</strong></a><strong> </strong>combines these layers into a single endpoint, so you don’t manage proxies, browser configuration, or retry logic in your code.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GI-3NIUs3YSVKD2d.jpg" /></figure><h3>Build for Scale and Stay Compliant</h3><p>You can access publicly available data. The boundary comes in how that data is used. The Meta vs Bright Data case clarified this distinction and made the approach to public data scraping at scale clearer.</p><p>What limits most scraping systems today is not access, but reliability. Blocks, detection updates, and ongoing maintenance reduce consistency across pipelines.</p><p>The practical approach is to focus on infrastructure. You should separate extraction from access, reduce manual fixes, and design systems that handle retries, sessions, and anti-bot changes by default.</p><p><strong>Ready to build scraping pipelines that can adapt to changing anti-bot systems?</strong> Create a new <strong>Bright Data Pay-As-You-Go account</strong> and receive <strong>5,000 free credits every month</strong>, with <strong>no credit card required</strong> and a hard spending limit that prevents unexpected charges. It’s a simple way to evaluate Web Unlocker API, Scrapers, SERP API, and Scraper Studio while building resilient web scraping infrastructure for production. <a href="https://get.brightdata.com/bd7914?utm_content=how_to_build_scrapers_that_survive_anti_bot_updates_2026"><strong>Create your Bright Data account and start building today.</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=03efea0fc080" width="1" height="1" alt=""><hr><p><a href="https://javascript.plainenglish.io/how-to-build-scrapers-that-survive-anti-bot-updates-2026-03efea0fc080">How to Build Scrapers That Survive Anti-Bot Updates (2026)</a> was originally published in <a href="https://javascript.plainenglish.io">JavaScript in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Is Web Scraping Legal? What Scraper Developers Need to Know]]></title>
            <link>https://blog.stackademic.com/is-web-scraping-legal-what-scraper-developers-need-to-know-dd66df895728?source=rss-a7b125868703------2</link>
            <guid isPermaLink="false">https://medium.com/p/dd66df895728</guid>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Sunil Sandhu]]></dc:creator>
            <pubDate>Thu, 30 Jul 2026 08:35:20 GMT</pubDate>
            <atom:updated>2026-07-30T08:35:20.475Z</atom:updated>
            <content:encoded><![CDATA[<p>Your legal team keeps warning you that your scraping pipeline is risky because the terms of service are unclear or there might be copyright problems. You know the data is public and that courts have previously supported the use of scrapers. Still, your project is stalled, waiting for approval that never comes.</p><p>Asking if web scraping is legal misses the real issue. It makes it seem like collecting public data is either legal or illegal. It is not that simple. What matters is what data you collect, <a href="https://get.brightdata.com/bd-ethical?utm_content=is_web_scraping_legal_what_scraper_developers_need_to_know">how you handle it</a> and what you build with it afterward.</p><p>Scraping publicly accessible data is legally defensible, and courts have repeatedly confirmed this. The legal risk lives in what you do with the data after it arrives.</p><p>In 2026, scraping at scale has never carried more legal weight. As AI adoption increases, teams are scraping more data, and platforms are responding with stricter rules and more lawsuits. Instead of just adding a disclaimer, you need to know exactly where the legal limits are and what happens if you cross them.</p><p><strong>Want to test a scraping workflow without any upfront commitment?</strong> New Bright Data <strong>Pay-As-You-Go accounts</strong> include <strong>5,000 free credits every month</strong>, with <strong>no credit card required</strong> and a hard spending limit that prevents unexpected charges. It’s an easy way to test tools like Web Unlocker API, SERP API, Scrapers, and Scraper Studio before scaling your projects. <a href="https://get.brightdata.com/bd7914?utm_content=is_web_scraping_legal_what_scraper_developers_need_to_know"><strong>Get started with Bright Data today</strong></a><strong>.</strong></p><h3>The Legal Baseline for Scraping Public Data</h3><p>To understand your legal risk, start with the basics. Scraping publicly accessible data sits on solid legal ground. US courts have confirmed this many times. If you can view the data in an incognito browser without logging in, it is considered public data.</p><p>If your scraping process needs a login, the legal situation changes. You are no longer just looking at public information. As a logged-in user, you have to follow the platform’s terms of service.</p><p>The Computer Fraud and Abuse Act (CFAA) is the main federal anti-hacking law in the US. In the past, it was often used against scrapers, turning contract issues into federal crimes. It changed in April 2022 with the Ninth Circuit’s decision in <em>hiQ Labs v. LinkedIn Corp.</em> The court confirmed that the CFAA’s “without authorization” rule does not apply to data that is publicly available. If a website shows content to anyone without a password, scraping that content does not break the CFAA.</p><p>This decision made the rule clear. The CFAA is not designed to stop companies from scraping public data. Its main goal is to protect systems from unauthorized access. If you scrape public pages, you are staying within the law’s basic limits.</p><p>If you scrape public data without logging in, you have basic legal protection. But if you scrape data that requires a login, you lose that protection and must follow the platform’s rules as part of the contract.​</p><h3>The Court Cases That Set the Rules</h3><p>Two court cases set the rules every scraper developer needs to know. Both are practical precedents you can point to when your legal team questions a scraping project.</p><h4><a href="https://cdn.ca9.uscourts.gov/datastore/opinions/2022/04/18/17-16783.pdf">hiQ Labs v. LinkedIn (2022)</a></h4><p>The Ninth Circuit confirmed that scraping public data does not violate the CFAA. The court considered whether “without authorization” could apply to data that anyone can view without logging in. The answer was no.</p><h4><a href="https://brightdata.com/blog/web-data/court-rules-in-favor-of-bright-data-in-meta-v-bright-data-case?utm_content=is_web_scraping_legal_what_scraper_developers_need_to_know">Meta v. Bright Data (2024)</a></h4><p>Judge Edward M. Chen ruled in favor of Bright Data, making it clear that Facebook and Instagram’s terms of service do not stop people from scraping public data while logged out. The court also said that these terms do not prohibit the sale of public data. This decision matters because it clearly separates data you access as a visitor from data you access as a logged-in user.</p><h3>What both cases establish together</h3><p>Together, these rulings make it clear to developers that scraping public data without logging in does not violate the CFAA or platform terms of service. You are simply accessing information that the site owner chose to make public.</p><p>​<strong>Want to validate your scraping workflow with minimal risk?</strong> Bright Data lets new Pay-As-You-Go users start with <strong>5,000 free credits every month</strong>, so you can test public web data collection without providing a credit card. With built-in spending protection, you can experiment confidently while evaluating the platform for production use. <a href="https://get.brightdata.com/bd7914?utm_content=is_web_scraping_legal_what_scraper_developers_need_to_know"><strong>Start building with Bright Data today.</strong></a></p><h3>Where Legal Scraping Becomes Liability</h3><h4>PII and GDPR</h4><p>Public visibility does not waive privacy rights. Under GDPR, a name, email address, or profile photo on a public page remains personal data. The <a href="https://www.cnil.fr/en/legal-basis-legitimate-interests-focus-sheet-measures-implement-case-data-collection-web-scraping">CNIL</a> provided specific guidance on this: scraping publicly accessible data can be lawful on the basis of legitimate interest, but only with strict safeguards. You must practice data minimization, exclude sensitive categories, respect opt-out signals like robots.txt and provide transparency to data subjects. The liability comes from retaining more data than your documented purpose requires or using it in ways that violate the rights of the individuals involved.</p><h4>Copyright</h4><p>Public does not mean unlicensed. Scraping content from a public site does not grant you the rights to reproduce, redistribute, or use that content in derivative products. The <a href="https://www.oecd.org/content/dam/oecd/en/publications/reports/2025/02/intellectual-property-issues-in-artificial-intelligence-trained-on-scraped-data_a07f010b/d5241a23-en.pdf">OECD’s February 2025</a> report on intellectual property in AI training highlighted that unlicensed use of scraped content often breaches copyright protections, even when that content appears freely available. The risk zone is using scraped text or images to train models or build competing products without a license. The exposure comes from what you build with it afterward, not the collection itself.​</p><h4>Terms of Service and Breach of Contract</h4><p>Courts have significantly narrowed the CFAA’s reach, but contracts remain fully enforceable. While the Ninth Circuit ruled in <a href="https://www.zwillgen.com/alternative-data/hiq-v-linkedin-wrapped-up-web-scraping-lessons-learned/"><em>hiQ v. LinkedIn</em></a> that scraping publicly available data does not violate the CFAA, LinkedIn successfully argued a breach of contract. When you create an account to access a site, you sign a contract. If you scrape while logged in, you are bound by those terms. Authenticating trades your publicly accessible defense for a binding agreement. You can be held to it.​</p><h4>Jurisdiction</h4><p>The law does not apply uniformly across jurisdictions. US precedent protects you within the Ninth Circuit, but it does not grant you global immunity. GDPR applies across the EU and to any company processing data of EU residents, regardless of where your servers are located. Assuming that a favorable ruling in a US court protects your global operations is a mistake with real consequences. You must map your compliance requirements to where your target data resides and where you physically conduct your business.​</p><h3>A Compliance Checklist for Scraper Developers</h3><p>The <a href="https://docs.brightdata.com/scraping-automation/concepts/ethical-web-scraping?utm_content=is_web_scraping_legal_what_scraper_developers_need_to_know">checklist</a> below turns these legal points into clear steps for engineers.</p><ol><li><strong>Stay in the incognito zone:</strong> If you need to log in to see the data, find another source. Accessing data available only to logged-in users requires you to comply with the platform’s terms. Only scrape data you can see in an incognito browser window. This is the clearest rule anywhere.</li><li><strong>Respect robots.txt:</strong> Treat the robots.txt file as a clear sign that a site owner wants to limit access. Regulators like the CNIL view this file as a primary means of showing restrictions. Ignoring it can lead to claims that you acted improperly.</li><li><strong>Use rate limiting:</strong> Scraping too aggressively and slowing down a site can get you into legal trouble, even if you are not breaking data laws. If your requests overload a server and hurt its performance, you could be sued. Set up strict rate limits that mimic real users.</li><li><strong>Collect only what you need: </strong>Only gather the data needed for your specific purpose. This is a key GDPR rule and a good habit for any project. For example, if you just need a user’s ID and public handle, don’t collect their whole profile history. The less data you collect, the lower your risk.</li><li><strong>Keep track of your legal reasons:</strong> If you handle personal data, know your reason for each piece. GDPR and similar laws require you to show your legitimate interest or consent for every data point. Keep a simple record that links each type of data you collect to its business use.</li><li><strong>Don’t create accounts: </strong>If you make an account to access restricted content, you agree to the platform’s rules. You lose your “publicly accessible” defense and are now bound by a contract. Only scrape as an anonymous visitor.</li><li><strong>Check how you use the data: </strong>The biggest legal risk is usually not collecting the data, but what you do with it later. Before using scraped data in an AI model, analytics tool or public app, review your use of it. Make sure you are not creating new copyright or privacy problems.</li></ol><h3>Why Bright Data’s Infrastructure Fits the Compliance Framework</h3><p>When <a href="https://brightdata.com/blog/web-data/court-rules-in-favor-of-bright-data-in-meta-v-bright-data-case?utm_content=is_web_scraping_legal_what_scraper_developers_need_to_know">Meta demanded that Bright Data</a> stop allowing its customers to collect public data from Facebook and Instagram, Bright Data refused and took the matter to court. Judge Edward M. Chen ruled in Bright Data’s favor. The ruling confirmed that scraping public data after logging off does not violate the platform’s terms of service.</p><p><a href="https://get.brightdata.com/bd-products-web-scraper?utm_content=is_web_scraping_legal_what_scraper_developers_need_to_know">Bright Data’s infrastructure</a> is built to operate within the parameters established by the courts, including logged-off access, residential IP rotation and session management that does not require platform authentication. Bright Data built its infrastructure to handle the technical complexities of scraping so your team can focus on the data.</p><h3>The Bottom Line</h3><p>Web scraping itself is not illegal. Scraping publicly available data is permitted by law, and <a href="https://www.quinnemanuel.com/the-firm/news-events/client-alert-meta-v-bright-data-significant-decision-for-web-scraping-industry/">courts have repeatedly confirmed</a> this.</p><p>The legal risk comes later. It depends on what you collect, how long you keep it and how you use it. Teams that stick to public data, respect opt-out signals, collect only what they need and check how they use the data are following the law.</p><p>Bright Data built its infrastructure to support compliant public web data collection while handling the technical challenges of large-scale scraping. <strong>New Bright Data Pay-As-You-Go accounts receive 5,000 free credits every month</strong>, with <strong>no credit card required</strong> and a hard spending limit that prevents unexpected charges. Whether you’re validating a new scraping project or evaluating production-ready tools, you can get started with Web Unlocker API, SERP API, Scrapers, and Scraper Studio at no upfront cost. <a href="https://get.brightdata.com/bd7914?utm_content=is_web_scraping_legal_what_scraper_developers_need_to_know"><strong>Create your Bright Data account and start building today.</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dd66df895728" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/is-web-scraping-legal-what-scraper-developers-need-to-know-dd66df895728">Is Web Scraping Legal? What Scraper Developers Need to Know</a> was originally published in <a href="https://blog.stackademic.com">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Biggest Mistakes I Made Building Startups — by Sunil Sandhu]]></title>
            <link>https://sunilsandhu.medium.com/the-biggest-mistakes-i-made-building-startups-by-sunil-sandhu-d5487e783ed3?source=rss-a7b125868703------2</link>
            <guid isPermaLink="false">https://medium.com/p/d5487e783ed3</guid>
            <category><![CDATA[entrepreneur]]></category>
            <category><![CDATA[business]]></category>
            <category><![CDATA[entrepreneurship]]></category>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[solopreneur]]></category>
            <dc:creator><![CDATA[Sunil Sandhu]]></dc:creator>
            <pubDate>Fri, 03 Jul 2026 13:17:27 GMT</pubDate>
            <atom:updated>2026-07-03T13:17:27.936Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Sunil Sandhu, founder of In Plain English, shares the biggest mistakes he made building startups — from ignoring distribution to chasing perfection — and what he’d tell his younger self.</em></p><p><strong>TL;DR:</strong> The biggest mistakes Sunil Sandhu made building startups were treating distribution as an afterthought, chasing perfection over momentum, building before validating, doing too much alone, and underestimating how long things take. Here’s what each one cost — and what he does differently now.</p><p>I’ve built a few companies now — In Plain English, Stackademic, Circuit, Differ, and most recently Obsurfable. I’ve also made most of the mistakes you can make along the way. This is an honest list, in the spirit of Messy Founder: momentum over perfection, and real stories over highlight reels.</p><h3><strong>What are the biggest mistakes founders make?</strong></h3><p>In my experience, the most damaging startup mistakes aren’t technical. They’re about distribution, timing, ego, and patience. Here are mine.</p><h4><strong>Mistake 1: Treating distribution as an afterthought</strong></h4><p>Early on, I believed that if I built something good enough, people would find it. They don’t. The best product in the world dies quietly if nobody knows it exists. I learned (slowly) that distribution is not a phase that comes after building; it’s a discipline you plan from day one. Ironically, the thing that saved me was content: writing that reached people became my distribution. That lesson eventually became an entire company in Circuit.</p><h4><strong>Mistake 2: Chasing perfection instead of momentum</strong></h4><p>I’ve lost weeks polishing things nobody asked for. Perfectionism feels like diligence, but it’s usually fear wearing a nicer outfit. The projects that worked were the ones I shipped before I felt ready. This is the whole ethos behind Messy Founder. Momentum beats perfection, almost every time.</p><h4><strong>Mistake 3: Building before validating</strong></h4><p>More than once I built the thing first and asked “does anyone want this?” second. That’s the wrong order. Now I try to find the demand before I write the code, publish the idea, talk to people, see if anyone leans in. It’s cheaper to be wrong in a conversation than in a codebase.</p><h4><strong>Mistake 4: Doing too much alone</strong></h4><p>With Stackademic I did everything. I built the platform, wrote the curriculum, taught the students, marked the homework. I’m proud of it, but it wasn’t sustainable, and it capped how far it could go. Trying to be every function in the company isn’t heroic; it’s a bottleneck. Knowing when to bring in other people is a skill I had to learn the hard way.</p><h4><strong>Mistake 5: Underestimating how long everything takes</strong></h4><p>Audiences, trust, SEO, brand — the things that actually compound are frustratingly slow at the start. I used to interpret “slow” as “failing” and kill things too early. Some of my best outcomes came from projects that looked flat for a long time before they took off. Patience is a strategy.</p><h3><strong>What I’d tell my younger self</strong></h3><ul><li>Plan distribution before you build.</li><li>Ship embarrassing, then improve.</li><li>Validate demand in conversations, not code.</li><li>Don’t be the whole company forever.</li><li>Give the compounding things time to compound.</li></ul><p>None of this makes building easy. But knowing the common traps means you spend your energy on the hard parts that actually matter, instead of the self-inflicted ones.</p><h3><strong>FAQ</strong></h3><p><strong>What is the biggest mistake founders make according to Sunil Sandhu?</strong></p><p>Treating distribution as an afterthought. Sunil argues distribution should be planned from day one, not bolted on after the product is built.</p><p><strong>What does Sunil Sandhu mean by “momentum over perfection”?</strong></p><p>That shipping and maintaining momentum matters more than polishing something to perfection before anyone sees it. It’s a core idea behind his community, Messy Founder.</p><p><strong>What would Sunil Sandhu tell a first-time founder?</strong></p><p>Validate demand before building, plan distribution early, don’t try to do everything alone, and be patient with the things that compound slowly.</p><p><strong><em>About the author</em></strong><em>: Sunil Sandhu is a founder and entrepreneur based in Barcelona. He founded </em><a href="https://plainenglish.io"><em>In Plain English</em></a><em>, one of the world’s largest developer education platforms, and is currently building </em><a href="https://obsurfable.com"><em>Obsurfable</em></a><em>, which helps brands see how they appear in AI answers. Read more at </em><a href="https://sunilsandhu.com"><em>sunilsandhu.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d5487e783ed3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Obsurfable’s free AI Visibility Checker is way more fun than it should be]]></title>
            <link>https://javascript.plainenglish.io/obsurfables-free-ai-visibility-checker-is-way-more-fun-than-it-should-be-228a823dddf3?source=rss-a7b125868703------2</link>
            <guid isPermaLink="false">https://medium.com/p/228a823dddf3</guid>
            <category><![CDATA[marketing]]></category>
            <category><![CDATA[seo]]></category>
            <category><![CDATA[saas]]></category>
            <dc:creator><![CDATA[Sunil Sandhu]]></dc:creator>
            <pubDate>Thu, 25 Jun 2026 08:09:29 GMT</pubDate>
            <atom:updated>2026-06-25T08:09:29.619Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-5RN6C-0gNyNHUWwRDb70A.png" /></figure><p>The way people discover companies is changing. More buyers are skipping traditional search and asking AI assistants questions like:</p><ul><li>“What is the best CRM for a startup?”</li><li>“Which project management tools work well for remote teams?”</li><li>“What software should I use for this?”</li></ul><p>The brands that show up in those answers get considered. The ones that do not might never make it into the conversation.</p><p>That is why Obsurfable has launched a free <a href="https://obsurfable.com/ai-visibility-checker">AI Visibility Checker</a>, helping companies understand how visible they are inside AI-generated answers.</p><p>The tool simulates how ChatGPT might respond to buyer-style questions about your business using its existing knowledge, rather than live web browsing.</p><p>Simply add your company details, including your website and description, and Obsurfable will analyze how your brand is likely to be represented in AI responses.</p><p>Because ranking on Google does not automatically mean showing up in AI. A company can have great SEO but still be invisible to AI systems if they do not have enough information about the brand.</p><h3>Why AI Visibility Matters</h3><p>Search engines helped people discover websites. AI assistants are helping people discover answers.</p><p>This is where Answer Engine Optimization (AEO) and Generative Engine Optimization (GEO) come in. Businesses are starting to think beyond “How do I rank?” and asking “How do I get mentioned?”</p><h3>Who Is It For?</h3><p>The AI Visibility Checker is useful for founders, SaaS companies, marketers, SEO professionals, and agencies that want to understand their position in the AI discovery space.</p><p>It can help answer questions like:</p><p>“Would ChatGPT recommend my company?”</p><p>“How does my visibility compare to competitors?”</p><p>“What does AI already know about my brand?”</p><p>AI analysis takes a moment, so they added a little surprise.</p><p>While you wait for your results, you can play a hidden game inside the checker. Because staring at a loading screen is not exactly the future of technology.</p><h3>The Future of Brand Discovery</h3><p>AI is becoming a bigger part of how people research products, compare options, and make decisions.</p><p>Obsurfable helps companies monitor how they appear in AI answers, track competitor positioning, and understand changes in AI-generated recommendations over time.</p><p>If you are curious whether AI knows your brand exists, start with the free AI Visibility Checker.</p><p>Try it here: <a href="https://obsurfable.com/ai-visibility-checker">https://obsurfable.com/ai-visibility-checker</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=228a823dddf3" width="1" height="1" alt=""><hr><p><a href="https://javascript.plainenglish.io/obsurfables-free-ai-visibility-checker-is-way-more-fun-than-it-should-be-228a823dddf3">Obsurfable’s free AI Visibility Checker is way more fun than it should be</a> was originally published in <a href="https://javascript.plainenglish.io">JavaScript in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Your Scraper Keeps Failing in Production (Hint: It’s Your IPs)]]></title>
            <link>https://python.plainenglish.io/why-your-scraper-keeps-failing-in-production-hint-its-your-ips-01566e3511e7?source=rss-a7b125868703------2</link>
            <guid isPermaLink="false">https://medium.com/p/01566e3511e7</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Sunil Sandhu]]></dc:creator>
            <pubDate>Mon, 11 May 2026 10:45:59 GMT</pubDate>
            <atom:updated>2026-07-24T07:41:03.737Z</atom:updated>
            <content:encoded><![CDATA[<p>In this article, you will learn:</p><ul><li>Why production scrapers fail even when the automation code is correct</li><li>How bot-protection platforms detect and block datacenter IPs before your logic runs</li><li>The practical difference between datacenter and residential proxies at scale</li><li>How to integrate <a href="https://get.brightdata.com/bd-residential-proxies?utm_content=why_your_scraper_keeps_failing_in_production_hint_its_your_ips">Bright Data’s residential network</a> into any existing scraper framework</li><li>When to move beyond proxies to a fully managed unblocking API</li></ul><p>Let’s dive in!</p><h3>Why Scrapers Break in Production</h3><p>You’ve already built the scraper, complete with Playwright script, clean selectors and tested it locally. Everything works, till you ship it to production. Three days later, 40% of requests are coming back empty, so you spend a weekend adjusting wait times, swapping Chromium for Firefox, rewriting the element detection logic.</p><p>Most developers respond by debugging the wrong layer, but nothing helps, because the framework was never the problem. Your IP stack is.<strong> </strong>The Playwright script and selectors are fine, but the scraper is failing because datacenter IPs get flagged at the network level before your automation logic ever runs, and switching frameworks will not change that.</p><p>Before teams invest in fixing their IP stack, the first question they usually ask is whether scraping is even allowed. In January 2024, a federal court<a href="https://techcrunch.com/2024/01/24/court-rules-in-favor-of-a-web-scraper-bright-data-which-meta-had-used-and-then-sued/"> ruled in Bright Data’s favor against Meta</a>, affirming that accessing publicly available web data is a lawful activity. Meta dropped the remaining claims the following month rather than appeal.</p><p>The distinction that matters across every region and use case is this: accessing public data is generally legal, misusing that data is not. Most production scraper failures are not a legal problem or a code problem, they are an infrastructure problem with your IPs.</p><p>To understand these failures better, let’s look at what anti-bot systems actually detect.</p><h3>What Anti-Bot Systems Actually Detect</h3><p>Modern bot-protection platforms (<a href="https://www.cloudflare.com/application-services/products/bot-management/">Cloudflare</a>,<a href="https://datadome.co"> DataDome</a>,<a href="https://www.kasada.io"> Kasada</a>,<a href="https://www.humansecurity.com/products/human-bot-defender"> PerimeterX</a>) do not read your automation code. They evaluate the request before the page loads, in milliseconds. Four signals drive most blocking decisions:</p><ul><li><strong>IP reputation.</strong> Is this address associated with a known data center provider?</li><li><strong>ASN classification.</strong> Does the autonomous system number belong to AWS, Google Cloud, or a commercial proxy vendor?</li><li><strong>Request velocity and pattern.</strong> Are hundreds of requests coming from the same /24 subnet?</li><li><strong>TLS fingerprint.</strong> Does the handshake match a real browser on a real ISP connection?</li></ul><p>Datacenter IPs fail the first two checks immediately. Amazon, DigitalOcean, and Hetzner publish their IP ranges. Every major bot-protection vendor maintains continuously updated blocklists of those ranges. By the time your scraper tries to render a page, the decision to block has already been made.</p><p>What makes this easy to miss is how the failure appears in your logs. A blocked datacenter IP typically does not return a 403. It returns a CAPTCHA page, a redirect, or a 200 response with garbage in the body. Your scraper reports success, but the content is empty. That failure signature looks exactly like a selector problem or a timing issue, so developers fix the selector and the timing and stay blocked.</p><p>This is why switching from Selenium to Playwright does not fix the problem. The framework is irrelevant when the IP is flagged at the edge.</p><p>This plays out constantly in developer communities.<a href="https://www.reddit.com/r/automation/comments/1ncuv8k/best_web_scraping_tools_ive_tried_and_what_i/"> A recent thread</a> comparing<a href="https://scrapy.org/"> Scrapy</a>,<a href="https://www.crummy.com/software/BeautifulSoup/"> BeautifulSoup</a>, and Requests-HTML runs for dozens of comments without anyone mentioning proxies. The framework debate is a more comfortable problem to have, because it is a problem developers know how to approach. IP infrastructure feels like a vendor problem, something external and opaque, so it gets deprioritized until everything else has been ruled out.</p><p>Once you accept that the IP is the problem, the next question is obvious: what kind of IP actually works?</p><h3>Datacenter Proxies vs. Residential Proxies: What Changes</h3><p>Choosing a proxy network is the first architectural decision that determines whether the rest of your scraping infrastructure matters at all.</p><p>Datacenter proxies route requests through commercial server infrastructure. They are fast and cheap, and for sites without serious bot protection they work reliably. For anything running Cloudflare Business or enterprise-tier DataDome, they fail consistently.</p><p>Residential proxies route requests through real ISP-assigned addresses on consumer devices. To the target site, the traffic is indistinguishable from an ordinary user browsing from home. The ASN belongs to Comcast, BT, or Vodafone, the IP has legitimate browsing history, and the TLS fingerprint matches a real OS and browser version. For a deeper breakdown of how the two proxy types compare across speed, cost, and detection rates, see Bright Data’s<a href="https://get.brightdata.com/gm-differences-between-datacenter-and-residential-proxies?utm_content=why_your_scraper_keeps_failing_in_production_hint_its_your_ips"> datacenter vs. residential proxy guide</a>.</p><p>The performance gap on protected targets is not marginal. On sites running enterprise-tier bot management, datacenter proxies routinely drop below 50% success rates while residential networks hold above 90% on the same targets.<a href="https://proxyway.com/research/proxy-market-research-2025"> Proxyway’s 2025 Proxy Market Research</a> confirms the pattern: real-world targets reflect the synthetic benchmarks, and the gap widens sharply once Cloudflare or Akamai bot management enters the picture. That gap is what determines whether your pipeline is reliable.</p><p>The reason residential proxies perform that way comes back to how anti-bot systems score requests. A datacenter ASN is one of the clearest signals that traffic is non-human, and residential IPs pass that check because they come from the same ISPs, subnets, and device types as actual users.</p><p>This is also where the legal foundation we talked about earlier matters in practice. The<a href="https://techcrunch.com/2024/01/24/court-rules-in-favor-of-a-web-scraper-bright-data-which-meta-had-used-and-then-sued/"> Meta v. Bright Data ruling</a> established that collecting publicly available data at scale is legally defensible. The distinction that holds across jurisdictions is the same one that holds technically: access public data lawfully, handle what you collect responsibly. Bright Data operates on the right side of that line, with an ethically sourced network built entirely on explicitly opted-in residential peers.</p><h3>How Bright Data’s Residential Network Is Different</h3><p>Bright Data operates the largest ethically sourced residential proxy network available, with more than 150 million IPs across 195 countries. Every peer in the network has explicitly opted in, and no personal data is collected from them.</p><p>Here is what that means in practice for your scraper:</p><ul><li><strong>Framework-agnostic integration.</strong> Bright Data’s<a href="https://get.brightdata.com/bd-residential-proxies?utm_content=why_your_scraper_keeps_failing_in_production_hint_its_your_ips"> residential proxies</a> work as a drop-in with<a href="https://playwright.dev/"> Playwright</a>,<a href="https://pptr.dev/"> Puppeteer</a>,<a href="https://www.selenium.dev/"> Selenium</a>,<a href="https://scrapy.org/"> Scrapy</a>, Requests, and any HTTP client that supports proxy configuration. You do not rewrite your scraper. You change the proxy endpoint.</li><li><strong>Automatic IP rotation.</strong> Each request can rotate to a fresh residential IP, eliminating the pattern-detection problem that flags repeated requests from a fixed address.</li><li><strong>Geo-targeting.</strong> Route requests through specific countries, cities, or carriers to pull geo-restricted content or match the expected location of a real user on a given site.</li><li><strong>Pay for success.</strong> With the<a href="https://get.brightdata.com/bd-web-unlocker?utm_content=why_your_scraper_keeps_failing_in_production_hint_its_your_ips"> Web Unlocker API</a>, you are charged only for successful requests. Failed attempts (CAPTCHAs, blocks, empty responses) do not count.</li></ul><blockquote>🚀 <strong>Run your own diagnostic, free:</strong> Before you dig into the ASN checklist below, you can test Web Unlocker against your actual target with 5,000 free monthly credits — no payment method required, so there’s no cost to ruling this layer in or out.</blockquote><blockquote><a href="https://get.brightdata.com/bd-web-unlocker?utm_content=why_your_scraper_keeps_failing_in_production_hint_its_your_ips">Start free</a>.</blockquote><p>Here is what a Playwright integration looks like with Bright Data’s proxy:</p><pre>from playwright.sync_api import sync_playwright<br>PROXY = {<br> &quot;server&quot;: &quot;http://brd.superproxy.io:33335&quot;,<br> &quot;username&quot;: &quot;brd-customer-YOUR_ID-zone-residential&quot;,<br> &quot;password&quot;: &quot;YOUR_PASSWORD&quot;<br>}<br>with sync_playwright() as p:<br> browser = p.chromium.launch(proxy=PROXY)<br> context = browser.new_context(ignore_https_errors=True)<br> page = context.new_page()<br> page.goto(&quot;https://target-site.com&quot;)<br> page.wait_for_load_state(&quot;networkidle&quot;)<br> print(page.content())<br> browser.close()</pre><p>For most pipelines, that is enough. But some sites layer defenses beyond what IP reputation alone determines.</p><h3>When to Move Beyond Proxies to a Full Web Scraper API</h3><p>Residential proxies solve the IP-blocking problem, but some sites have moved further. Platforms running<a href="https://datadome.co"> DataDome</a> or<a href="https://www.kasada.io"> Kasada</a> do not just evaluate where a request comes from. They evaluate things like mouse movement patterns, scroll velocity, time between interactions, and the consistency of those signals across a session are all evaluated in real time. A residential IP with robotic interaction patterns still fails all those checks.</p><p>For this class of target, a managed web scraper API handles the full stack. <a href="https://get.brightdata.com/bd-web-unlocker?utm_content=why_your_scraper_keeps_failing_in_production_hint_its_your_ips">Bright Data’s Web Unlocker API</a> manages proxy selection, CAPTCHA solving, browser fingerprinting, and behavioral simulation automatically. You send a URL and receive clean HTML or Markdown.</p><pre>import requests<br>response = requests.post(<br> &quot;https://api.brightdata.com/request&quot;,<br> json={<br> &quot;url&quot;: &quot;https://target-site.com&quot;,<br> &quot;zone&quot;: &quot;YOUR_ZONE_NAME&quot;,<br> &quot;format&quot;: &quot;raw&quot;<br> },<br> headers={<br> &quot;Content-Type&quot;: &quot;application/json&quot;,<br> &quot;Authorization&quot;: &quot;Bearer YOUR_API_TOKEN&quot;<br> }<br>)<br>if response.status_code == 200:<br> print(response.text)<br>else:<br> print(f&quot;Error {response.status_code}: {response.text}&quot;)</pre><p>With the web scraper API, you are charged only for successful requests. Failed attempts, CAPTCHAs, blocks, and empty responses do not count. This is the right architecture for pipelines where a failed scrape has a downstream cost: broken ML training runs, stale pricing data, missed competitive signals.</p><p>If you are not sure which layer is failing, here is how to find out fast.</p><h3>The Production Checklist</h3><p>Before touching any automation code, run through this:</p><ul><li><strong>Check your proxy’s ASN before anything else.</strong> Use<a href="https://ipinfo.io"> ipinfo.io</a> to confirm whether your proxy resolves to a datacenter ASN (AWS, DigitalOcean, Hetzner). If it does, the rest of the checklist is irrelevant until you switch to<a href="https://brightdata.com/proxy-types/residential-proxies"> </a><a href="https://get.brightdata.com/bd-residential-proxies?utm_content=why_your_scraper_keeps_failing_in_production_hint_its_your_ips">residential</a>. The<a href="https://techcrunch.com/2024/01/24/court-rules-in-favor-of-a-web-scraper-bright-data-which-meta-had-used-and-then-sued/"> Meta v. Bright Data ruling</a> already confirmed that collecting public data at scale is lawful — the only question is whether your IP stack can actually reach it.</li><li><strong>Isolate the IP from the automation.</strong> Send a raw HTTP request to the target using the same proxy, without any browser automation. If that request succeeds, the browser fingerprint is the issue. If it fails, the IP is flagged.</li><li><strong>Read what the 200 response actually contains.</strong> Grep for CAPTCHA keywords, bot-check indicators, or empty body length in your response logs. A silent block is a 200 with nothing useful in it.</li><li><strong>Escalate to a web scraper API for behavioral targets.</strong> Sites running<a href="https://datadome.co"> DataDome</a> or<a href="https://www.kasada.io"> Kasada</a> score mouse movement, scroll behavior, and interaction timing.<a href="https://get.brightdata.com/bd-web-unlocker?utm_content=why_your_scraper_keeps_failing_in_production_hint_its_your_ips"> Bright Data’s Web Unlocker</a> handles that at the infrastructure level. Scraping behind authenticated sessions or collecting personal data raises separate legal questions, but if your pipeline targets public pages, you are on solid ground.</li></ul><blockquote>💡 <strong>Test the escalation path free:</strong> New Bright Data accounts get 5,000 free Web Unlocker credits every month (~$7.50 value) — enough to see how it handles a DataDome/Kasada-protected target before you commit. No card, hard stop when credits run out.</blockquote><blockquote><a href="https://get.brightdata.com/bd-web-unlocker?utm_content=why_your_scraper_keeps_failing_in_production_hint_its_your_ips"><strong>Try Web Unlocker free</strong></a>.</blockquote><h3>Stop Debugging the Wrong Layer</h3><p>Most scraper failures in production have already been solved. The solution is not a better framework, a smarter selector strategy, or a more carefully tuned wait time. It’s an IP that looks like a real user, because to a bot-protection system, that is the only thing that matters.</p><p>Developers lose days(sometimes weeks) to this misdiagnosis because the tooling they know best is the tooling they reach for first. The fix sits one layer below the code, in infrastructure that can be swapped in an afternoon without touching a single line of automation logic.</p><p>Every hour your pipeline runs on flagged datacenter IPs is an hour of incomplete data feeding downstream decisions. <a href="https://get.brightdata.com/bd7914?utm_content=why_your_scraper_keeps_failing_in_production_hint_its_your_ips">Start your free trial</a> and run your existing scraper against Bright Data’s residential network. One test will tell you more than another weekend of framework debugging ever could.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=01566e3511e7" width="1" height="1" alt=""><hr><p><a href="https://python.plainenglish.io/why-your-scraper-keeps-failing-in-production-hint-its-your-ips-01566e3511e7">Why Your Scraper Keeps Failing in Production (Hint: It’s Your IPs)</a> was originally published in <a href="https://python.plainenglish.io">Python in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>