<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Arun Kumar on Medium]]></title>
        <description><![CDATA[Stories by Arun Kumar on Medium]]></description>
        <link>https://medium.com/@cr-arunkumar201?source=rss-003f48653228------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*zSaw1Qf1jHSguHpy</url>
            <title>Stories by Arun Kumar on Medium</title>
            <link>https://medium.com/@cr-arunkumar201?source=rss-003f48653228------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 02:10:31 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@cr-arunkumar201/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com" rel="hub"/>
        <item>
            <title><![CDATA[Understanding Rate Limiting with a Custom Deque in TypeScript]]></title>
            <link>https://javascript.plainenglish.io/understanding-rate-limiting-with-a-custom-deque-in-typescript-d8c13d94bb20?source=rss-003f48653228------2</link>
            <guid isPermaLink="false">https://medium.com/p/d8c13d94bb20</guid>
            <category><![CDATA[typescript]]></category>
            <category><![CDATA[rate-limiter]]></category>
            <category><![CDATA[dequeue]]></category>
            <category><![CDATA[datastrucutre]]></category>
            <category><![CDATA[algorithms]]></category>
            <dc:creator><![CDATA[Arun Kumar]]></dc:creator>
            <pubDate>Thu, 09 Oct 2025 12:24:08 GMT</pubDate>
            <atom:updated>2025-10-14T10:30:51.827Z</atom:updated>
            <content:encoded><![CDATA[<p>Introduction</p><p>Rate limiting is a crucial technique in modern software development that controls the frequency of operations or requests. Whether you’re building an API, managing resource-intensive tasks, or implementing throttling mechanisms, understanding how to build a rate limiter from scratch is an invaluable skill.</p><p>In this comprehensive guide, we’ll explore how to implement a rate limiter using a custom double-ended queue (Deque) in TypeScript. This implementation demonstrates fundamental data structure concepts while solving a practical real-world problem.</p><p>What is a Deque?</p><p>A Deque (Double-Ended Queue) is a versatile data structure that allows insertion and deletion of elements from both ends. Unlike a standard queue (FIFO) or stack (LIFO), a deque provides flexibility to add or remove elements from either the front or the back.</p><p>Key operations:</p><p>• addFirst() — Add element to the front<br>• addLast() — Add element to the back<br>• removeFirst() — Remove element from the front<br>• removeLast() — Remove element from the back<br>• peekFirst() — View front element without removing<br>• peekLast() — View back element without removing</p><p>Implementing the Deque</p><p>Let’s start by building our custom Deque class:</p><blockquote>class Deque&lt;T&gt; {<br> private items: T[];</blockquote><blockquote>constructor() {<br> this.items = [];<br> }</blockquote><blockquote>addFirst(element: T): void {<br> this.items.unshift(element);<br> }</blockquote><blockquote>addLast(element: T): void {<br> this.items.push(element);<br> }</blockquote><blockquote>removeFirst(): T | undefined {<br> return this.items.shift();<br> }</blockquote><blockquote>removeLast(): T | undefined {<br> return this.items.pop();<br> }</blockquote><blockquote>peekFirst(): T | undefined {<br> return this.items[0];<br> }</blockquote><blockquote>peekLast(): T | undefined {<br> return this.items[this.items.length — 1];<br> }</blockquote><blockquote>size(): number {<br> return this.items.length;<br> }</blockquote><blockquote>isEmpty(): boolean {<br> return this.items.length === 0;<br> }</blockquote><blockquote>clear(): void {<br> this.items = [];<br> }<br>}</blockquote><p>This implementation uses TypeScript generics (&lt;T&gt;) to make our Deque type-safe and reusable with any data type. The underlying storage mechanism is a simple array, but the Deque interface provides semantic meaning to our operations.</p><p>Building the Rate Limiter</p><p>Now let’s implement our rate limiter using the Deque. The core concept is the sliding window algorithm:</p><p>1. Maintain a queue of timestamps representing when requests were made<br>2. Before processing a new request, remove all timestamps older than our time window<br>3. Check if the number of remaining timestamps is below our limit<br>4. If allowed, add the current timestamp and process the request</p><p>class RateLimiter {<br> private requestTimestamps: Deque&lt;number&gt;;<br> private maxRequests: number;<br> private timeWindowMs: number;</p><p>constructor(maxRequests: number, timeWindowMs: number) {<br> this.requestTimestamps = new Deque&lt;number&gt;();<br> this.maxRequests = maxRequests;<br> this.timeWindowMs = timeWindowMs;<br> }</p><p>allowRequest(): boolean {<br> const currentTime = Date.now();<br> <br> while (<br> !this.requestTimestamps.isEmpty() &amp;&amp;<br> currentTime — this.requestTimestamps.peekFirst()! &gt; this.timeWindowMs<br> ) {<br> this.requestTimestamps.removeFirst();<br> }</p><p>if (this.requestTimestamps.size() &lt; this.maxRequests) {<br> this.requestTimestamps.addLast(currentTime);<br> return true;<br> }</p><p>return false;<br> }</p><p>getCurrentRequestCount(): number {<br> const currentTime = Date.now();<br> <br> while (<br> !this.requestTimestamps.isEmpty() &amp;&amp;<br> currentTime — this.requestTimestamps.peekFirst()! &gt; this.timeWindowMs<br> ) {<br> this.requestTimestamps.removeFirst();<br> }</p><p>return this.requestTimestamps.size();<br> }</p><p>reset(): void {<br> this.requestTimestamps.clear();<br> }<br>}</p><p>How the Rate Limiter Works</p><p>The sliding window approach offers several advantages:</p><p>• Precise control: Unlike fixed window approaches, the sliding window accurately tracks requests within any time period, not just discrete intervals.</p><p>• Memory efficient: Old timestamps are automatically removed, preventing unbounded memory growth.</p><p>• Simple logic: The implementation is straightforward and easy to understand.</p><p>The key insight is using peekFirst() to check the oldest timestamp without removing it, then conditionally calling removeFirst() to eliminate outdated entries. This pattern efficiently maintains our sliding window.</p><p>Practical Usage Example</p><p>Here’s how you might use this rate limiter in a real application:</p><p>const rateLimiter = new RateLimiter(5, 10000);</p><p>function processRequest(requestId: number): void {<br> if (rateLimiter.allowRequest()) {<br> console.log(`Request ${requestId}: ALLOWED`);<br> } else {<br> console.log(`Request ${requestId}: DENIED — Rate limit exceeded`);<br> }<br>}</p><p>for (let i = 1; i &lt;= 7; i++) {<br> processRequest(i);<br> await new Promise(resolve =&gt; setTimeout(resolve, 500));<br>}</p><p>console.log(`Current requests: ${rateLimiter.getCurrentRequestCount()}`);</p><p>Performance Considerations</p><p>Time Complexity:</p><p>• allowRequest(): O(n) worst case, where n is the number of expired timestamps to remove. In practice, this is typically O(1) amortized.<br>• getCurrentRequestCount(): O(n) worst case, O(1) amortized<br>• Other operations: O(1)</p><p>Space Complexity: O(k) where k is the maximum number of requests allowed in the time window.</p><p>For high-performance scenarios, consider:</p><p>• Using a circular buffer instead of array-based deque<br>• Implementing lazy cleanup to batch timestamp removal<br>• Adding periodic cleanup in low-traffic periods</p><p>Real-World Applications</p><p>1. API Rate Limiting: Protect your API endpoints from abuse by limiting requests per user or IP address.</p><p>2. Resource Throttling: Control access to expensive operations like database queries or external API calls.</p><p>3. User Experience: Prevent users from spamming actions like form submissions or button clicks.</p><p>4. DOS Protection: Mitigate denial-of-service attacks by limiting request rates.</p><p>5. Fair Resource Distribution: Ensure equitable access to shared resources among multiple users or services.</p><p>Extensions and Improvements</p><p>This basic implementation can be extended in several ways:</p><p>• Multi-tier rate limiting: Implement different limits for different time windows (e.g., 100/minute, 1000/hour)</p><p>• Token bucket algorithm: Add burst capacity while maintaining average rate limits</p><p>• Distributed rate limiting: Use Redis or similar for rate limiting across multiple servers</p><p>• User-specific limits: Maintain separate rate limiters per user/API key</p><p>• Graceful degradation: Implement priority queues for important requests during high load</p><p>Conclusion</p><p>Building a rate limiter from scratch using a custom Deque demonstrates several important concepts:</p><p>• Data structure design and implementation<br>• The sliding window algorithm<br>• Time-based state management<br>• Practical application of abstract data types</p><p>The combination of a Deque and timestamp tracking provides an elegant solution to rate limiting that’s both efficient and easy to understand. While production systems might use more sophisticated approaches or existing libraries, understanding these fundamentals helps you make informed decisions about rate limiting strategies.</p><p>Whether you’re building microservices, APIs, or user-facing applications, rate limiting is a critical tool for maintaining system stability, fairness, and security. The patterns demonstrated here form a solid foundation for implementing robust rate limiting in your TypeScript applications.</p><p>Feel free to adapt this implementation to your specific needs, and remember that rate limiting is just one part of a comprehensive strategy for building resilient, scalable systems.</p><p>Happy coding!</p><h3>A message from our Founder</h3><p><strong>Hey, </strong><a href="https://linkedin.com/in/sunilsandhu"><strong>Sunil</strong></a><strong> here.</strong> I wanted to take a moment to thank you for reading until the end and for being a part of this community.</p><p>Did you know that our team run these publications as a volunteer effort to over 3.5m monthly readers? <strong>We don’t receive any funding, we do this to support the community. ❤️</strong></p><p>If you want to show some love, please take a moment to <strong>follow me on </strong><a href="https://linkedin.com/in/sunilsandhu"><strong>LinkedIn</strong></a><strong>, </strong><a href="https://tiktok.com/@messyfounder"><strong>TikTok</strong></a>, <a href="https://instagram.com/sunilsandhu"><strong>Instagram</strong></a>. You can also subscribe to our <a href="https://newsletter.plainenglish.io/"><strong>weekly newsletter</strong></a>.</p><p>And before you go, don’t forget to <strong>clap</strong> and <strong>follow</strong> the writer️!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d8c13d94bb20" width="1" height="1" alt=""><hr><p><a href="https://javascript.plainenglish.io/understanding-rate-limiting-with-a-custom-deque-in-typescript-d8c13d94bb20">Understanding Rate Limiting with a Custom Deque in TypeScript</a> was originally published in <a href="https://javascript.plainenglish.io">JavaScript in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Engineer Skills 2025 — Complete Guide to Responsibilities, Tools & Future Trends]]></title>
            <link>https://medium.com/@cr-arunkumar201/data-engineer-skills-2025-complete-guide-to-responsibilities-tools-future-trends-3a8d1e0cacf9?source=rss-003f48653228------2</link>
            <guid isPermaLink="false">https://medium.com/p/3a8d1e0cacf9</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Arun Kumar]]></dc:creator>
            <pubDate>Tue, 02 Sep 2025 12:40:50 GMT</pubDate>
            <atom:updated>2025-09-15T09:09:12.683Z</atom:updated>
            <content:encoded><![CDATA[<h3>Data Engineer Skills 2025 — Complete Guide to Responsibilities, Tools, Future Trends &amp; AI-First Era</h3><p>Explore the must-have data engineer skills: programming, big data, cloud platforms, ETL, and pipelines. Learn key responsibilities and future trends in data engineering.</p><h3>Introduction to Data Engineering</h3><p>Data engineering is a rapidly growing field focused on building and maintaining the infrastructure required for <strong>data processing and analysis</strong>. It involves designing, constructing, and managing scalable systems for collecting, storing, and transforming large volumes of data.</p><p>A strong foundation in this field requires a comprehensive <strong>data engineer skill set</strong> that blends programming, data systems knowledge, and soft skills.</p><h3>Key Responsibilities of a Data Engineer</h3><p>Data engineers wear many hats across the <strong>data lifecycle</strong>. Their responsibilities go beyond just moving data — they ensure it’s reliable, usable, and secure.</p><h3>🔹 Data Ingestion and Extraction</h3><ul><li>Designing pipelines to collect data from APIs, databases, and cloud storage.</li><li>Tools: <a href="https://kafka.apache.org/">Apache Kafka</a>, Apache Flume, AWS Kinesis.</li></ul><h3>🔹 Data Storage and Management</h3><ul><li>Choosing storage solutions for structured (SQL) and unstructured (NoSQL, data lakes) data.</li><li>Tools: PostgreSQL, MySQL, MongoDB, <a href="https://www.snowflake.com/">Snowflake</a>.</li></ul><h3>🔹 Data Transformation</h3><ul><li>Converting raw data into usable formats via ETL/ELT processes.</li><li>Skills: SQL, Python (Pandas), Scala.</li></ul><h3>🔹 Data Pipelines</h3><ul><li>Automating workflows for moving data between systems.</li><li>Tools: <a href="https://airflow.apache.org/">Apache Airflow</a>, Prefect, Dagster.</li></ul><h3>🔹 Data Monitoring and Maintenance</h3><ul><li>Ensuring <strong>data quality, observability, and availability</strong> with dashboards and alerts.</li><li>Tools: Grafana, Prometheus, CloudWatch.</li></ul><h3>🔹 Cloud Technologies</h3><ul><li>Deploying data systems on AWS, Azure, or GCP.</li><li>Skills: Containerization (Docker, Kubernetes), cloud-native data services.</li></ul><h3>Essential Skills for a Data Engineer</h3><p>The <strong>data engineer skills set</strong> can be divided into <strong>technical skills</strong> and <strong>soft skills</strong>.</p><h3>🛠️ Technical Skills</h3><p>Category Skills / Tools <strong>Programming Languages</strong> Python, Java, Scala, SQL <strong>Databases</strong> SQL (PostgreSQL, MySQL), NoSQL (MongoDB, Cassandra, Redis) <strong>Big Data Technologies</strong> Hadoop, Apache Spark, Hive <strong>Cloud Platforms</strong> AWS, Azure, GCP <strong>Data Warehousing</strong> Snowflake, BigQuery, Redshift <strong>ETL Tools</strong> Apache NiFi, Kafka, Informatica <strong>Workflow Orchestration</strong> Airflow, Prefect, Dagster</p><h3>🤝 Soft Skills</h3><ul><li><strong>Problem-Solving</strong>: Breaking down complex data challenges.</li><li><strong>Communication</strong>: Explaining technical concepts clearly to non-technical teams.</li><li><strong>Collaboration</strong>: Partnering with analysts, data scientists, and product teams.</li><li><strong>Adaptability</strong>: Staying updated with fast-changing data tools and trends.</li></ul><h3>Data Architecture &amp; Warehousing</h3><p>A modern data engineer must understand how to <strong>design scalable data architectures</strong>.</p><h3>🔹 Key Principles</h3><ul><li><strong>Scalability</strong>: Handle exponential growth in data.</li><li><strong>Performance</strong>: Optimize for fast queries and processing.</li><li><strong>Security</strong>: Implement encryption, access control, and compliance.</li><li><strong>Maintainability</strong>: Keep systems modular and easy to update.</li></ul><h3>🔹 Data Warehousing Solutions</h3><p>Tool Description Type <a href="https://aws.amazon.com/redshift/">AWS Redshift</a> Managed, scalable data warehouse Cloud <a href="https://cloud.google.com/bigquery">Google BigQuery</a> Serverless, analytics-optimized warehouse Cloud <a href="https://www.snowflake.com/">Snowflake</a> High-performance multi-cloud warehouse Cloud Oracle Database Enterprise RDBMS with warehousing features On-Prem/Cloud Teradata Advanced analytical data warehouse On-Prem/Cloud</p><h3>Big Data &amp; NoSQL Technologies</h3><p>Modern data engineers also work with <strong>big data frameworks and NoSQL databases</strong>.</p><h3>🔹 Apache Spark</h3><ul><li>Fast, distributed engine for batch + real-time data.</li><li>APIs for Python, Java, Scala, R.</li></ul><h3>🔹 Hadoop Ecosystem</h3><ul><li><strong>HDFS</strong> (storage), <strong>YARN</strong> (resource manager), <strong>Hive</strong> (SQL querying).</li></ul><h3>🔹 NoSQL Databases</h3><p>Type Description Examples <strong>Document</strong> JSON-like document storage MongoDB, Couchbase <strong>Key-Value</strong> Fast lookups with key-value pairs Redis, DynamoDB <strong>Column-Family</strong> Scalable column storage Cassandra, HBase <strong>Graph</strong> Nodes &amp; edges for relationships Neo4j, Amazon Neptune</p><h3>Cloud Platforms in Data Engineering</h3><p>Cloud computing is at the heart of modern data engineering.</p><ul><li><strong>AWS</strong>: S3 (data lakes), Redshift, Glue (ETL), Kinesis (streaming).</li><li><strong>Azure</strong>: Synapse Analytics, Databricks, Data Lake Gen2.</li><li><strong>Google Cloud</strong>: BigQuery, Dataflow, Dataproc, Pub/Sub.</li></ul><h3>Data Pipelines: Design &amp; Best Practices</h3><h3>Common Architectures</h3><ul><li><strong>Batch Processing</strong>: Efficient for large datasets at intervals.</li><li><strong>Real-time Processing</strong>: For fraud detection, streaming analytics.</li><li><strong>Lambda Architecture</strong>: Combines batch + streaming.</li></ul><h3>Key Tools</h3><p>Tool Type Strengths Weaknesses Kafka Streaming High throughput, fault-tolerance Complex setup Spark Batch + Streaming Versatile, powerful Resource-heavy Airflow Orchestration Flexible, monitors workflows Overkill for small jobs AWS Glue Serverless ETL Easy to scale, cost-effective Less customizable</p><h3>Future Trends in Data Engineering 🚀</h3><ul><li><strong>Cloud-native pipelines</strong> (serverless, Kubernetes).</li><li><strong>Data observability &amp; governance</strong> for reliability.</li><li><strong>Machine Learning pipelines</strong> (MLOps, feature stores).</li><li><strong>Real-time analytics</strong> with Kafka, Flink, Spark Streaming.</li><li><strong>Continuous learning</strong>: Certifications, open-source contributions.</li></ul><h3>Final Thoughts</h3><p>The demand for skilled <strong>data engineers</strong> continues to grow rapidly. Success in this field requires a mix of <strong>programming expertise, database skills, big data knowledge, and cloud proficiency</strong>, alongside strong problem-solving and collaboration abilities.</p><p>With <strong>real-time analytics, AI, and ML pipelines</strong> becoming standard, data engineers are no longer just pipeline builders — they are <strong>key enablers of data-driven innovation</strong>.</p><h3>❓ What does a data engineer do daily?</h3><p>A data engineer’s daily tasks include designing data pipelines, ingesting and transforming raw data, managing databases, monitoring pipeline health, and collaborating with analysts and data scientists to deliver reliable datasets.</p><h3>❓ Which programming language is best for data engineering?</h3><ul><li><strong>Python</strong>: Most popular for ETL, data wrangling, and automation.</li><li><strong>SQL</strong>: Essential for querying and transforming data.</li><li><strong>Scala/Java</strong>: Commonly used with big data frameworks like Apache Spark.</li></ul><h3>❓ Is data engineering harder than data science?</h3><p>They’re different:</p><ul><li><strong>Data Engineers</strong> focus on infrastructure, pipelines, and scalability.</li><li><strong>Data Scientists</strong> focus on analysis, modeling, and predictions.</li><li>Neither is “harder” universally, but data engineering requires deep <strong>system design and scalability expertise</strong>.</li></ul><h3>❓ What tools should a beginner learn first?</h3><ul><li><strong>SQL</strong> for querying.</li><li><strong>Python</strong> for scripting.</li><li><strong>Airflow</strong> for workflow orchestration.</li><li>A <strong>cloud platform</strong> (AWS, Azure, or GCP).</li></ul><h3>❓ What is the future of data engineering?</h3><p>The field is evolving toward:</p><ul><li><strong>Cloud-native data pipelines</strong> (serverless, containerised).</li><li><strong>Data observability &amp; governance</strong> for reliability.</li><li><strong>AI-driven automation</strong> in ETL/ELT workflows.</li><li><strong>Real-time analytics</strong> as a default requirement.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3a8d1e0cacf9" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>