<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Vishnu Sivan on Medium]]></title>
        <description><![CDATA[Stories by Vishnu Sivan on Medium]]></description>
        <link>https://medium.com/@codemaker2016?source=rss-b7a60e472dfc------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*dj_jwDIShpJxw6-T</url>
            <title>Stories by Vishnu Sivan on Medium</title>
            <link>https://medium.com/@codemaker2016?source=rss-b7a60e472dfc------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 02:42:09 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@codemaker2016/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com" rel="hub"/>
        <item>
            <title><![CDATA[PageIndex: Vectorless RAG That Reasons Instead of Searches]]></title>
            <link>https://codemaker2016.medium.com/pageindex-vectorless-rag-that-reasons-instead-of-searches-de7ecd95a0a2?source=rss-b7a60e472dfc------2</link>
            <guid isPermaLink="false">https://medium.com/p/de7ecd95a0a2</guid>
            <category><![CDATA[retrieval-augmented-gen]]></category>
            <category><![CDATA[pageindex]]></category>
            <category><![CDATA[python-for-beginners]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Vishnu Sivan]]></dc:creator>
            <pubDate>Mon, 12 Oct 2026 09:19:52 GMT</pubDate>
            <atom:updated>2026-10-12T09:25:38.196Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uNSnIbwR79KWp8YeT_KV3A.png" /></figure><p>What happens when you replace embeddings, chunking and vector databases with a table of contents and an LLM that knows how to read it.</p><p>Traditional RAG systems rely on chunking, embeddings, and vector databases to retrieve relevant information. But with long, structured documents like annual reports and legal contracts, this approach can struggle with fragmented tables, repeated keywords, and cross-references that lose their context.</p><p>PageIndex, an open-source framework from VectifyAI, takes a different approach: instead of relying on vector similarity, it uses a document’s table of contents and an LLM to navigate its structure and identify relevant sections. This makes retrieval more like how humans search through complex documents.</p><p>In this article, we will explore how PageIndex works, build a practical implementation, compare it with a traditional vector-based RAG pipeline, and examine where this approach excels — and where conventional RAG may still be the better choice.</p><h3>Getting Started</h3><h4>Table of contents</h4><ul><li><a href="#7a36">What is PageIndex</a></li><li><a href="#340c">Architecture at a glance</a></li><li><a href="#b329">PageIndex vs. traditional RAG</a></li><li><a href="#384b">When should you use PageIndex?</a></li><li><a href="#7287">Experimenting with PageIndex</a></li><li><a href="#c271">Initialize a project with uv</a></li><li><a href="#1183">Install dependencies</a></li><li><a href="#7535">Create the environment file</a></li><li><a href="#b7e6">Initialize PageIndex client</a></li><li><a href="#1dd8">Ingesting a document and building the PageIndex tree</a></li><li><a href="#523d">Ask a query</a></li><li><a href="#c5a2">Hands-On: Financial Report Analyser</a></li><li><a href="#4fea">Querying PageIndex with LLM Tree Search</a></li><li><a href="#de61">Generating Answers from Retrieved Context</a></li><li><a href="#9064">Building the Vector RAG Baseline</a></li><li><a href="#2ec4">Putting It All Together — The main() Function</a></li><li><a href="#3a8f">Complete script</a></li><li><a href="#be26">Streamlit interface</a></li></ul><h4>What is PageIndex</h4><p>PageIndex is an open-source, reasoning-based Retrieval-Augmented Generation (RAG) framework developed by VectifyAI. Instead of using embeddings, chunking, and vector databases to retrieve information, it creates a hierarchical tree index of a document, similar to a table of contents, and uses an LLM to navigate the tree and identify relevant sections.</p><p>PageIndex is a vectorless, reasoning-based RAG engine. Its docs describe two steps: index each document into a tree, then retrieve by letting an LLM search that tree agentically. <br>The project took its inspiration from AlphaGo: search a structured space with reasoning instead of scanning it blindly.</p><p>The core idea is simple:</p><ul><li><strong>Vector RAG</strong> finds text that looks similar to your query.</li><li><strong>PageIndex</strong> finds sections that are likely to contain the answer.</li></ul><p>That is the difference between scanning every page of a book for matching words and opening the table of contents to find the right chapter first.</p><p>There are no embeddings, no vector database and no chunking step. Instead, the pipeline has two phases.</p><p><strong>1. Index.</strong> PageIndex builds a hierarchical tree that mirrors the document’s natural structure: chapters, sections, subsections. Each node carries a title, a page reference, a summary and its child nodes. You choose where this runs: on your own machine (the open-source local mode, powered by the Flash engine) or in PageIndex Cloud.</p><p><strong>2. Retrieve.</strong> An LLM agent, always running on your own model and API key, reads the tree (titles and summaries, not full text), decides which sections are likely to hold the answer, reads the pages it selected, and keeps looking if it needs more. The answer is written from what it actually read, with page citations.</p><pre>Traditional RAG:  Document → Chunk → Embed → Cosine similarity → Top-K<br>PageIndex:  Document → Build tree → LLM reasons over tree → Exact sections</pre><p>PageIndex builds a tree once, then an LLM agent searches it for every question</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/821/1*pCfyocMk7pqrqvB7X9FaKA.png" /></figure><p>Indexing runs once per document. Every question then lets an agent search the saved tree, read the pages it picks, and answer from them.</p><h4>Why vector RAG struggles with structured documents</h4><p>Three failure modes show up repeatedly on financial and legal material:</p><ol><li><strong>Split content.</strong> A balance sheet divided across two chunks loses the relationships between its line items. Neither half looks relevant on its own.</li><li><strong>Repeated terms.</strong> Annual reports mention the same terms constantly. A question about one division’s growth pulls chunks from every mention, ranked almost equally.</li><li><strong>Cross-references.</strong> Similarity search cannot follow a pointer like “see Appendix G”. The answer lives somewhere the query never resembles.</li></ol><p>PageIndex sidesteps these because the document stays whole. The tree preserves hierarchy, the reasoning step can follow references or combine sections, and every node knows where it lives in the original pages.</p><h4>Architecture at a glance</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/812/1*2-Ytg2dcWVespXh9c8yN7A.png" /></figure><p>Indexing runs locally or in PageIndex Cloud; either way, the LLM that searches the tree and writes the answer. Use chat() for a ready-made document-QA agent, or hand PageIndex&#39;s tools to your own agent.</p><h4>PageIndex vs. traditional RAG</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/917/1*UbM5Ki0-DCBfPjalExLM8A.png" /></figure><h4>When should you use PageIndex?</h4><p><strong>Good fit</strong></p><ul><li>Financial reports and annual filings</li><li>Legal contracts and regulatory documents</li><li>Technical manuals and research papers</li><li>Questions requiring evidence from multiple sections of one long document</li></ul><p><strong>Consider alternatives or a hybrid approach</strong></p><ul><li>Millions of short, unrelated records that need high-throughput search</li><li>Frequently changing documents that require constant re-indexing</li><li>Simple FAQ retrieval, where conventional RAG may be sufficient</li><li>Documents whose layout or text extraction is poor</li></ul><h4>Local or cloud: where indexing runs</h4><p>PageIndex splits the work in two. Indexing can run on your own machine or in PageIndex Cloud. Retrieval always uses your own LLM, in either mode, so the only setup difference is which API keys you need.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/917/1*cvoM6VxYWkBuc2YoFI_fJw.png" /></figure><h3>Experimenting with PageIndex</h3><p>Now that we understand the fundamentals of PageIndex and how it differs from traditional RAG, it’s time to put the concepts into practice. We will start with a simple implementation and gradually explore how PageIndex uses LLM-based reasoning to navigate document structures and answer questions.</p><h4>Initialize a project with uv</h4><p>Install uv if it is not already available, then create a project:</p><pre>uv init pageindex-demo<br>cd pageindex-demo</pre><p>The command creates the project metadata and a starter Python entry point.</p><h4>Install dependencies</h4><p>Install the PageIndex SDK and python-dotenv:</p><pre>uv add pageindex python-dotenv</pre><h4>Create the environment file</h4><p>Create .env file in the project root and add your credentials to it:</p><pre># OpenAI<br>OPENAI_API_KEY=sk-proj-hdCPXqIo13xktYjbU9FSkL1VXt4...</pre><p>Get the OpenAI values from your <a href="https://platform.openai.com/settings/organization/api-keys">OpenAI portal</a> keys configuration. Add a <a href="https://developer.pageindex.ai/overview">PageIndex</a> API key only if you index in Cloud (the free credits for new accounts are enough to experiment).</p><h4>Initialize PageIndex client</h4><p>The next step is to initialize the PageIndex client. It requires two settings: index and chat, each responsible for a different task.</p><ul><li><strong>index</strong>: Specifies the model or service used to process documents and build the tree index.</li><li><strong>chat</strong>: Specifies the model used to navigate the tree, reason over relevant sections, and generate answers.</li></ul><pre>from pageindex import PageIndexClient<br><br>from dotenv import load_dotenv<br>import os<br>load_dotenv()<br>assert os.getenv(&#39;OPENAI_API_KEY&#39;), &quot;Environment variable &#39;OPENAI_API_KEY&#39; is not set&quot;<br><br># Local processing for supported text-based PDFs<br>client = PageIndexClient(<br>    index=&quot;gpt-4o&quot;,<br>    chat=&quot;gpt-4o&quot;<br>)<br><br># Cloud-based indexing for documents requiring OCR or image processing<br># client = PageIndexClient(<br>#     index=&quot;cloud&quot;,<br>#     chat=&quot;gpt-4o&quot;<br># )</pre><p>The two settings are configured separately because indexing and question answering have different requirements. The indexing model prepares the document structure, while the chat model reasons over that structure to answer questions. Separating these roles can help balance processing costs and answer quality.</p><p>You can also use models from different providers. PageIndex follows LiteLLM-style model naming conventions for supported integrations. For example, a Claude-based configuration might look like this:</p><pre>import os<br>from pageindex import PageIndexClient<br><br># Set ANTHROPIC_API_KEY in your environment beforehand<br>client = PageIndexClient(<br>    index=&quot;cloud&quot;,<br>    chat=&quot;anthropic/claude-model-name&quot;<br>)</pre><p>Replace the example model name with one currently supported by your provider, and configure the required API credentials.</p><blockquote><strong>Important:</strong> Local indexing does not necessarily mean the entire workflow runs offline. Model availability, API requirements, and supported document formats depend on your configuration. Check the current PageIndex documentation before choosing a local or cloud setup.</blockquote><p>Once the client is initialized, we’re ready to move on to processing a document and exploring its tree index.</p><h4>Ingesting a document and building the PageIndex tree</h4><p>For the hands-on, we will use the <a href="https://www.federalreserve.gov/publications/files/financial-stability-report-20260508.pdf">Federal Reserve Board’s <em>Financial Stability Report</em> from May 2026</a>. It suits reasoning-based retrieval well: a long, formal report with a clear hierarchy, numbers in tables, and main text that often points to a box instead of repeating its details.</p><p><strong>Step 1: Index a document<br></strong>Submitting a file is one call, and it returns a doc_id that every other operation uses.</p><pre>doc_id = client.submit_document(&quot;./data/fed_financial_stability_report_2026.pdf&quot;, wait=True)[&quot;doc_id&quot;]</pre><p>In local mode indexing finishes inside the call. In Cloud, processing is asynchronous: wait=True blocks until the document is ready, or you can submit many files and poll get_document(doc_id)[&quot;status&quot;], which moves through queued, processing, completed or failed. You pay the indexing cost once per document.</p><p><strong>Step 2: Inspect the tree<br></strong>The tree is plain JSON. Ask for summaries, and leave the full node text out to keep the response small.</p><pre>tree = client.get_tree(doc_id, node_summary=True, include_text=False)<br>for node in tree[&quot;result&quot;]:<br>    print(node[&quot;node_id&quot;], node[&quot;title&quot;], &quot;pages&quot;, node[&quot;start_index&quot;], &quot;-&quot;, node[&quot;end_index&quot;])</pre><pre>{<br>  &quot;doc_id&quot;: &quot;pi-84543e96b79f4d958b8c021db118298a&quot;,<br>  &quot;status&quot;: &quot;completed&quot;,<br>  &quot;retrieval_ready&quot;: true,<br>  &quot;result&quot;: [<br>    {<br>      &quot;title&quot;: &quot;Financial Stability Report&quot;,<br>      &quot;node_id&quot;: &quot;0000&quot;,<br>      &quot;start_index&quot;: 1,<br>      &quot;end_index&quot;: 3,<br>      &quot;summary&quot;: &quot;The May 2026 Financial Stability Report from the Federal Reserve outlines its five core functions: conducting monetary policy for maximum employment and stable prices, ensuring financial system stability and mitigating systemic risks, promoting the safety of individual financial institutions, enhancing payment system safety and efficiency, and advancing consumer protection and community development. The report includes sections on asset valuations, borrowing by businesses and households, financial sector leverage, funding risks, and near-term risks to the financial system. It also features updates on nonbank financial institutions and a survey of salient risks. The information reflects data available as of April 23, 2026.&quot;<br>    },<br>    {<br>      &quot;title&quot;: &quot;Contents&quot;,<br>      &quot;node_id&quot;: &quot;0001&quot;,<br>      &quot;start_index&quot;: 3,<br>      &quot;end_index&quot;: 5,<br>      &quot;summary&quot;: &quot;The document outlines the Federal Reserve Board&#39;s assessment of U.S. financial system stability, emphasizing transparency and accountability. It defines financial stability as the ability of financial institutions to provide necessary financing amid shocks. The report distinguishes between unpredictable shocks and monitorable vulnerabilities, focusing on assessing the latter across four broad categories. It also discusses asset valuations, borrowing trends, financial sector leverage, funding risks, and near-term financial system risks, with supporting boxes detailing specific developments and surveys. For further information, it directs readers to the Federal Reserve&#39;s website for monitoring efforts and resources.&quot;<br>    },<br>    {<br>      &quot;title&quot;: &quot;Purpose and Framework&quot;,<br>      &quot;node_id&quot;: &quot;0002&quot;,<br>      &quot;start_index&quot;: 5,<br>      &quot;end_index&quot;: 9,<br>      &quot;summary&quot;: &quot;The section discusses the Federal Reserve&#39;s efforts to monitor and promote the stability of the U.S. financial system, essential for achieving full employment and price stability. It details a monitoring framework that assesses various financial vulnerabilities, including valuation pressures, excessive borrowing, leverage, and funding risks, both domestically and internationally. The Federal Reserve&#39;s post-2007-09 crisis measures such as higher capital requirements, stress-testing, and the countercyclical capital buffer are highlighted as means to enhance resilience. The report identifies current vulnerabilities related to asset valuations, borrowing trends, and financial sector leverage, noting elevated pressures in real estate and corporate bonds, along with fluctuations in credit quality and liquid asset management in banks and financial entities.&quot;,<br>      &quot;nodes&quot;: [<br>        {<br>          &quot;title&quot;: &quot;More on the Federal Reserve’s Monitoring Efforts&quot;,<br>          &quot;node_id&quot;: &quot;0003&quot;,<br>          &quot;start_index&quot;: 5,<br>          &quot;end_index&quot;: 7,<br>          &quot;summary&quot;: &quot;The report outlines the Federal Reserve Board&#39;s assessment of the U.S. financial system&#39;s stability, emphasizing the importance of financial stability in achieving full employment, price stability, and a sound banking system. It describes how a stable financial system enables access to financing even during adverse events. The monitoring framework differentiates between shocks and vulnerabilities, focusing on four categories: valuation pressures, excessive borrowing, excessive leverage in the financial sector, and funding risks. The framework tracks both domestic and international risks and relies on ongoing research to address novel vulnerabilities. The Federal Reserve promotes financial resilience through supervision and regulation, emphasizing high-quality capital and stress tests following the 2007-09 crisis, and utilizes the countercyclical capital buffer to enhance bank resilience during periods of elevated risk.&quot;<br>        },<br>        {<br>          &quot;title&quot;: &quot;Federal Reserve actions to promote the resilience of the financial system&quot;,<br>          &quot;node_id&quot;: &quot;0004&quot;,<br>          &quot;start_index&quot;: 7,<br>          &quot;end_index&quot;: 9,<br>          &quot;summary&quot;: &quot;The Federal Reserve evaluates financial vulnerabilities to enhance the resilience of the financial system, collaborating with agencies through the Financial Stability Oversight Council. Post-2007–09 crisis actions include higher capital requirements, stress-testing for large banks, and the countercyclical capital buffer to sustain credit supply. The report identifies vulnerabilities related to asset valuations, business and household borrowing, financial sector leverage, and funding risks as of April 23, 2026. Asset valuations show elevated pressures in real estate, low corporate bond spreads, and recovery in Treasury liquidity. Borrowing by businesses and households shows a decline in debt-to-GDP ratios and strong credit quality, despite rising delinquencies for certain loans. Financial sector leverage remains high in hedge funds and life insurers, while banks maintain high liquid assets and reduced reliance on uninsured deposits. The report details growth in business development company redemptions and cash-management vehicle assets.&quot;<br>        }<br>      ]<br>    },<br>    ...<br>  ],<br>  &quot;metadata&quot;: null,<br>  &quot;features&quot;: {}<br>}</pre><p>This is one of PageIndex’s quiet advantages: the index is something you can read. If retrieval goes wrong, open the tree and see why. Try that with a 1,536-dimensional embedding.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/767/1*tpdZdSveNvOe-XdTIdf8fA.png" /></figure><h4>Ask a query</h4><p>Retrieval is now agentic: chat() runs PageIndex&#39;s own document-QA agent against your model. You do not write the tree search yourself.</p><pre>question = &quot;How many market contacts did the New York Fed survey this spring, and which risks did they cite most?&quot;<br>answer = client.chat(question, doc_id=doc_id)<br>print(answer)</pre><p>A few parameters you will reach for:</p><ul><li><strong>Scope:</strong> doc_id takes one ID or a list. In Cloud, folder_id scopes the question to a folder, and omitting both searches your whole library.</li><li><strong>Conversation:</strong> Pass a list of role and content messages instead of a string for multi-turn chat.</li><li><strong>Effort:</strong> reasoning_effort accepts &quot;low&quot;, &quot;medium&quot; or &quot;high&quot;, and max_turns caps how many agent turns one call can take.</li><li><strong>Protocols:</strong> protocol= returns the native format of the Chat Completions, Responses or Messages API if you want to slot PageIndex into existing code.</li></ul><h4>Citations</h4><p>Pass citations=True and the answer carries a tag after each claim, naming the document and page, as in &lt;cite doc=&quot;report.pdf&quot; page=&quot;12&quot;/&gt;. On Cloud the tag also names the exact layout block.</p><pre>question = &quot;How many market contacts did the New York Fed survey this spring, and which risks did they cite most?&quot;<br>answer = client.chat(question, doc_id=doc_id, citations=True)<br>resolved = client.resolve_citations(answer)<br>print(resolved[&quot;answer&quot;])<br><br>for c in resolved[&quot;citations&quot;]:<br>    print(c[&quot;index&quot;], c[&quot;document&quot;], c[&quot;page&quot;])</pre><p>get_citations() gives you the parsed list instead, and for Cloud documents each entry can include a bounding box. Combine that with get_page_image() and the SDK&#39;s highlight_region helper and you can draw the cited passage on the original page, which is the kind of evidence trail reviewers of contracts and filings ask for.</p><h4>Complete script</h4><p>Let’s bring all the components together into a single Python script.</p><pre># basic/app.py<br><br>from pageindex import PageIndexClient<br><br># Load environment variables<br>from dotenv import load_dotenv<br>import os<br>load_dotenv()<br>assert os.getenv(&#39;OPENAI_API_KEY&#39;), &quot;Environment variable &#39;OPENAI_API_KEY&#39; is not set&quot;<br><br># Initialize pageindex client<br>client = PageIndexClient(<br>    index=&quot;gpt-4o-mini&quot;,               # model to build the tree index<br>    chat=&quot;gpt-4o-mini&quot;,                # model to search the tree<br>)<br><br># Cloud-based indexing for documents requiring OCR or image processing<br># client = PageIndexClient(<br>#     index=&quot;cloud&quot;,<br>#     chat=&quot;gpt-4o&quot;<br># )<br><br># Index the document<br>doc_id = client.submit_document(&quot;./data/fed_financial_stability_report_2026.pdf&quot;)[&quot;doc_id&quot;]<br><br># Inspect the tree<br>tree = client.get_tree(doc_id, node_summary=True, include_text=False)<br>for node in tree[&quot;result&quot;]:<br>    print(node[&quot;node_id&quot;], node[&quot;title&quot;], &quot;pages&quot;, node[&quot;start_index&quot;], &quot;-&quot;, node[&quot;end_index&quot;])<br><br># Ask a query<br># question = &quot;How many market contacts did the New York Fed survey this spring, and which risks did they cite most?&quot;<br># answer = client.chat(question, doc_id=doc_id)<br># print(answer)<br><br>question = &quot;How many market contacts did the New York Fed survey this spring, and which risks did they cite most?&quot;<br>answer = client.chat(question, doc_id=doc_id, citations=True)<br>print(answer)<br><br># Resolve citations<br>resolved = client.resolve_citations(answer)<br>print(resolved[&quot;answer&quot;])<br><br>for c in resolved[&quot;citations&quot;]:<br>    print(c[&quot;index&quot;], c[&quot;document&quot;], c[&quot;page&quot;])</pre><h4>Run the app</h4><p>Lets execute the app using the following command to see the result.</p><pre>python app.py</pre><p>Expected output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZpsnQXCIHHg20oLxzLg7xA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DOG7qhZAittTrDAf79cr0A.png" /></figure><h3>Hands-On: Financial Report Analyser</h3><p>This tutorial walks you through building a PDF question-answering system that uses PageIndex to reason over document structure, then benchmarks it against a classic vector RAG pipeline built with FAISS and OpenAI embeddings. By the end you will have a working analyser for the Federal Reserve’s Financial Stability Report (May 2026) and a clear picture of where tree-based retrieval beats chunk-based search. The final step runs three identical queries through both pipelines and prints the answers side-by-side, so you can see the difference in quality and traceability.</p><h4><strong>Prerequisites</strong></h4><p>You need Python 3.13 or higher, an OpenAI API key, and either uv (recommended) or plain pip. The script uses gpt-4o-mini for both indexing and search by default, so make sure your key has access to that model. The PDF (~10 MB) is downloaded automatically on first run.</p><h4><strong>Install Dependencies</strong></h4><p>Install the required packages using the following command.</p><pre>pip install pageindex openai pymupdf faiss-cpu numpy python-dotenv</pre><ul><li><strong>pageindex</strong>: handles tree-based indexing and search.</li><li><strong>openai:</strong> drives the LLM calls for both tree search and answer generation.</li><li><strong>pymupdf:</strong> extracts raw page text from the PDF for the vector baseline.</li><li><strong>faiss-cpu:</strong> provides the in-memory vector index.</li><li><strong>numpy:</strong> handles the embedding arrays.</li><li><strong>python-dotenv:</strong> loads your API key from a .env file.</li></ul><h4><strong>Set Up Environment Variables</strong></h4><p>Create a .env file in the project root containing your OpenAI key:</p><pre>OPENAI_API_KEY=sk-…</pre><h4><strong>Imports and Client Initialisation</strong></h4><p>With dependencies installed, the script sets up two clients. The PageIndexClient takes two model parameters: index controls which model is used during the one-time tree-building step, and chat controls which model reasons over the tree at query time. Both are set to gpt-4o-mini here to keep costs low while still producing accurate results. The OpenAI clients handle embeddings and answer generation directly.</p><ul><li>At the top of app.py, all dependencies are imported and the environment is loaded:</li></ul><pre>import copy<br>import time<br>import json<br>import asyncio<br>import requests<br>import pymupdf<br>from openai import AsyncOpenAI, OpenAI<br>import numpy as np<br>import faiss<br>from pageindex import PageIndexClient<br>import pageindex.utils as utils<br><br>from dotenv import load_dotenv<br>import os<br>load_dotenv()<br>assert os.getenv(&#39;OPENAI_API_KEY&#39;), &quot;Environment variable &#39;OPENAI_API_KEY&#39; is not set&quot;</pre><h4><strong>Querying PageIndex with LLM Tree Search</strong></h4><ul><li><strong>Tree Search Prompt<br></strong>Immediately after the imports, the script defines two module-level prompt templates. The first, TREE_SEARCH_PROMPT, instructs the LLM to reason over the document tree and return a structured JSON object with a reasoning trace and a list of node IDs. Requiring JSON output and setting temperature=0 makes the response deterministic and easy to parse.</li></ul><pre>TREE_SEARCH_PROMPT = &quot;&quot;&quot;You are a document retrieval assistant.<br>Given a document&#39;s tree structure and a user query, identify which nodes (sections)<br>are most likely to contain the answer.<br><br>Document tree:<br>{tree_json}<br><br>User query: {query}<br><br>Return a JSON object with the following format:<br>{{<br>  &quot;reasoning&quot;: &quot;Your step-by-step reasoning about which sections to retrieve&quot;,<br>  &quot;node_ids&quot;: [&quot;id1&quot;, &quot;id2&quot;, ...]<br>}}<br><br>Return ONLY the JSON object, no other text.&quot;&quot;&quot;</pre><ul><li><strong>Tree Search Function<br></strong>It strips the full node text from the tree before sending it to the LLM — keeping only titles, IDs, and summaries — so the prompt stays within context limits even for long documents. It serialises the slim tree to JSON, formats the prompt, and returns the parsed response.</li></ul><pre>async def tree_search(tree, query: str, openai_client: AsyncOpenAI, model: str = &quot;gpt-4o-mini&quot;) -&gt; dict:<br>    slim_tree = utils.remove_fields(copy.deepcopy(tree), fields=[&quot;text&quot;])<br>    tree_json = json.dumps(slim_tree, indent=2)<br>    prompt = TREE_SEARCH_PROMPT.format(tree_json=tree_json, query=query)<br>    response = await openai_client.chat.completions.create(<br>        model=model,<br>        messages=[{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: prompt}],<br>        temperature=0,<br>        response_format={&quot;type&quot;: &quot;json_object&quot;},<br>    )<br>    return json.loads(response.choices[0].message.content)</pre><h4><strong>Generating Answers from Retrieved Context</strong></h4><ul><li><strong>Answer Prompt<br></strong>The second prompt template, ANSWER_PROMPT, tells the LLM to answer strictly from the provided context and to cite the specific sections it draws from. Because generate_answer will prefix each section with its title and page range before passing it here, the model naturally produces traceable citations.</li></ul><pre>ANSWER_PROMPT = &quot;&quot;&quot;You are a financial document analyst. Answer the user&#39;s question<br>using ONLY the provided context. Cite the specific section(s) you are drawing from.<br>If the context does not contain enough information to answer, say so clearly.<br><br>Context:<br>{context}<br><br>Question: {question}<br><br>Provide a precise, well-cited answer.&quot;&quot;&quot;</pre><ul><li><strong>Answer Generation Function<br></strong>generate_answer looks up each selected node ID in the flat node_map, prefixes each section’s text with a header showing its title and page range, and assembles the full context. This is what makes citations possible: the generation model receives context that explicitly names its sources.</li></ul><pre>async def generate_answer(node_ids: list, query: str, node_map: dict, openai_client: AsyncOpenAI, model: str = &quot;gpt-4o-mini&quot;) -&gt; str:<br>    context_parts = []<br>    for node_id in node_ids:<br>        node = node_map.get(node_id)<br>        if node:<br>            section_header = f&quot;[{node[&#39;title&#39;]} | Pages {node[&#39;start_index&#39;]}-{node[&#39;end_index&#39;]}]&quot;<br>            context_parts.append(f&quot;{section_header}\n{node.get(&#39;text&#39;, &#39;&#39;)}&quot;)<br>    context = &quot;\n\n---\n\n&quot;.join(context_parts)<br>    prompt = ANSWER_PROMPT.format(context=context, question=query)<br>    response = await openai_client.chat.completions.create(<br>        model=model,<br>        messages=[{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: prompt}],<br>        temperature=0,<br>    )<br>    return response.choices[0].message.content</pre><ul><li><strong>Full PageIndex Pipeline<br></strong>pageindex_pipeline chains tree_search and generate_answer into a single awaitable call and returns the query, reasoning, retrieved section IDs, and the final answer together as a dict.</li></ul><pre>async def pageindex_pipeline(tree, query: str, node_map: dict, openai_client: AsyncOpenAI) -&gt; dict:<br>    search_result = await tree_search(tree, query, openai_client=openai_client)<br>    answer = await generate_answer(search_result[&quot;node_ids&quot;], query, node_map, openai_client=openai_client)<br>    return {<br>        &quot;query&quot;: query,<br>        &quot;reasoning&quot;: search_result[&quot;reasoning&quot;],<br>        &quot;retrieved_sections&quot;: search_result[&quot;node_ids&quot;],<br>        &quot;answer&quot;: answer,<br>    }</pre><h4><strong>Building the Vector RAG Baseline</strong></h4><p>To have something meaningful to compare against, the script defines five helper functions that implement a conventional chunked-embedding pipeline. These are all defined at module level, before main().</p><ul><li>extract_text_from_pdf uses PyMuPDF to read each page and labels it with its page number so chunks retain a location reference.</li></ul><pre>def extract_text_from_pdf(pdf_path: str) -&gt; str:<br>    doc = pymupdf.open(pdf_path)<br>    pages = []<br>    for page_num, page in enumerate(doc):<br>        text = page.get_text()<br>        if text.strip():<br>            pages.append(f&quot;[Page {page_num + 1}]\n{text}&quot;)<br>    return &quot;\n\n&quot;.join(pages)</pre><ul><li>chunk_text splits the concatenated text into 500-word windows with a 50-word overlap so sentences are not cut at chunk boundaries.</li></ul><pre>def chunk_text(text: str, chunk_size: int = 500, overlap: int = 50) -&gt; list[str]:<br>    words = text.split()<br>    chunks = []<br>    start = 0<br>    while start &lt; len(words):<br>        end = min(start + chunk_size, len(words))<br>        chunks.append(&quot; &quot;.join(words[start:end]))<br>        start += chunk_size - overlap<br>    return chunks</pre><ul><li>embed_chunks calls the OpenAI embeddings API in batches of 100 and returns all embeddings as a NumPy array.</li></ul><pre>def embed_chunks(chunks: list[str], sync_openai: OpenAI, model: str = &quot;text-embedding-3-small&quot;) -&gt; np.ndarray:<br>    all_embeddings = []<br>    batch_size = 100<br>    for i in range(0, len(chunks), batch_size):<br>        batch = chunks[i : i + batch_size]<br>        response = sync_openai.embeddings.create(input=batch, model=model)<br>        all_embeddings.extend(item.embedding for item in response.data)<br>    return np.array(all_embeddings, dtype=&quot;float32&quot;)</pre><ul><li>build_faiss_index L2-normalises the embeddings so that inner-product search is equivalent to cosine similarity, then adds them to a FAISS flat index.</li></ul><pre>def build_faiss_index(embeddings: np.ndarray) -&gt; faiss.IndexFlatIP:<br>    faiss.normalize_L2(embeddings)<br>    index = faiss.IndexFlatIP(embeddings.shape[1])<br>    index.add(embeddings)<br>    return index</pre><ul><li>vector_rag_pipeline embeds the query, retrieves the top five nearest chunks, and passes them to the generation model — the full retrieval-augmented generation loop in one function.</li></ul><pre>def vector_rag_pipeline(query: str, chunks: list[str], index: faiss.IndexFlatIP, sync_openai: OpenAI, k: int = 5, model: str = &quot;gpt-4o-mini&quot;) -&gt; str:<br>    query_vec = np.array(<br>        [sync_openai.embeddings.create(input=[query], model=&quot;text-embedding-3-small&quot;).data[0].embedding],<br>        dtype=&quot;float32&quot;,<br>    )<br>    faiss.normalize_L2(query_vec)<br>    _, indices = index.search(query_vec, k)<br>    context = &quot;\n\n---\n\n&quot;.join(chunks[i] for i in indices[0] if i &lt; len(chunks))<br>    response = sync_openai.chat.completions.create(<br>        model=model,<br>        messages=[<br>            {&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: &quot;Answer the question using only the provided context. Be precise.&quot;},<br>            {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: f&quot;Context:\n{context}\n\nQuestion: {query}&quot;},<br>        ],<br>        temperature=0,<br>    )<br>    return response.choices[0].message.content</pre><h4><strong>Putting It All Together — The main() Function</strong></h4><p>With all helper functions defined, main() orchestrates the full workflow: initialise clients, download the PDF, submit it to PageIndex, wait for indexing, inspect the tree, run PageIndex queries, build the FAISS baseline, and run the side-by-side comparison.</p><ul><li><strong>Client Initialisation</strong></li></ul><pre>async def main() -&gt; None:<br>    pi_client = PageIndexClient(index=&quot;gpt-4o-mini&quot;, chat=&quot;gpt-4o-mini&quot;)<br>    openai_client = AsyncOpenAI()<br>    sync_openai = OpenAI()</pre><p>The PageIndexClient takes two model parameters: index controls which model builds the tree during the one-time indexing step, and chat controls which model reasons over it at query time.</p><ul><li><strong>Downloading the Document<br></strong>The script downloads the Federal Reserve Financial Stability Report (May 2026) once and reuses the local copy on subsequent runs.</li></ul><pre>    pdf_path = os.path.join(&quot;./data&quot;, &quot;fed_financial_stability_report_2026.pdf&quot;)<br><br>    if not os.path.exists(pdf_path):<br>        print(&quot;Downloading Federal Reserve Financial Stability Report (May 2026)...&quot;)<br>        response = requests.get(<br>            &quot;https://www.federalreserve.gov/publications/files/financial-stability-report-20260508.pdf&quot;,<br>            timeout=60,<br>        )<br>        response.raise_for_status()<br>        with open(pdf_path, &quot;wb&quot;) as file:<br>            file.write(response.content)<br>        print(f&quot;Saved to {pdf_path}&quot;)<br>    else:<br>        print(f&quot;Already present: {pdf_path}&quot;)</pre><ul><li><strong>Submitting and Polling<br></strong>submit_document sends the PDF and returns a doc_id immediately. The script then polls every ten seconds until the status reaches ”completed”, which typically takes one to three minutes for a document of this length.</li></ul><pre>    print(&quot;Submitting document to PageIndex...&quot;)<br>    submit_result = pi_client.submit_document(pdf_path)<br>    doc_id = submit_result[&quot;doc_id&quot;]<br>    print(f&quot;Document ID: {doc_id}&quot;)<br><br>    print(&quot;Waiting for tree generation...&quot;)<br>    while True:<br>        status_result = pi_client.get_document(doc_id)<br>        status = status_result.get(&quot;status&quot;)<br>        print(f&quot;  Status: {status}&quot;)<br>        if status == &quot;completed&quot;:<br>            break<br>        if status == &quot;failed&quot;:<br>            raise RuntimeError(f&quot;Processing failed: {status_result}&quot;)<br>        time.sleep(10)<br><br>    print(&quot;Done.&quot;)</pre><ul><li><strong>Inspecting the Tree<br></strong>Once processing completes, get_tree returns the full hierarchy. utils.create_node_mapping flattens it into a {node_id: node} dict that generate_answer uses for fast lookups. Each node has a node_id, title, start_index and end_index for the page range, and a plain-English summary. Major sections can contain nested child nodes for subsections.</li></ul><pre>    tree_result = pi_client.get_tree(doc_id, node_summary=True)<br>    tree = tree_result[&quot;result&quot;]<br>    node_map = utils.create_node_mapping(tree)<br><br>    print(f&quot;\nTop-level nodes ({len(tree)} sections):\n&quot;)<br>    for node in tree:<br>        print(f&quot;  [{node[&#39;node_id&#39;]}] {node[&#39;title&#39;]}&quot;)<br>        print(f&quot;       Pages {node[&#39;start_index&#39;]}-{node[&#39;end_index&#39;]}&quot;)<br>        print(f&quot;       {node[&#39;summary&#39;][:120]}...&quot;)<br>        print()</pre><ul><li><strong>Running a PageIndex Query<br></strong>With the tree available, tree_search reasons over it to identify the most relevant sections, and pageindex_pipeline then fetches their text and generates a cited answer.</li></ul><pre>    query_1 = &quot;What are the main vulnerabilities the Fed identified in asset valuations as of October 2023, and which markets were flagged as stretched?&quot;<br>    search_result = await tree_search(tree, query_1, openai_client=openai_client)<br><br>    print(&quot;LLM Reasoning:&quot;)<br>    print(search_result[&quot;reasoning&quot;])<br>    print(&quot;\nSelected node IDs:&quot;, search_result[&quot;node_ids&quot;])<br><br>    pi_result = await pageindex_pipeline(tree, query_1, node_map, openai_client=openai_client)<br>    print(f&quot;Query: {pi_result[&#39;query&#39;]}\n&quot;)<br>    print(f&quot;Retrieved sections: {pi_result[&#39;retrieved_sections&#39;]}\n&quot;)<br>    print(f&quot;Answer:\n{pi_result[&#39;answer&#39;]}&quot;)</pre><ul><li><strong>Building the FAISS Index<br></strong>The vector RAG baseline is built from the same PDF by calling the five helper functions defined earlier.</li></ul><pre>    print(&quot;\n\n## Comparing PageIndex Against Vector RAG&quot;)<br>    print(&quot;Extracting text from PDF...&quot;)<br>    raw_text = extract_text_from_pdf(pdf_path)<br><br>    print(&quot;Chunking text...&quot;)<br>    chunks = chunk_text(raw_text, chunk_size=500, overlap=50)<br>    print(f&quot;  {len(chunks)} chunks created&quot;)<br><br>    print(&quot;Embedding chunks...&quot;)<br>    embeddings = embed_chunks(chunks, sync_openai)<br>    print(f&quot;  Embeddings shape: {embeddings.shape}&quot;)<br><br>    print(&quot;Building FAISS index...&quot;)<br>    faiss_index = build_faiss_index(embeddings)<br>    print(&quot;Done.\n&quot;)</pre><ul><li><strong>Running the Comparison<br></strong>Three queries are designed to stress different retrieval challenges: a direct factual lookup, a methodology question that lives inside a nested box section, and a multi-concept question requiring information from two separate chapters.</li></ul><pre>    queries = [<br>        &quot;What does the Fed consider the most significant near-term risk to financial stability as of October 2023?&quot;,<br>        &quot;What methodology does the Fed use to assess climate-related financial risks, and where in the report is it described?&quot;,<br>        &quot;How do elevated asset valuations interact with leverage in the financial sector to amplify systemic risk, according to the report?&quot;,<br>    ]<br><br>    results = []<br>    for query in queries:<br>        print(f&quot;\nQuery: {query}\n&quot;)<br>        pi_result = await pageindex_pipeline(tree, query, node_map, openai_client=openai_client)<br>        vec_answer = vector_rag_pipeline(query, chunks, faiss_index, sync_openai)<br><br>        results.append({<br>            &quot;query&quot;: query,<br>            &quot;pageindex_answer&quot;: pi_result[&quot;answer&quot;],<br>            &quot;vector_rag_answer&quot;: vec_answer,<br>        })<br><br>        print(&quot;PageIndex answer:&quot;)<br>        print(pi_result[&quot;answer&quot;][:500])<br>        print(&quot;\nVector RAG answer:&quot;)<br>        print(vec_answer[:500])<br>        print(&quot;\n&quot; + &quot;=&quot; * 60)<br><br><br>if __name__ == &quot;__main__&quot;:<br>    asyncio.run(main())</pre><h4>Complete script</h4><p>Let’s bring all the components together into a single Python script.</p><pre># app.py<br><br>import copy<br>import time<br>import json<br>import asyncio<br>import requests<br>import pymupdf<br>from openai import AsyncOpenAI, OpenAI<br>import numpy as np<br>import faiss<br>from pageindex import PageIndexClient<br>import pageindex.utils as utils<br><br>from dotenv import load_dotenv<br>import os<br>load_dotenv()<br>assert os.getenv(&#39;OPENAI_API_KEY&#39;), &quot;Environment variable &#39;OPENAI_API_KEY&#39; is not set&quot;<br><br><br>TREE_SEARCH_PROMPT = &quot;&quot;&quot;You are a document retrieval assistant.<br>Given a document&#39;s tree structure and a user query, identify which nodes (sections)<br>are most likely to contain the answer.<br><br>Document tree:<br>{tree_json}<br><br>User query: {query}<br><br>Return a JSON object with the following format:<br>{{<br>  &quot;reasoning&quot;: &quot;Your step-by-step reasoning about which sections to retrieve&quot;,<br>  &quot;node_ids&quot;: [&quot;id1&quot;, &quot;id2&quot;, ...]<br>}}<br><br>Return ONLY the JSON object, no other text.&quot;&quot;&quot;<br><br>async def tree_search(tree, query: str, openai_client: AsyncOpenAI, model: str = &quot;gpt-4o-mini&quot;) -&gt; dict:<br>    slim_tree = utils.remove_fields(copy.deepcopy(tree), fields=[&quot;text&quot;])<br>    tree_json = json.dumps(slim_tree, indent=2)<br>    prompt = TREE_SEARCH_PROMPT.format(tree_json=tree_json, query=query)<br>    response = await openai_client.chat.completions.create(<br>        model=model,<br>        messages=[{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: prompt}],<br>        temperature=0,<br>        response_format={&quot;type&quot;: &quot;json_object&quot;},<br>    )<br>    return json.loads(response.choices[0].message.content)<br><br>ANSWER_PROMPT = &quot;&quot;&quot;You are a financial document analyst. Answer the user&#39;s question<br>using ONLY the provided context. Cite the specific section(s) you are drawing from.<br>If the context does not contain enough information to answer, say so clearly.<br><br>Context:<br>{context}<br><br>Question: {question}<br><br>Provide a precise, well-cited answer.&quot;&quot;&quot;<br><br>async def generate_answer(node_ids: list, query: str, node_map: dict, openai_client: AsyncOpenAI, model: str = &quot;gpt-4o-mini&quot;) -&gt; str:<br>    context_parts = []<br>    for node_id in node_ids:<br>        node = node_map.get(node_id)<br>        if node:<br>            section_header = f&quot;[{node[&#39;title&#39;]} | Pages {node[&#39;start_index&#39;]}-{node[&#39;end_index&#39;]}]&quot;<br>            context_parts.append(f&quot;{section_header}\n{node.get(&#39;text&#39;, &#39;&#39;)}&quot;)<br>    context = &quot;\n\n---\n\n&quot;.join(context_parts)<br>    prompt = ANSWER_PROMPT.format(context=context, question=query)<br>    response = await openai_client.chat.completions.create(<br>        model=model,<br>        messages=[{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: prompt}],<br>        temperature=0,<br>    )<br>    return response.choices[0].message.content<br><br><br>async def pageindex_pipeline(tree, query: str, node_map: dict, openai_client: AsyncOpenAI) -&gt; dict:<br>    search_result = await tree_search(tree, query, openai_client=openai_client)<br>    answer = await generate_answer(search_result[&quot;node_ids&quot;], query, node_map, openai_client=openai_client)<br>    return {<br>        &quot;query&quot;: query,<br>        &quot;reasoning&quot;: search_result[&quot;reasoning&quot;],<br>        &quot;retrieved_sections&quot;: search_result[&quot;node_ids&quot;],<br>        &quot;answer&quot;: answer,<br>    }<br><br><br>def extract_text_from_pdf(pdf_path: str) -&gt; str:<br>    doc = pymupdf.open(pdf_path)<br>    pages = []<br>    for page_num, page in enumerate(doc):<br>        text = page.get_text()<br>        if text.strip():<br>            pages.append(f&quot;[Page {page_num + 1}]\n{text}&quot;)<br>    return &quot;\n\n&quot;.join(pages)<br><br><br>def chunk_text(text: str, chunk_size: int = 500, overlap: int = 50) -&gt; list[str]:<br>    words = text.split()<br>    chunks = []<br>    start = 0<br>    while start &lt; len(words):<br>        end = min(start + chunk_size, len(words))<br>        chunks.append(&quot; &quot;.join(words[start:end]))<br>        start += chunk_size - overlap<br>    return chunks<br><br><br>def embed_chunks(chunks: list[str], sync_openai: OpenAI, model: str = &quot;text-embedding-3-small&quot;) -&gt; np.ndarray:<br>    all_embeddings = []<br>    batch_size = 100<br>    for i in range(0, len(chunks), batch_size):<br>        batch = chunks[i : i + batch_size]<br>        response = sync_openai.embeddings.create(input=batch, model=model)<br>        all_embeddings.extend(item.embedding for item in response.data)<br>    return np.array(all_embeddings, dtype=&quot;float32&quot;)<br><br><br>def build_faiss_index(embeddings: np.ndarray) -&gt; faiss.IndexFlatIP:<br>    faiss.normalize_L2(embeddings)<br>    index = faiss.IndexFlatIP(embeddings.shape[1])<br>    index.add(embeddings)<br>    return index<br><br><br>def vector_rag_pipeline(query: str, chunks: list[str], index: faiss.IndexFlatIP, sync_openai: OpenAI, k: int = 5, model: str = &quot;gpt-4o-mini&quot;) -&gt; str:<br>    query_vec = np.array(<br>        [sync_openai.embeddings.create(input=[query], model=&quot;text-embedding-3-small&quot;).data[0].embedding],<br>        dtype=&quot;float32&quot;,<br>    )<br>    faiss.normalize_L2(query_vec)<br>    _, indices = index.search(query_vec, k)<br>    context = &quot;\n\n---\n\n&quot;.join(chunks[i] for i in indices[0] if i &lt; len(chunks))<br>    response = sync_openai.chat.completions.create(<br>        model=model,<br>        messages=[<br>            {&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: &quot;Answer the question using only the provided context. Be precise.&quot;},<br>            {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: f&quot;Context:\n{context}\n\nQuestion: {query}&quot;},<br>        ],<br>        temperature=0,<br>    )<br>    return response.choices[0].message.content<br><br><br>async def main() -&gt; None:<br>    pi_client = PageIndexClient(index=&quot;gpt-4o-mini&quot;, chat=&quot;gpt-4o-mini&quot;)<br>    openai_client = AsyncOpenAI()<br>    sync_openai = OpenAI()<br><br>    pdf_path = os.path.join(&quot;./data&quot;, &quot;fed_financial_stability_report_2026.pdf&quot;)<br><br>    if not os.path.exists(pdf_path):<br>        print(&quot;Downloading Federal Reserve Financial Stability Report (May 2026)...&quot;)<br>        response = requests.get(<br>            &quot;https://www.federalreserve.gov/publications/files/financial-stability-report-20260508.pdf&quot;,<br>            timeout=60,<br>        )<br>        response.raise_for_status()<br>        with open(pdf_path, &quot;wb&quot;) as file:<br>            file.write(response.content)<br>        print(f&quot;Saved to {pdf_path}&quot;)<br>    else:<br>        print(f&quot;Already present: {pdf_path}&quot;)<br><br>    print(&quot;Submitting document to PageIndex...&quot;)<br>    submit_result = pi_client.submit_document(pdf_path)<br>    doc_id = submit_result[&quot;doc_id&quot;]<br>    print(f&quot;Document ID: {doc_id}&quot;)<br><br>    print(&quot;Waiting for tree generation...&quot;)<br>    while True:<br>        status_result = pi_client.get_document(doc_id)<br>        status = status_result.get(&quot;status&quot;)<br>        print(f&quot;  Status: {status}&quot;)<br>        if status == &quot;completed&quot;:<br>            break<br>        if status == &quot;failed&quot;:<br>            raise RuntimeError(f&quot;Processing failed: {status_result}&quot;)<br>        time.sleep(10)<br><br>    print(&quot;Done.&quot;)<br><br>    tree_result = pi_client.get_tree(doc_id, node_summary=True)<br>    tree = tree_result[&quot;result&quot;]<br>    node_map = utils.create_node_mapping(tree)<br><br>    print(f&quot;\nTop-level nodes ({len(tree)} sections):\n&quot;)<br>    for node in tree:<br>        print(f&quot;  [{node[&#39;node_id&#39;]}] {node[&#39;title&#39;]}&quot;)<br>        print(f&quot;       Pages {node[&#39;start_index&#39;]}-{node[&#39;end_index&#39;]}&quot;)<br>        print(f&quot;       {node[&#39;summary&#39;][:120]}...&quot;)<br>        print()<br><br>    query_1 = &quot;What are the main vulnerabilities the Fed identified in asset valuations as of October 2023, and which markets were flagged as stretched?&quot;<br>    search_result = await tree_search(tree, query_1, openai_client=openai_client)<br><br>    print(&quot;LLM Reasoning:&quot;)<br>    print(search_result[&quot;reasoning&quot;])<br>    print(&quot;\nSelected node IDs:&quot;, search_result[&quot;node_ids&quot;])<br><br>    pi_result = await pageindex_pipeline(tree, query_1, node_map, openai_client=openai_client)<br>    print(f&quot;Query: {pi_result[&#39;query&#39;]}\n&quot;)<br>    print(f&quot;Retrieved sections: {pi_result[&#39;retrieved_sections&#39;]}\n&quot;)<br>    print(f&quot;Answer:\n{pi_result[&#39;answer&#39;]}&quot;)<br><br>    print(&quot;\n\n## Comparing PageIndex Against Vector RAG&quot;)<br>    print(&quot;Extracting text from PDF...&quot;)<br>    raw_text = extract_text_from_pdf(pdf_path)<br><br>    print(&quot;Chunking text...&quot;)<br>    chunks = chunk_text(raw_text, chunk_size=500, overlap=50)<br>    print(f&quot;  {len(chunks)} chunks created&quot;)<br><br>    print(&quot;Embedding chunks...&quot;)<br>    embeddings = embed_chunks(chunks, sync_openai)<br>    print(f&quot;  Embeddings shape: {embeddings.shape}&quot;)<br><br>    print(&quot;Building FAISS index...&quot;)<br>    faiss_index = build_faiss_index(embeddings)<br>    print(&quot;Done.\n&quot;)<br><br>    queries = [<br>        &quot;What does the Fed consider the most significant near-term risk to financial stability as of October 2023?&quot;,<br>        &quot;What methodology does the Fed use to assess climate-related financial risks, and where in the report is it described?&quot;,<br>        &quot;How do elevated asset valuations interact with leverage in the financial sector to amplify systemic risk, according to the report?&quot;,<br>    ]<br><br>    results = []<br>    for query in queries:<br>        print(f&quot;\nQuery: {query}\n&quot;)<br>        pi_result = await pageindex_pipeline(tree, query, node_map, openai_client=openai_client)<br>        vec_answer = vector_rag_pipeline(query, chunks, faiss_index, sync_openai)<br><br>        results.append({<br>            &quot;query&quot;: query,<br>            &quot;pageindex_answer&quot;: pi_result[&quot;answer&quot;],<br>            &quot;vector_rag_answer&quot;: vec_answer,<br>        })<br><br>        print(&quot;PageIndex answer:&quot;)<br>        print(pi_result[&quot;answer&quot;][:500])<br>        print(&quot;\nVector RAG answer:&quot;)<br>        print(vec_answer[:500])<br>        print(&quot;\n&quot; + &quot;=&quot; * 60)<br><br><br>if __name__ == &quot;__main__&quot;:<br>    asyncio.run(main())</pre><h4>Run the app</h4><p>Lets execute the app using the following command to see the result.</p><pre>python app.py</pre><p>Expected output:</p><pre>(pageindex-demo) E:\Experiments\PageIndex-Demo&gt;python financial_report_analyser\app.py<br>Already present: ./data\fed_financial_stability_report_2026.pdf<br>Submitting document to PageIndex...<br>E:\Experiments\PageIndex-Demo\financial_report_analyser\app.py:165: UserWarning: Document &quot;fed_financial_stability_report_2026.pdf&quot; was stored as &quot;fed_financial_stability_report_2026_3.pdf&quot;.<br>  submit_result = pi_client.submit_document(pdf_path)<br>Document ID: pi-49f686c6313344e39e41af9472d92056<br>Waiting for tree generation...<br>  Status: completed<br>Done.<br><br>Top-level nodes (10 sections):<br><br>  [0000] Financial Stability Report<br>       Pages 1-3<br>       The Financial Stability Report by the Federal Reserve System outlines its key functions aimed at promoting the U.S. econ...<br><br>  [0001] Contents<br>       Pages 3-5<br>       The report outlines the Federal Reserve Board&#39;s assessment of the U.S. financial system&#39;s stability, emphasizing the imp...<br><br>  [0002] Purpose and Framework<br>       Pages 5-9<br>       The section discusses the Federal Reserve&#39;s monitoring efforts and actions to ensure the resilience of the U.S. financia...<br><br>  [0005] Overview<br>       Pages 9-13<br>       The report evaluates vulnerabilities in the U.S. financial system, focusing on asset valuations, borrowing by businesses...<br><br>  [0006] 1 Asset Valuations<br>       Pages 13-23<br>       The section on asset valuations highlights a persistent high level of asset valuations across various markets, including...<br><br>  [0012] 2 Borrowing by Businesses and Households<br>       Pages 23-33<br>       The section addresses the borrowing trends of businesses and households, revealing moderate vulnerabilities in debt leve...<br><br>  [0018] 3 Leverage in the Financial Sector<br>       Pages 33-43<br>       The section explores leverage dynamics within the financial sector, emphasizing stable high leverage levels in hedge fun...<br><br>  [0027] 4 Funding Risks<br>       Pages 43-55<br>       The section on Funding Risks highlights moderate funding risks for banks, with stable uninsured deposits and sufficient ...<br><br>  [0035] 5 Near-Term Risks to the Financial System<br>       Pages 55-59<br>       The section outlines near-term risks to financial stability as identified by the Federal Reserve, focusing on cyberattac...<br><br>  [0038] Appendix: Figure Notes<br>       Pages 59-71<br>       The section presents an analysis of financial metrics related to Treasury yields, interest rates, corporate bond yields,...<br><br>LLM Reasoning:<br>To answer the query about the main vulnerabilities identified by the Fed in asset valuations as of October 2023, I will focus on sections that discuss asset valuations and vulnerabilities. The &#39;Overview&#39; section (node_id: &#39;0005&#39;) provides a general evaluation of vulnerabilities in the U.S. financial system, including asset valuations. The &#39;1 Asset Valuations&#39; section (node_id: &#39;0006&#39;) specifically addresses asset valuations and highlights high levels across various markets. Additionally, the &#39;2 Borrowing by Businesses and Households&#39; section (node_id: &#39;0012&#39;) may provide context on how these valuations impact borrowing trends, which could relate to vulnerabilities. Therefore, I will retrieve the &#39;Overview&#39;, &#39;1 Asset Valuations&#39;, and &#39;2 Borrowing by Businesses and Households&#39; sections.<br><br>Selected node IDs: [&#39;0005&#39;, &#39;0006&#39;, &#39;0012&#39;]<br>Query: What are the main vulnerabilities the Fed identified in asset valuations as of October 2023, and which markets were flagged as stretched?<br><br>Retrieved sections: [&#39;0005&#39;, &#39;0006&#39;]<br><br>Answer:<br>As of October 2023, the main vulnerabilities identified by the Fed in asset valuations included elevated asset valuation pressures, particularly in equity markets and residential real estate. Specifically, the report noted that measures of broad equity valuations remained elevated, with the forward equity price-to-earnings ratio staying in the upper ranges of its historical distribution. Additionally, the ratio of house prices to rents was also in the upper ranges of its historical distribution, indicating that residential real estate prices were well above their historical relationship with fundamentals, even as house price growth continued to slow (see Overview, Pages 9-13 and Section 1, Asset Valuations, Pages 13-23).<br><br>The markets flagged as stretched included public equities and residential real estate, which were described as the two largest asset markets, significantly larger than Treasury securities and commercial real estate (CRE) (see Section 1, Asset Valuations, Pages 13-23).<br><br><br>## Comparing PageIndex Against Vector RAG<br>Extracting text from PDF...<br>Chunking text...<br>  48 chunks created<br>Embedding chunks...<br>  Embeddings shape: (48, 1536)<br>Building FAISS index...<br>Done.<br><br><br>Query: What does the Fed consider the most significant near-term risk to financial stability as of October 2023?<br><br>PageIndex answer:<br>As of October 2023, the Federal Reserve considers geopolitical risks and an oil shock to be the most significant near-term risks to financial stability. This is highlighted in the &quot;Survey of Salient Risks to Financial Stability,&quot; where respondents noted the inflationary implications of energy supply disruptions following the outbreak of the Iran conflict as a top concern (Box 5.1, Pages 57-59). Additionally, the potential for a prolonged period of higher inflation due to these geopolitical tensi<br><br>Vector RAG answer:<br>As of October 2023, the Federal Reserve considers geopolitical risks and an oil shock as the most significant near-term risks to financial stability.<br><br>============================================================<br><br>Query: What methodology does the Fed use to assess climate-related financial risks, and where in the report is it described?<br><br>PageIndex answer:<br>The provided context does not contain any information regarding the methodology the Fed uses to assess climate-related financial risks or where it is described in the report. Therefore, I cannot answer the question based on the available information.<br><br>Vector RAG answer:<br>The provided context does not mention a specific methodology used by the Federal Reserve to assess climate-related financial risks, nor does it indicate where such a description might be found in the report. The focus is primarily on financial vulnerabilities, asset valuations, borrowing, leverage, and funding risks, without reference to climate-related assessments.<br><br>============================================================<br><br>Query: How do elevated asset valuations interact with leverage in the financial sector to amplify systemic risk, according to the report?<br><br>PageIndex answer:<br>The report indicates that elevated asset valuations, particularly in the context of increased risk premiums and volatility, can interact with leverage in the financial sector to amplify systemic risk. Specifically, it notes that &quot;asset valuations remained elevated, although risk premiums increased in several markets amid periods of higher volatility&quot; (Section 1, Asset Valuations). This suggests that while asset prices are high, the increased risk premiums reflect growing concerns about potential<br><br>Vector RAG answer:<br>Elevated asset valuations can increase the possibility of outsized drops in asset prices, which, when combined with high leverage in the financial sector, raises systemic risk. High leverage means that financial institutions may not have the ability to absorb losses without disruptions to their operations. If asset prices fall significantly, leveraged institutions may be forced to sell assets at depressed prices, leading to further declines in asset values and creating stress across markets. Thi<br><br>============================================================</pre><h4>Streamlit interface</h4><p>For a better user experience, we can add a simple Streamlit interface on top of the Financial Report Analyser application. This provides a lightweight web-based UI for uploading documents, entering questions, and viewing the generated answers.</p><p>The complete Streamlit implementation is shown below.</p><p>Create a streamlit_app.py and add the following code to it.</p><pre>import asyncio<br>import os<br>import time<br>import requests<br>import streamlit as st<br>from openai import AsyncOpenAI, OpenAI<br>from pageindex import PageIndexClient<br>import pageindex.utils as utils<br><br>from app import (<br>    tree_search,<br>    pageindex_pipeline,<br>    extract_text_from_pdf,<br>    chunk_text,<br>    embed_chunks,<br>    build_faiss_index,<br>    vector_rag_pipeline,<br>)<br><br>PDF_URL = &quot;https://www.federalreserve.gov/publications/files/financial-stability-report-20260508.pdf&quot;<br>DEFAULT_PDF_PATH = os.path.join(&quot;.&quot;, &quot;data&quot;, &quot;fed_financial_stability_report_2026.pdf&quot;)<br><br>COMPARISON_QUERIES = [<br>    &quot;What does the Fed consider the most significant near-term risk to financial stability as of October 2023?&quot;,<br>    &quot;What methodology does the Fed use to assess climate-related financial risks, and where in the report is it described?&quot;,<br>    &quot;How do elevated asset valuations interact with leverage in the financial sector to amplify systemic risk, according to the report?&quot;,<br>]<br><br><br>def run_async(coro):<br>    return asyncio.run(coro)<br><br><br>def get_clients():<br>    if &quot;pi_client&quot; not in st.session_state:<br>        st.session_state.pi_client = PageIndexClient(index=&quot;gpt-4o-mini&quot;, chat=&quot;gpt-4o-mini&quot;)<br>    if &quot;openai_client&quot; not in st.session_state:<br>        st.session_state.openai_client = AsyncOpenAI()<br>    if &quot;sync_openai&quot; not in st.session_state:<br>        st.session_state.sync_openai = OpenAI()<br><br><br>def get_default_pdf() -&gt; str:<br>    os.makedirs(os.path.dirname(DEFAULT_PDF_PATH), exist_ok=True)<br>    if not os.path.exists(DEFAULT_PDF_PATH):<br>        with st.spinner(&quot;Downloading Federal Reserve Financial Stability Report (May 2026)...&quot;):<br>            r = requests.get(PDF_URL, timeout=60)<br>            r.raise_for_status()<br>            with open(DEFAULT_PDF_PATH, &quot;wb&quot;) as f:<br>                f.write(r.content)<br>    return DEFAULT_PDF_PATH<br><br><br>def save_uploaded_pdf(uploaded_file) -&gt; str:<br>    os.makedirs(os.path.join(&quot;.&quot;, &quot;data&quot;), exist_ok=True)<br>    save_path = os.path.join(&quot;.&quot;, &quot;data&quot;, uploaded_file.name)<br>    with open(save_path, &quot;wb&quot;) as f:<br>        f.write(uploaded_file.getbuffer())<br>    return save_path<br><br><br>def index_document(pdf_path: str):<br>    pi_client = st.session_state.pi_client<br><br>    with st.spinner(&quot;Submitting document to PageIndex...&quot;):<br>        result = pi_client.submit_document(pdf_path)<br>        doc_id = result[&quot;doc_id&quot;]<br><br>    status_box = st.empty()<br>    while True:<br>        status = pi_client.get_document(doc_id).get(&quot;status&quot;)<br>        status_box.info(f&quot;Indexing status: **{status}**&quot;)<br>        if status == &quot;completed&quot;:<br>            break<br>        if status == &quot;failed&quot;:<br>            status_box.error(&quot;Indexing failed.&quot;)<br>            return<br>        time.sleep(10)<br><br>    status_box.success(&quot;Indexing complete!&quot;)<br><br>    tree_result = pi_client.get_tree(doc_id, node_summary=True)<br>    st.session_state.doc_id = doc_id<br>    st.session_state.tree = tree_result[&quot;result&quot;]<br>    st.session_state.node_map = utils.create_node_mapping(st.session_state.tree)<br>    st.session_state.pdf_path = pdf_path<br>    st.session_state.indexed = True<br><br><br>def build_vector_index():<br>    with st.spinner(&quot;Extracting text from PDF...&quot;):<br>        raw_text = extract_text_from_pdf(st.session_state.pdf_path)<br><br>    with st.spinner(&quot;Chunking text...&quot;):<br>        chunks = chunk_text(raw_text, chunk_size=500, overlap=50)<br><br>    with st.spinner(f&quot;Embedding {len(chunks)} chunks (this may take a minute)...&quot;):<br>        embeddings = embed_chunks(chunks, st.session_state.sync_openai)<br><br>    with st.spinner(&quot;Building FAISS index...&quot;):<br>        faiss_index = build_faiss_index(embeddings)<br><br>    st.session_state.chunks = chunks<br>    st.session_state.faiss_index = faiss_index<br>    st.session_state.faiss_ready = True<br><br><br># ── Page config ───────────────────────────────────────────────────────────────<br><br>st.set_page_config(<br>    page_title=&quot;Financial Report Analyser&quot;,<br>    page_icon=&quot;📊&quot;,<br>    layout=&quot;wide&quot;,<br>)<br><br>st.title(&quot;📊 Financial Report Analyser&quot;)<br>st.caption(&quot;PageIndex tree-based RAG vs Vector RAG&quot;)<br><br>get_clients()<br><br># ── Sidebar ───────────────────────────────────────────────────────────────────<br><br>with st.sidebar:<br>    st.header(&quot;Setup&quot;)<br><br>    if not st.session_state.get(&quot;indexed&quot;):<br>        source = st.radio(<br>            &quot;Document source&quot;,<br>            [&quot;Use default Federal Reserve report&quot;, &quot;Upload your own PDF&quot;],<br>            index=0,<br>        )<br><br>        pdf_path_to_index = None<br><br>        if source == &quot;Upload your own PDF&quot;:<br>            uploaded = st.file_uploader(&quot;Upload a PDF&quot;, type=&quot;pdf&quot;)<br>            if uploaded:<br>                pdf_path_to_index = save_uploaded_pdf(uploaded)<br>                st.caption(f&quot;Ready: `{uploaded.name}`&quot;)<br>        else:<br>            pdf_path_to_index = &quot;default&quot;<br>            st.caption(&quot;Federal Reserve Financial Stability Report (May 2026)&quot;)<br><br>        ready_to_index = pdf_path_to_index is not None<br>        if st.button(&quot;Index Document&quot;, type=&quot;primary&quot;, use_container_width=True, disabled=not ready_to_index):<br>            actual_path = get_default_pdf() if pdf_path_to_index == &quot;default&quot; else pdf_path_to_index<br>            index_document(actual_path)<br>            st.rerun()<br>    else:<br>        doc_name = os.path.basename(st.session_state.pdf_path)<br>        st.success(f&quot;PageIndex ready · `{doc_name}`&quot;)<br>        st.code(st.session_state.doc_id, language=None)<br><br>        if not st.session_state.get(&quot;faiss_ready&quot;):<br>            st.info(&quot;Build the vector index to enable Vector RAG and the comparison tab.&quot;)<br>            if st.button(&quot;Build Vector RAG Index&quot;, use_container_width=True):<br>                build_vector_index()<br>                st.rerun()<br>        else:<br>            st.success(&quot;Vector RAG index ready&quot;)<br><br>        st.divider()<br>        if st.button(&quot;Re-index&quot;, use_container_width=True):<br>            for key in [&quot;indexed&quot;, &quot;doc_id&quot;, &quot;tree&quot;, &quot;node_map&quot;, &quot;pdf_path&quot;,<br>                        &quot;faiss_ready&quot;, &quot;chunks&quot;, &quot;faiss_index&quot;]:<br>                st.session_state.pop(key, None)<br>            st.rerun()<br><br>    st.divider()<br>    st.markdown(&quot;**Models**&quot;)<br>    st.markdown(&quot;- PageIndex index/chat: `gpt-4o-mini`&quot;)<br>    st.markdown(&quot;- Embeddings: `text-embedding-3-small`&quot;)<br>    st.markdown(&quot;- Generation: `gpt-4o-mini`&quot;)<br><br># ── Guard ─────────────────────────────────────────────────────────────────────<br><br>if not st.session_state.get(&quot;indexed&quot;):<br>    st.info(&quot;Choose a document source in the sidebar and click **Index Document** to begin.&quot;)<br>    st.stop()<br><br>doc_name = os.path.basename(st.session_state.pdf_path)<br>st.caption(f&quot;Document: **{doc_name}**&quot;)<br><br># ── Tabs ──────────────────────────────────────────────────────────────────────<br><br>tab_tree, tab_query, tab_compare = st.tabs([&quot;🌳 Document Tree&quot;, &quot;🔍 Query&quot;, &quot;⚖️ Compare&quot;])<br><br># ── Tab 1: Document Tree ──────────────────────────────────────────────────────<br><br>with tab_tree:<br>    st.subheader(&quot;Document Tree&quot;)<br>    st.caption(&quot;Top-level sections identified by PageIndex — click to expand subsections.&quot;)<br><br>    for node in st.session_state.tree:<br>        with st.expander(f&quot;[{node[&#39;node_id&#39;]}] {node[&#39;title&#39;]}  ·  pages {node[&#39;start_index&#39;]}–{node[&#39;end_index&#39;]}&quot;):<br>            if node.get(&quot;summary&quot;):<br>                st.write(node[&quot;summary&quot;])<br><br>            children = node.get(&quot;nodes&quot;, [])<br>            if children:<br>                st.markdown(&quot;**Subsections**&quot;)<br>                for child in children:<br>                    st.markdown(<br>                        f&quot;- `{child[&#39;node_id&#39;]}` &amp;nbsp; **{child[&#39;title&#39;]}** &quot;<br>                        f&quot;(pages {child[&#39;start_index&#39;]}–{child[&#39;end_index&#39;]})&quot;<br>                    )<br><br># ── Tab 2: Single Query ───────────────────────────────────────────────────────<br><br>with tab_query:<br>    st.subheader(&quot;Ask a Question&quot;)<br><br>    query = st.text_area(<br>        &quot;Your question&quot;,<br>        placeholder=&quot;What are the main vulnerabilities the Fed identified in asset valuations?&quot;,<br>        height=80,<br>        label_visibility=&quot;collapsed&quot;,<br>    )<br><br>    col_pi_btn, col_vec_btn, _ = st.columns([1, 1, 3])<br>    run_pi = col_pi_btn.button(<br>        &quot;PageIndex&quot;, type=&quot;primary&quot;, use_container_width=True,<br>        disabled=not query,<br>    )<br>    run_vec = col_vec_btn.button(<br>        &quot;Vector RAG&quot;, use_container_width=True,<br>        disabled=not (query and st.session_state.get(&quot;faiss_ready&quot;)),<br>    )<br><br>    if not st.session_state.get(&quot;faiss_ready&quot;):<br>        st.caption(&quot;Build the Vector RAG index in the sidebar to enable Vector RAG queries.&quot;)<br><br>    if run_pi and query:<br>        with st.spinner(&quot;Running PageIndex pipeline...&quot;):<br>            pi_res = run_async(pageindex_pipeline(<br>                st.session_state.tree, query,<br>                st.session_state.node_map,<br>                openai_client=st.session_state.openai_client,<br>            ))<br><br>        with st.expander(&quot;🧠 LLM Reasoning&quot;, expanded=True):<br>            st.write(pi_res[&quot;reasoning&quot;])<br><br>        st.markdown(f&quot;**Retrieved sections:** `{&#39;`, `&#39;.join(pi_res[&#39;retrieved_sections&#39;])}`&quot;)<br><br>        st.markdown(&quot;#### Answer&quot;)<br>        st.write(pi_res[&quot;answer&quot;])<br><br>    if run_vec and query:<br>        with st.spinner(&quot;Running Vector RAG pipeline...&quot;):<br>            vec_answer = vector_rag_pipeline(<br>                query,<br>                st.session_state.chunks,<br>                st.session_state.faiss_index,<br>                st.session_state.sync_openai,<br>            )<br><br>        st.markdown(&quot;#### Vector RAG Answer&quot;)<br>        st.write(vec_answer)<br><br># ── Tab 3: Comparison ─────────────────────────────────────────────────────────<br><br>with tab_compare:<br>    st.subheader(&quot;Side-by-Side Comparison&quot;)<br><br>    if not st.session_state.get(&quot;faiss_ready&quot;):<br>        st.warning(&quot;Build the Vector RAG index in the sidebar to run the comparison.&quot;)<br>        st.stop()<br><br>    if st.button(&quot;Run All Queries&quot;, type=&quot;primary&quot;):<br>        for i, q in enumerate(COMPARISON_QUERIES, 1):<br>            st.markdown(f&quot;---\n**Query {i}:** {q}&quot;)<br><br>            with st.spinner(f&quot;Running query {i} of {len(COMPARISON_QUERIES)}...&quot;):<br>                pi_res = run_async(pageindex_pipeline(<br>                    st.session_state.tree, q,<br>                    st.session_state.node_map,<br>                    openai_client=st.session_state.openai_client,<br>                ))<br>                vec_answer = vector_rag_pipeline(<br>                    q,<br>                    st.session_state.chunks,<br>                    st.session_state.faiss_index,<br>                    st.session_state.sync_openai,<br>                )<br><br>            col_pi, col_vec = st.columns(2)<br><br>            with col_pi:<br>                st.markdown(&quot;**PageIndex**&quot;)<br>                st.caption(f&quot;Sections: `{&#39;`, `&#39;.join(pi_res[&#39;retrieved_sections&#39;])}`&quot;)<br>                with st.expander(&quot;Reasoning&quot;):<br>                    st.write(pi_res[&quot;reasoning&quot;])<br>                st.write(pi_res[&quot;answer&quot;])<br><br>            with col_vec:<br>                st.markdown(&quot;**Vector RAG**&quot;)<br>                st.write(vec_answer)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*Mdn1mlnbTVzDOuflo-MCdQ.gif" /></figure><p><em>Thanks for reading this article !!</em></p><p>If you enjoyed this article, please click on the clap button 👏 and share to help others find it!</p><p>The full source code for this tutorial can be found here,</p><p><a href="https://github.com/codemaker2015/pageindex-demo">GitHub - codemaker2015/pageindex-demo: A hands-on demonstration of [PageIndex](https://pageindex.ai) - a tree-based document retrieval framework that lets an LLM reason over a document&#39;s hierarchical structure instead of searching through embedding-similarity chunks.</a></p><h4>Resources</h4><ul><li><a href="https://github.com/VectifyAI/PageIndex">GitHub - VectifyAI/PageIndex: 📑 PageIndex: Document Index for Vectorless, Reasoning-based RAG</a></li><li><a href="https://docs.pageindex.ai/sdk/client">PageIndex</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=de7ecd95a0a2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Your Practical Guide to LLM Observability with Langfuse]]></title>
            <link>https://codemaker2016.medium.com/your-practical-guide-to-llm-observability-with-langfuse-85644f35a4ca?source=rss-b7a60e472dfc------2</link>
            <guid isPermaLink="false">https://medium.com/p/85644f35a4ca</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[langchain]]></category>
            <category><![CDATA[langfuse]]></category>
            <category><![CDATA[llm-observability]]></category>
            <dc:creator><![CDATA[Vishnu Sivan]]></dc:creator>
            <pubDate>Sat, 26 Sep 2026 13:09:38 GMT</pubDate>
            <atom:updated>2026-09-26T13:09:38.642Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x75cXr0dmhpcrw3VhtQNDQ.png" /></figure><p>Large Language Model (LLM) applications often look impressive in development environments, but production introduces a very different reality. Responses become inconsistent, latency fluctuates, prompts behave unpredictably and debugging turns into a challenge because LLM systems are inherently non-deterministic. Unlike traditional software, where failures usually leave clear traces, LLM applications can fail in subtle, context-dependent ways that are difficult to reproduce and diagnose.</p><p>This is where Langfuse emerges as a critical platform for modern AI engineering. Built as an open-source observability and management solution for LLM applications, Langfuse helps developers gain visibility into model behavior, monitor performance, trace complex workflows, manage prompts with version control, and evaluate output quality at scale. As AI-powered applications continue to grow in complexity, tools like Langfuse are becoming essential for maintaining reliability, controlling costs, and improving user experience.</p><p>In this article, we will explore how Langfuse addresses the operational challenges of building production-ready LLM systems. Through a hands-on document Q&amp;A application, you’ll learn how to trace LLM interactions for debugging, organize and version prompts effectively, monitor application performance, and implement evaluation strategies that bring structure and confidence to AI development workflows.</p><h3>Getting Started</h3><h4>Table of contents</h4><ul><li><a href="#7832">What is Langfuse</a></li><li><a href="#a01b">Components of Langfuse</a></li><li><a href="#25fe">Why Langfuse</a></li><li><a href="#0dbd">Hands-On 1: Introduction to Langfuse Tracing</a></li><li><a href="#6438">Hands-on 2: Building a Document Q&amp;A Tool with LLM Observability</a></li><li><a href="#b759">Setting Up the Environment</a></li><li><a href="#8da9">Add Your Source PDFAdd Your Source PDF</a></li><li><a href="#a14c">Building the Q&amp;A Chain</a></li><li><a href="#ccb6">Tracing Your First Call</a></li><li><a href="#bd36">Session Tracking</a></li><li><a href="#765d">User Tracking</a></li><li><a href="#eeca">Prompt Management</a></li><li><a href="#7b7e">Complete Combined Script</a></li><li><a href="#66d3">Streamlit interface</a></li></ul><h4>What is Langfuse</h4><p>Langfuse is an open-source observability and evaluation platform designed for applications powered by Large Language Models (LLMs). It helps developers monitor, debug, manage, and improve AI workflows by providing complete visibility into how LLM applications behave in production.</p><p>As LLM applications become more complex, debugging issues and understanding model behavior can be challenging due to the non-deterministic nature of AI systems. Langfuse addresses this by enabling teams to trace every stage of an LLM interaction — from prompts and intermediate processing steps to final responses — across both simple and multi-step AI workflows.</p><p>Langfuse seamlessly integrates with modern AI development ecosystems and supports popular frameworks such as LangChain, LlamaIndex, AutoGen, and CrewAI. It is model-agnostic, allowing developers to work with a wide range of providers including OpenAI, Anthropic, Google Gemini, Mistral, and Hugging Face models without additional complexity.</p><h4>Components of Langfuse</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cYmsd2SEYsQagexsUdIKxw.png" /></figure><ul><li><strong>Tracing<br></strong>Tracing is one of the core capabilities of Langfuse. It helps developers monitor the complete lifecycle of an LLM interaction, from the initial prompt to the final response. Langfuse visualizes multi-step AI workflows, making it easier to inspect prompts, outputs, latency, token usage, metadata, and intermediate processing steps. This detailed visibility helps teams debug complex AI pipelines and quickly identify bottlenecks or unexpected behavior.</li><li><strong>Prompt Management<br></strong>Prompt management allows teams to centralize, organize, and version prompts instead of hardcoding them across applications. Developers can maintain prompt history, test different prompt variations, create environment-specific configurations, and safely deploy prompt updates. This improves collaboration and enables faster experimentation without modifying production code directly.</li><li><strong>Evaluation</strong><br>Langfuse provides tools to evaluate the quality and effectiveness of LLM responses. Teams can create automated evaluation pipelines, define custom quality metrics, use LLM-as-a-judge techniques, and compare prompt or model performance over time. This systematic evaluation process helps ensure consistent and reliable AI outputs.</li><li><strong>Observability<br></strong>Observability focuses on monitoring the operational health of AI applications. Langfuse tracks latency, token usage, API costs, errors, and workflow execution in real time. Developers can filter traces, search interactions, analyze dashboards, and receive insights into performance regressions or system issues, helping maintain production reliability.</li><li><strong>Metadata &amp; Tagging<br></strong>Metadata and tagging features help organize and analyze AI interactions more effectively. Developers can attach custom metadata, tags, user information, session identifiers, and version details to traces. This contextual information improves debugging, reporting, analytics, and workflow segmentation.</li><li><strong>Integrations</strong><br>Langfuse integrates seamlessly with popular AI frameworks and model providers, including LangChain, LlamaIndex, OpenAI, Anthropic, Cohere, and more. It offers SDKs for Python and TypeScript, allowing developers to connect Langfuse easily with existing AI stacks and export observability data into their workflows.</li><li><strong>Projects &amp; Access Control<br></strong>Langfuse supports project-based organization and role-based access control for teams. Developers can manage multiple projects, control user permissions, share insights securely, and collaborate efficiently across teams. This makes Langfuse suitable for both startups and enterprise-scale AI deployments.</li></ul><p>By combining tracing, evaluation, prompt management, and observability, Langfuse helps teams:</p><ul><li>Improve debugging efficiency</li><li>Build higher-quality AI applications</li><li>Reduce operational and API costs</li><li>Accelerate development iterations</li><li>Deliver more reliable LLM-powered systems</li></ul><h4>Why Langfuse?</h4><p>Traditional observability tools are not designed for the unpredictable nature of LLM-powered applications. Langfuse addresses several key challenges faced during AI application development and deployment.</p><ul><li><strong>Non-Deterministic Responses</strong><br>LLMs can produce different outputs for the same input, making debugging difficult. Langfuse logs every interaction, including prompts, responses, and intermediate steps, helping developers reproduce and analyze issues effectively.</li><li><strong>Prompt Sensitivity</strong><br>Even small prompt changes can significantly affect model responses. Langfuse provides prompt versioning and performance tracking, making it easier to test and compare prompt variations.</li><li><strong>Complex AI Workflows</strong><br>Modern LLM applications often involve multi-step pipelines, tools, agents, and RAG architectures. Langfuse offers detailed tracing and visual timelines to understand workflow execution and identify bottlenecks or failures quickly.</li><li><strong>Evaluating Response Quality</strong><br>LLM output quality is often subjective. Langfuse supports both automated evaluations and human feedback, enabling teams to measure quality using metrics such as relevance, accuracy, latency, and user satisfaction.</li><li><strong>Cost and Usage Monitoring</strong><br>Since LLM APIs can become expensive at scale, Langfuse tracks token usage, request volume, and operational costs, helping teams optimize resource consumption.</li><li><strong>Limited Production Visibility<br></strong>Without proper observability, improving AI applications becomes challenging. Langfuse provides actionable insights into real-world application behavior, allowing teams to continuously refine prompts, workflows, and models through a more data-driven development process.</li></ul><h3><strong>Hands-On 1: Introduction to Langfuse Tracing</strong></h3><p>Let’s build a small Python application from scratch with uv, calls an Azure OpenAI model, and records the request in Langfuse. By the end, you will have a trace containing an outer application span and a nested LLM generation.</p><h4><strong>Initialize a project with uv</strong></h4><p>Install uv if it is not already available, then create a project:</p><pre>uv init langfuse-basic<br>cd langfuse-basic</pre><p>The command creates the project metadata and a starter Python entry point.</p><h4><strong>Install dependencies</strong></h4><p>Install the Langfuse SDK, the OpenAI client, and python-dotenv:</p><pre>uv add langfuse openai python-dotenv</pre><p>These packages do the following:</p><ul><li><strong>langfuse</strong>: creates and sends observations and traces.</li><li><strong>openai</strong>: provides the OpenAI client.</li><li><strong>python-dotenv</strong>: loads local environment variables from .env.</li></ul><h4><strong>Create the environment file</strong></h4><p>Create .env file in the project root and add your credentials to it:</p><pre># OpenAI<br>OPENAI_API_KEY=sk-proj-hdCPXqIo13xktYjbU9FSkL1VXt4...<br># Langfuse<br>LANGFUSE_SECRET_KEY=&quot;sk-lf-...&quot;<br>LANGFUSE_PUBLIC_KEY=&quot;pk-lf-...&quot;<br>LANGFUSE_BASE_URL=&quot;https://jp.cloud.langfuse.com&quot;<br>LANGFUSE_HOST=&quot;https://cloud.langfuse.com&quot;<br>LANGFUSE_ENVIRONMENT=&quot;development&quot;</pre><p>Get the Langfuse keys from the project settings in <a href="https://cloud.langfuse.com/auth/sign-up">Langfuse</a>. Get the OpenAI values from your <a href="https://platform.openai.com/settings/organization/api-keys">OpenAI portal</a> keys configuration.</p><ul><li><a href="https://cloud.langfuse.com/auth/sign-up">Sign up | Langfuse Cloud</a></li><li><a href="https://platform.openai.com/settings/organization/api-keys">OpenAI Platform</a></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uTM65_0CYkoZH1XHlgqRBw.png" /></figure><h4><strong>Create the simple script</strong></h4><p>Open main.py and load the environment variables before constructing either client:</p><pre>import os<br>from openai import OpenAI<br>from langfuse import Langfuse<br>from dotenv import load_dotenv<br>load_dotenv()<br><br># Initialize Langfuse<br>langfuse = Langfuse(<br>    public_key=os.environ.get(&quot;LANGFUSE_PUBLIC_KEY&quot;),<br>    secret_key=os.environ.get(&quot;LANGFUSE_SECRET_KEY&quot;),<br>    host=os.environ.get(&quot;LANGFUSE_HOST&quot;),<br>)<br><br># Initialize OpenAI client<br>client = OpenAI(<br>    api_key=os.environ.get(&quot;OPENAI_API_KEY&quot;),<br>)<br><br>DEPLOYMENT_NAME = os.environ.get(&quot;OPENAI_MODEL&quot;, &quot;gpt-4o-mini&quot;)</pre><h4><strong>Trace the model call</strong></h4><p>Add one function that wraps the complete request lifecycle: the outer span, the nested generation, the Azure OpenAI call, usage recording, error handling, and trace flushing.</p><pre>def simple_llm_call_with_trace(user_input: str):<br>    with langfuse.start_as_current_observation(<br>        name=&quot;simple-query&quot;,<br>        as_type=&quot;span&quot;,<br>        input=user_input,<br>        metadata={&quot;user_id&quot;: &quot;user-123&quot;, &quot;session_id&quot;: &quot;sess-abc&quot;},<br>    ):<br>        try:<br>            with langfuse.start_as_current_observation(<br>                name=&quot;openai-generation&quot;,<br>                as_type=&quot;generation&quot;,<br>                input=user_input,<br>                model=DEPLOYMENT_NAME,<br>                model_parameters={&quot;temperature&quot;: 0.7, &quot;max_tokens&quot;: 100},<br>                metadata={&quot;prompt_type&quot;: &quot;standard&quot;, &quot;provider&quot;: &quot;openai&quot;},<br>            ):<br>                chat_completion = client.chat.completions.create(<br>                    model=DEPLOYMENT_NAME,<br>                    messages=[{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: user_input}],<br>                    temperature=0.7,<br>                    max_tokens=100,<br>                )<br><br>                response_content = chat_completion.choices[0].message.content<br><br>                langfuse.update_current_generation(<br>                    output=response_content,<br>                    usage_details={<br>                        &quot;input&quot;: chat_completion.usage.prompt_tokens,<br>                        &quot;output&quot;: chat_completion.usage.completion_tokens,<br>                        &quot;total&quot;: chat_completion.usage.total_tokens,<br>                    },<br>                )<br><br>            print(f&quot;LLM Response: {response_content}&quot;)<br>            return response_content<br><br>        except Exception as e:<br>            langfuse.update_current_span(level=&quot;ERROR&quot;, status_message=str(e))<br>            print(f&quot;An error occurred: {e}&quot;)<br>            raise<br><br>        finally:<br>            langfuse.flush()<br><br><br># Example call<br>simple_llm_call_with_trace(&quot;What is the capital of India?&quot;)</pre><p><strong>Understand the code:</strong></p><ul><li>The simple-query span represents the complete application operation and stores the input and request metadata.</li><li>The nested openai-generation observation represents the model call and records the deployment and generation parameters.</li><li>client.chat.completions.create(…) sends the input to Azure OpenAI.</li><li>update_current_generation(…) records the response and input, output, and total token counts.</li><li>The except block marks the current span as ERROR, prints the failure, and re-raises the exception.</li><li>The finally block calls langfuse.flush() so buffered trace data is sent before the short-lived script exits.</li></ul><h4><strong>Run the application</strong></h4><p>Run the script through the project environment:</p><pre>uv run python main.py</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*mgRhC5gTkFrPrm_VScLWlg.png" /></figure><p>You should see an LLM Response line in the terminal. Open the configured Langfuse project and inspect the simple-query trace. Expand the nested openai-generation observation to see the input, output, model parameters, and token usage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WljkHdzWjw3fy2bR0dvLFA.png" /></figure><h3><strong>Hands-on 2: Building a Document Q&amp;A Tool with LLM Observability</strong></h3><p>Now that we’ve covered the fundamentals of Langfuse and LLM observability, let’s put them into practice by building a simple Document Q&amp;A application.</p><p>In this hands-on section, we will build a system that can read and understand information from a PDF and answer questions based on its content. We will then integrate Langfuse to trace the LLM workflow, monitor execution, and gain visibility into what happens behind each query.</p><p>Before writing any code, it helps to understand what we’re building and how the pieces connect. The Document Q&amp;A system has two layers working in parallel: the LangChain RAG pipeline that actually answers questions, and the Langfuse observability layer that watches everything the pipeline does.</p><pre>DATA FLOW<br>User Question → RAG Pipeline (LangChain) → LLM Answer<br>↕<br>Langfuse SDK → Langfuse Cloud → Dashboard<br><br>RAG PIPELINE STEPS<br>PDF Load → Chunk → Embed → Retrieve → Generate</pre><p>The Langfuse CallbackHandler is the bridge between both layers. You attach it to any LangChain chain invocation, and it automatically intercepts every step, records its inputs and outputs, and ships the data to Langfuse asynchronously — adding essentially zero latency to your app.</p><h4>Setting Up the Environment</h4><p>Let’s start by setting up the project environment and installing the dependencies required for the Document Q&amp;A application.</p><p>Initialize the project using uv and install the required packages:</p><pre>uv init document_qa_system<br>cd document_qa_system<br>uv add langfuse langchain langchain-openai langchain-community pypdf python-dotenv</pre><p>Next, create a free account on <a href="https://cloud.langfuse.com/auth/sign-up">Langfuse Cloud</a>. After signing in, create a new organization and project. From the project settings, navigate to the API keys section and copy the Public Key and Secret Key. You’ll also need an OpenAI API key, which you can obtain from <a href="https://platform.openai.com/settings/organization/api-keys">OpenAI Platform</a>.</p><p>To keep these credentials secure and separate from your source code, create a .env file in the project directory:</p><pre># OpenAI<br>OPENAI_API_KEY=sk-proj-hdCPXqIo13xktYjbU9FSkL1VXt4...<br># Langfuse<br>LANGFUSE_SECRET_KEY=&quot;sk-lf-...&quot;<br>LANGFUSE_PUBLIC_KEY=&quot;pk-lf-...&quot;<br>LANGFUSE_BASE_URL=&quot;https://jp.cloud.langfuse.com&quot;<br>LANGFUSE_HOST=&quot;https://cloud.langfuse.com&quot;<br>LANGFUSE_ENVIRONMENT=&quot;development&quot;</pre><h4>Add Your Source PDF</h4><p>For this hands-on exercise, you’ll need a PDF document to use as the knowledge source for the Q&amp;A application. In this tutorial, we use the <a href="https://docs.aws.amazon.com/pdfs/whitepapers/latest/navigating-security-landscape-genai/navigating-security-landscape-genai.pdf">AWS<strong> </strong>Generative AI Security Landscape whitepaper</a> as the sample document. Download the sample PDF below and place it in your project directory.</p><h4><strong>Building the Q&amp;A Chain</strong></h4><p>The core pipeline is a Retrieval-Augmented Generation (RAG) chain. Instead of asking the LLM to recall facts from training data, we give it relevant document chunks at query time. This keeps answers grounded and reduces hallucination.</p><p>Create build_chain.py — a reusable module that builds and returns the chain.</p><pre># build_chain.py<br>from langchain_community.document_loaders import PyPDFLoader<br>from langchain_text_splitters import RecursiveCharacterTextSplitter<br>from langchain_openai import OpenAIEmbeddings, ChatOpenAI<br>from langchain_core.vectorstores import InMemoryVectorStore<br>from langchain_core.prompts import ChatPromptTemplate<br>from langchain_core.runnables import RunnablePassthrough<br>from langchain_core.output_parsers import StrOutputParser<br>from dotenv import load_dotenv<br>load_dotenv()<br><br><br>def build_qa_chain():<br>    &quot;&quot;&quot;<br>    Builds a retrieval-augmented Q&amp;A chain over a PDF document.<br><br>    Steps:<br>      1. Load the PDF and extract its pages.<br>      2. Split pages into overlapping chunks for better context boundaries.<br>      3. Embed chunks and store them in an in-memory vector store.<br>      4. Build a ChatOpenAI LCEL chain that retrieves relevant chunks and generates answers.<br>    &quot;&quot;&quot;<br><br>    # ── STEP 1: Load PDF ──────────────────────────────────────────────<br>    # PyPDFLoader reads each page of the PDF as a separate Document object.<br>    loader = PyPDFLoader(&quot;navigating-security-landscape-genai.pdf&quot;)<br>    docs = loader.load()<br><br>    # ── STEP 2: Split into chunks ──────────────────────────────────────<br>    # chunk_size=1000 means each chunk is at most 1000 characters.<br>    # chunk_overlap=200 means adjacent chunks share 200 characters —<br>    # this prevents important context from being cut at a boundary.<br>    text_splitter = RecursiveCharacterTextSplitter(<br>        chunk_size=1000,<br>        chunk_overlap=200<br>    )<br>    splits = text_splitter.split_documents(docs)<br><br>    # ── STEP 3: Embed and store ────────────────────────────────────────<br>    # OpenAIEmbeddings converts each chunk into a vector using text-embedding-ada-002.<br>    embeddings = OpenAIEmbeddings()<br>    vectorstore = InMemoryVectorStore.from_documents(<br>        documents=splits,<br>        embedding=embeddings<br>    )<br><br>    # ── STEP 4: Build the QA chain using LCEL ─────────────────────────<br>    # RunnablePassthrough.assign(context=...) fetches and formats docs.<br>    # A second assign(answer=...) pipes the filled prompt through the LLM.<br>    # Result dict has keys: input, context, answer — result[&#39;answer&#39;] works.<br>    llm = ChatOpenAI(model=&quot;gpt-4o-mini&quot;)<br>    retriever = vectorstore.as_retriever()<br><br>    prompt = ChatPromptTemplate.from_messages([<br>        (&quot;system&quot;, (<br>            &quot;Use the given context to answer the question. &quot;<br>            &quot;If you don&#39;t know the answer, say you don&#39;t know. &quot;<br>            &quot;Keep the answer concise.\n\n{context}&quot;<br>        )),<br>        (&quot;human&quot;, &quot;{input}&quot;),<br>    ])<br><br>    qa_chain = (<br>        RunnablePassthrough.assign(<br>            context=lambda x: &quot;\n\n&quot;.join(<br>                doc.page_content for doc in retriever.invoke(x[&quot;input&quot;])<br>            )<br>        )<br>        | RunnablePassthrough.assign(<br>            answer=prompt | llm | StrOutputParser()<br>        )<br>    )<br><br>    return qa_chain</pre><p><strong>Why chunk with overlap? <br></strong>Imagine a sentence that straddles two chunks: “…the attacker exploits | the context window…” Without overlap, the retriever might return either chunk but not both, losing the sentence’s meaning. A 200-character overlap ensures that boundaries don’t silently destroy context.</p><blockquote>For production systems, swap InMemoryVectorStore with a persistent vector database like Pinecone, Weaviate, or pgvector. The rest of the chain code stays identical — LangChain abstracts the retrieval interface.</blockquote><p><strong>Why LCEL instead of legacy chains?<br></strong>LangChain 1.x removed langchain.chains in favour of LCEL (LangChain Expression Language). The pipe operator composes runnables directly. RunnablePassthrough.assign() adds keys to the passing dict, so the output still looks like {“input”: …, “context”: …, “answer”: …} — exactly the same as the old create_retrieval_chain.</p><h4><strong>Tracing Your First Call</strong></h4><p>Now for the observability layer. The key insight: you don’t need to rewrite your chain. You add a single CallbackHandler to the invoke() call, and Langfuse automatically instruments every step of your chain’s execution.</p><p>Create trace_demo.py and add the following code to it.</p><pre># trace_demo.py<br>from build_chain import build_qa_chain<br>from langfuse.langchain import CallbackHandler<br><br># Build the Q&amp;A chain (loads PDF, embeds chunks, wires up retriever)<br>qa_chain = build_qa_chain()<br><br># Instantiate the Langfuse handler.<br># It auto-reads LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY, LANGFUSE_HOST from .env.<br>langfuse_handler = CallbackHandler()<br><br># Run the chain with the callback attached.<br># config={&quot;callbacks&quot;: [langfuse_handler]} is all it takes —<br># LangChain passes the handler into every step of the chain automatically.<br>response = qa_chain.invoke(<br>    {&quot;input&quot;: &quot;What are the main security risks discussed?&quot;},<br>    config={&quot;callbacks&quot;: [langfuse_handler]}<br>)<br><br>print(f&quot;Answer: {response[&#39;answer&#39;]}&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u3kMcMAdlU2Qvq4UJgC8XA.png" /></figure><p>After running this, open your Langfuse dashboard and click Tracing in the sidebar. You’ll see a trace with a complete execution tree:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T4YTxmZXeRa9BRXoriJ7Lg.png" /></figure><p>The trace shows exactly where time was spent. In this case, 82% of the time was the LLM call — retrieval itself was fast. If you had a slow retrieval step, you’d see it here immediately, without any guesswork.</p><blockquote>The Langfuse handler sends data asynchronously after invoke() returns, so it doesn’t add latency to your users’ responses. If your process exits immediately after the call, add langfuse.flush() at the end to ensure all data is shipped before exit.</blockquote><h4><strong>Session Tracking</strong></h4><p>A single trace captures one question-answer cycle. But real users have conversations — they ask a follow-up, then another, all related to the same intent. A session groups multiple traces into a single conversation thread. Sessions are set via a single metadata field langfuse_session_id.</p><p>Create sessions_demo.py file add the following code to it:</p><pre># sessions_demo.py<br>from build_chain import build_qa_chain<br>from langfuse.langchain import CallbackHandler<br><br>qa_chain = build_qa_chain()<br>langfuse_handler = CallbackHandler()<br><br># ── SESSION 1: User asking about security risks ────────────────────<br># Both questions below share the same session_id, so Langfuse<br># groups them into one conversation thread in the dashboard.<br><br>response1 = qa_chain.invoke(<br>    {&quot;input&quot;: &quot;What are the main security risks discussed?&quot;},<br>    config={<br>        &quot;callbacks&quot;: [langfuse_handler],<br>        &quot;metadata&quot;: {&quot;langfuse_session_id&quot;: &quot;conversation-001&quot;}<br>    }<br>)<br>print(f&quot;Q1: {response1[&#39;answer&#39;]}\n&quot;)<br><br># Follow-up question — same session, continues the thread<br>response2 = qa_chain.invoke(<br>    {&quot;input&quot;: &quot;How can these risks be mitigated?&quot;},<br>    config={<br>        &quot;callbacks&quot;: [langfuse_handler],<br>        &quot;metadata&quot;: {&quot;langfuse_session_id&quot;: &quot;conversation-001&quot;}<br>    }<br>)<br>print(f&quot;Q2: {response2[&#39;answer&#39;]}\n&quot;)<br><br># ── SESSION 2: Different user, different topic ─────────────────────<br># A new session_id starts a completely separate thread in the dashboard.<br><br>response3 = qa_chain.invoke(<br>    {&quot;input&quot;: &quot;What AI applications are mentioned in the document?&quot;},<br>    config={<br>        &quot;callbacks&quot;: [langfuse_handler],<br>        &quot;metadata&quot;: {&quot;langfuse_session_id&quot;: &quot;conversation-002&quot;}<br>    }<br>)<br>print(f&quot;Q3: {response3[&#39;answer&#39;]}\n&quot;)<br><br>response4 = qa_chain.invoke(<br>    {&quot;input&quot;: &quot;What are the benefits of these applications?&quot;},<br>    config={<br>        &quot;callbacks&quot;: [langfuse_handler],<br>        &quot;metadata&quot;: {&quot;langfuse_session_id&quot;: &quot;conversation-002&quot;}<br>    }<br>)<br>print(f&quot;Q4: {response4[&#39;answer&#39;]}\n&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o8r0qzme0xIpkuZZAVvADQ.png" /></figure><p>Navigate to Sessions in your Langfuse sidebar. You will see two conversation threads: conversation-001 with 2 traces and its combined cost, and conversation-002 with 2 traces.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q1d_TFe2muYI7Svgo8u79w.png" /></figure><h4><strong>User Tracking</strong></h4><p>Sessions tell you about conversations. User IDs tell you about people. When you attribute traces to user IDs, Langfuse lets you answer questions like: “Which user is consuming the most tokens this month?”</p><p>Add langfuse_user_id alongside the session ID:</p><pre># user_tracking.py<br>from build_chain import build_qa_chain<br>from langfuse.langchain import CallbackHandler<br><br>qa_chain = build_qa_chain()<br>langfuse_handler = CallbackHandler()<br><br># In a real app, user_id would come from your auth system (JWT claims, session, etc.)<br>response = qa_chain.invoke(<br>    {&quot;input&quot;: &quot;What security controls are recommended for AI systems?&quot;},<br>    config={<br>        &quot;callbacks&quot;: [langfuse_handler],<br>        &quot;metadata&quot;: {<br>            &quot;langfuse_session_id&quot;: &quot;conversation-003&quot;,<br>            &quot;langfuse_user_id&quot;:    &quot;user-12345&quot;    # ← attribute to a specific user<br>        }<br>    }<br>)<br>print(f&quot;Response: {response[&#39;answer&#39;]}&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f-cXb9HzF7_l6PQbFm06NQ.png" /></figure><p>Go to Users in the Langfuse sidebar. Click user-12345 to see their complete history: all sessions, all traces, cumulative token spend, and a timeline of every interaction.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3WX_MTVw5vNd8JlKHi18rg.png" /></figure><h4><strong>Prompt Management</strong></h4><p>So far, the prompt is inside your code. Every change requires a code edit, a git commit, and a redeployment. Langfuse’s prompt management system solves this by treating prompts as versioned data, not code.</p><p><strong>Step 1 — Create the prompt via SDK</strong></p><pre># create_prompt.py<br>from langfuse import Langfuse<br>from dotenv import load_dotenv<br>load_dotenv()<br><br>langfuse = Langfuse()<br><br># Create (or version) the managed prompt.<br># Variables in {{double braces}} are replaced at runtime via .compile().<br># If a prompt with this name already exists, calling create_prompt() again<br># creates a NEW version — it never overwrites the previous one.<br>langfuse.create_prompt(<br>    name=&quot;document-qa-prompt&quot;,<br>    type=&quot;chat&quot;,<br>    prompt=[<br>        {<br>            &quot;role&quot;: &quot;system&quot;,<br>            &quot;content&quot;: (<br>                &quot;Use the given context to answer the question. &quot;<br>                &quot;If you don&#39;t know the answer, say you don&#39;t know. &quot;<br>                &quot;Keep the answer concise.\n\n{{context}}&quot;<br>            )<br>        },<br>        {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;{{input}}&quot;}<br>    ],<br>    config={<br>        &quot;model&quot;: &quot;gpt-4o-mini&quot;,<br>        &quot;temperature&quot;: 0.0<br>    },<br>    labels=[&quot;production&quot;]   # mark this version as production<br>)<br><br>print(&quot;✓ Prompt &#39;document-qa-prompt&#39; v1 created and labeled production.&quot;)</pre><p>You can also create and edit prompts through the Langfuse UI Playground— a visual editor where you can test variable substitution and compare outputs across versions before deploying.</p><p><strong>Step 2 — Use the managed prompt in your chain</strong></p><pre># managed_prompt_demo.py<br>from langchain_community.document_loaders import PyPDFLoader<br>from langchain_text_splitters import RecursiveCharacterTextSplitter<br>from langchain_openai import OpenAIEmbeddings, ChatOpenAI<br>from langchain_core.vectorstores import InMemoryVectorStore<br>from langchain_core.prompts import ChatPromptTemplate<br>from langchain_core.runnables import RunnablePassthrough<br>from langchain_core.output_parsers import StrOutputParser<br>from langfuse import Langfuse<br>from langfuse.langchain import CallbackHandler<br>from dotenv import load_dotenv<br><br>load_dotenv()<br><br>langfuse = Langfuse()<br><br># ── Fetch the prompt labeled &quot;production&quot; ─────────────────────────<br># Langfuse caches this client-side, so repeated calls don&#39;t add latency.<br># To fetch a specific version:  get_prompt(&quot;...&quot;, version=2, type=&quot;chat&quot;)<br># To fetch staging:             get_prompt(&quot;...&quot;, label=&quot;staging&quot;, type=&quot;chat&quot;)<br>managed_prompt = langfuse.get_prompt(&quot;document-qa-prompt&quot;, type=&quot;chat&quot;)<br><br># ── Compile the template into LangChain messages ──────────────────<br># compile() substitutes {{context}} and {{input}} with the provided values.<br>compiled_messages = managed_prompt.compile(<br>    context=&quot;{context}&quot;,<br>    input=&quot;{input}&quot;<br>)<br><br># Convert compiled messages into a LangChain ChatPromptTemplate<br>prompt = ChatPromptTemplate.from_messages(<br>    [(m[&quot;role&quot;], m[&quot;content&quot;]) for m in compiled_messages]<br>)<br><br># ── Build the chain using the managed prompt ──────────────────────<br>loader = PyPDFLoader(&quot;navigating-security-landscape-genai.pdf&quot;)<br>splits = RecursiveCharacterTextSplitter(<br>    chunk_size=1000, chunk_overlap=200<br>).split_documents(loader.load())<br><br>vectorstore = InMemoryVectorStore.from_documents(<br>    documents=splits,<br>    embedding=OpenAIEmbeddings()<br>)<br><br>llm = ChatOpenAI(model=&quot;gpt-4o-mini&quot;)<br>retriever = vectorstore.as_retriever()<br><br>qa_chain = (<br>    RunnablePassthrough.assign(<br>        context=lambda x: &quot;\n\n&quot;.join(<br>            doc.page_content for doc in retriever.invoke(x[&quot;input&quot;])<br>        )<br>    )<br>    | RunnablePassthrough.assign(<br>        answer=prompt | llm | StrOutputParser()<br>    )<br>)<br><br># ── Run with Langfuse tracing ──────────────────────────────────────<br>langfuse_handler = CallbackHandler()<br><br>response = qa_chain.invoke(<br>    {&quot;input&quot;: &quot;What are the recommended controls for AI security?&quot;},<br>    config={<br>        &quot;callbacks&quot;: [langfuse_handler],<br>        &quot;metadata&quot;: {<br>            &quot;langfuse_session_id&quot;: &quot;conversation-004&quot;,<br>            &quot;langfuse_user_id&quot;: &quot;user-12345&quot;,<br>            &quot;prompt_name&quot;: &quot;document-qa-prompt&quot;,<br>            &quot;prompt_version&quot;: managed_prompt.version<br>        }<br>    }<br>)<br><br>print(f&quot;[Prompt v{managed_prompt.version}] {response[&#39;answer&#39;]}&quot;)</pre><h4><strong>The prompt update workflow</strong></h4><ul><li><strong>Edit the prompt in Langfuse UI</strong> (or call create_prompt() again) — Langfuse automatically creates a new version. The old version is preserved.</li><li><strong>Test the new version in staging</strong>— assign it the staging label and point your staging environment at it via get_prompt(…, label=”staging”).</li><li><strong>Move the production label to the new version</strong> — your production app picks up the new prompt on the next get_prompt() call. Zero code change. Zero deployment.</li><li><strong>Rollback in one click if needed</strong> — move the production label back to the previous version.</li></ul><h4><strong>Complete Combined Script</strong></h4><p>Here is everything from this guide combined into a single runnable script langfuse_qa_complete.py. It demonstrates all capabilities such as RAG chain construction, prompt management, basic tracing, session grouping, user attribution, and managed prompts.</p><pre>&quot;&quot;&quot;<br>Langfuse Document Q&amp;A — Complete Progressive Demonstration Script<br><br>Run:  uv run python langfuse_qa_complete.py<br>Pre-requisite: .env with OPENAI_API_KEY, LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY<br>               and navigating-security-landscape-genai.pdf in the same directory.<br>&quot;&quot;&quot;<br><br>from langchain_community.document_loaders import PyPDFLoader<br>from langchain_text_splitters import RecursiveCharacterTextSplitter<br>from langchain_openai import OpenAIEmbeddings, ChatOpenAI<br>from langchain_core.vectorstores import InMemoryVectorStore<br>from langchain_core.prompts import ChatPromptTemplate<br>from langchain_core.runnables import RunnablePassthrough<br>from langchain_core.output_parsers import StrOutputParser<br><br>from langfuse import Langfuse<br>from langfuse.langchain import CallbackHandler<br><br>from dotenv import load_dotenv<br>load_dotenv()<br>print(&quot;✓ Environment loaded\n&quot;)<br><br><br># ═══════════════════════════════════════════════════════════════════════<br># Defines a reusable function that loads the PDF, splits it into chunks,<br># embeds them, and wires up a retrieval-augmented generation chain.<br># The prompt is accepted as a parameter so it can be swapped between<br># the inline fallback and the managed Langfuse prompt.<br># ═══════════════════════════════════════════════════════════════════════<br><br>def build_qa_chain(prompt):<br>    &quot;&quot;&quot;<br>    Builds a retrieval-augmented Q&amp;A chain over a PDF document.<br>    Equivalent to the standalone build_chain.py, but accepts the<br>    prompt as an argument so it works with both managed and inline prompts.<br>    &quot;&quot;&quot;<br><br>    # ── Load PDF ──────────────────────────────────────────────────<br>    loader = PyPDFLoader(&quot;navigating-security-landscape-genai.pdf&quot;)<br>    docs = loader.load()<br>    print(f&quot;  Loaded {len(docs)} pages from PDF&quot;)<br><br>    text_splitter = RecursiveCharacterTextSplitter(<br>        chunk_size=1000,<br>        chunk_overlap=200<br>    )<br>    splits = text_splitter.split_documents(docs)<br>    print(f&quot;  Split into {len(splits)} chunks&quot;)<br><br>    # ── Embed and store ────────────────────────────────────────────<br>    embeddings = OpenAIEmbeddings()<br>    vectorstore = InMemoryVectorStore.from_documents(<br>        documents=splits,<br>        embedding=embeddings<br>    )<br>    print(&quot;  Vector store ready&quot;)<br><br>    # ── Wire up the chain using LCEL ──────────────────────────────<br>    llm = ChatOpenAI(model=&quot;gpt-4o-mini&quot;)<br>    retriever = vectorstore.as_retriever()<br><br>    qa_chain = (<br>        RunnablePassthrough.assign(<br>            context=lambda x: &quot;\n\n&quot;.join(<br>                doc.page_content for doc in retriever.invoke(x[&quot;input&quot;])<br>            )<br>        )<br>        | RunnablePassthrough.assign(<br>            answer=prompt | llm | StrOutputParser()<br>        )<br>    )<br>    return qa_chain<br><br><br># ═══════════════════════════════════════════════════════════════════════<br>#   Register the prompt as a versioned artifact in Langfuse.<br>#   Calling create_prompt() again creates a NEW version — it never<br>#   overwrites the previous one, giving you a full history.<br># ═══════════════════════════════════════════════════════════════════════<br><br>langfuse = Langfuse()<br><br># ── register (or version) the prompt ──────────<br>print(&quot;Registering managed prompt in Langfuse ...&quot;)<br>langfuse.create_prompt(<br>    name=&quot;document-qa-prompt&quot;,<br>    type=&quot;chat&quot;,<br>    prompt=[<br>        {<br>            &quot;role&quot;: &quot;system&quot;,<br>            &quot;content&quot;: (<br>                &quot;Use the given context to answer the question. &quot;<br>                &quot;If you don&#39;t know the answer, say you don&#39;t know. &quot;<br>                &quot;Keep the answer concise.\n\n{{context}}&quot;<br>            )<br>        },<br>        {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;{{input}}&quot;}<br>    ],<br>    config={<br>        &quot;model&quot;: &quot;gpt-4&quot;,<br>        &quot;temperature&quot;: 0.0<br>    },<br>    labels=[&quot;production&quot;]<br>)<br>print(&quot;  ✓ Prompt &#39;document-qa-prompt&#39; created/versioned and labeled production.\n&quot;)<br><br># ── fetch the production prompt at runtime ──<br># Langfuse caches this client-side, so repeated calls don&#39;t add latency.<br># To fetch a specific version: get_prompt(&quot;...&quot;, version=2, type=&quot;chat&quot;)<br># To fetch staging:            get_prompt(&quot;...&quot;, label=&quot;staging&quot;, type=&quot;chat&quot;)<br>try:<br>    managed = langfuse.get_prompt(&quot;document-qa-prompt&quot;, type=&quot;chat&quot;)<br><br>    compiled = managed.compile(context=&quot;{context}&quot;, input=&quot;{input}&quot;)<br><br>    prompt = ChatPromptTemplate.from_messages(<br>        [(m[&quot;role&quot;], m[&quot;content&quot;]) for m in compiled]<br>    )<br>    prompt_version = managed.version<br>    print(f&quot;  Using managed prompt v{prompt_version} from Langfuse&quot;)<br><br>except Exception as e:<br>    print(f&quot;  Prompt not found in Langfuse ({e}). Using inline fallback.&quot;)<br>    prompt = ChatPromptTemplate.from_messages([<br>        (&quot;system&quot;, (<br>            &quot;Use the given context to answer the question. &quot;<br>            &quot;If you don&#39;t know the answer, say you don&#39;t know. &quot;<br>            &quot;Keep the answer concise.\n\n{context}&quot;<br>        )),<br>        (&quot;human&quot;, &quot;{input}&quot;),<br>    ])<br>    prompt_version = &quot;inline&quot;<br><br><br># ═══════════════════════════════════════════════════════════════════════<br># BUILD THE CHAIN with the resolved prompt<br># ═══════════════════════════════════════════════════════════════════════<br><br>print(&quot;\nBuilding RAG chain ...&quot;)<br>qa_chain = build_qa_chain(prompt)<br>print(&quot;  Chain ready\n&quot;)<br><br><br># ═══════════════════════════════════════════════════════════════════════<br># One handler instance is shared across all invoke() calls.<br># ═══════════════════════════════════════════════════════════════════════<br><br>langfuse_handler = CallbackHandler()<br><br><br>def ask(question, session_id, user_id=None, label=&quot;&quot;):<br>    &quot;&quot;&quot;<br>    Runs the chain with full Langfuse instrumentation.<br>    This helper pattern appears across trace_demo.py, sessions_demo.py,<br>    user_tracking.py, and managed_prompt_demo.py — unified here.<br>    &quot;&quot;&quot;<br>    metadata = {<br>        &quot;langfuse_session_id&quot;: session_id,<br>        &quot;prompt_version&quot;: prompt_version,<br>    }<br>    if user_id:<br>        metadata[&quot;langfuse_user_id&quot;] = user_id<br><br>    result = qa_chain.invoke(<br>        {&quot;input&quot;: question},<br>        config={<br>            &quot;callbacks&quot;: [langfuse_handler],<br>            &quot;metadata&quot;: metadata<br>        }<br>    )<br><br>    tag = f&quot;[{label}] &quot; if label else &quot;&quot;<br>    print(f&quot;{tag}Q: {question}&quot;)<br>    print(f&quot;{tag}A: {result[&#39;answer&#39;]}&quot;)<br>    print()<br>    return result<br><br><br># ═══════════════════════════════════════════════════════════════════════<br># A single question — creates one trace in the Langfuse dashboard.<br># Open Langfuse → Tracing to see the full execution tree with latency<br># and token counts for every step (retrieval, prompt, LLM call).<br># ═══════════════════════════════════════════════════════════════════════<br><br>print(&quot;═&quot; * 60)<br>print(&quot;DEMO 1: Basic Trace&quot;)<br>print(&quot;═&quot; * 60)<br><br>ask(<br>    question=&quot;What are the main security risks discussed?&quot;,<br>    session_id=&quot;demo-basic-001&quot;,<br>    label=&quot;Basic&quot;<br>)<br><br><br># ═══════════════════════════════════════════════════════════════════════<br># Traces sharing the same langfuse_session_id are grouped into one<br># conversation in Langfuse → Sessions, showing the full dialogue<br># with combined cost and all retrieved document chunks in order.<br># ═══════════════════════════════════════════════════════════════════════<br><br>print(&quot;═&quot; * 60)<br>print(&quot;DEMO 2: Session Tracking&quot;)<br>print(&quot;═&quot; * 60)<br><br># Session A: security-focused conversation<br>ask(&quot;What are the main security risks discussed?&quot;,<br>    session_id=&quot;session-security-001&quot;, label=&quot;Session-A, Turn-1&quot;)<br><br>ask(&quot;How can these risks be mitigated?&quot;,<br>    session_id=&quot;session-security-001&quot;, label=&quot;Session-A, Turn-2&quot;)<br><br># Session B: different topic — starts a completely separate thread<br>ask(&quot;What AI applications are mentioned in the document?&quot;,<br>    session_id=&quot;session-apps-002&quot;, label=&quot;Session-B, Turn-1&quot;)<br><br>ask(&quot;What are the benefits of these applications?&quot;,<br>    session_id=&quot;session-apps-002&quot;, label=&quot;Session-B, Turn-2&quot;)<br><br><br># ═══════════════════════════════════════════════════════════════════════<br># In Langfuse → Users, each user gets their own profile with cumulative<br># token spend, session history, and a timeline of every interaction.<br># In production, user_id comes from your auth system (JWT claims, etc.).<br># ═══════════════════════════════════════════════════════════════════════<br><br>print(&quot;═&quot; * 60)<br>print(&quot;DEMO 3: User Tracking&quot;)<br>print(&quot;═&quot; * 60)<br><br>ask(&quot;What security controls are recommended for AI systems?&quot;,<br>    session_id=&quot;session-user-alice&quot;, user_id=&quot;user-alice&quot;, label=&quot;Alice&quot;)<br><br>ask(&quot;What does the document say about data classification?&quot;,<br>    session_id=&quot;session-user-bob&quot;, user_id=&quot;user-bob&quot;, label=&quot;Bob&quot;)<br><br>print(&quot;═&quot; * 60)<br>print(f&quot;DEMO 4: Managed Prompt v{prompt_version} Tracing&quot;)<br>print(&quot;═&quot; * 60)<br><br>ask(<br>    question=&quot;What are the recommended controls for AI security?&quot;,<br>    session_id=&quot;conversation-004&quot;,<br>    user_id=&quot;user-12345&quot;,<br>    label=f&quot;Prompt v{prompt_version}&quot;<br>)<br><br><br># ═══════════════════════════════════════════════════════════════════════<br># The Langfuse handler ships trace data asynchronously after invoke().<br># Call flush() before the process exits to ensure nothing is dropped.<br># ═══════════════════════════════════════════════════════════════════════<br><br>langfuse.flush()<br>print(&quot;✓ All traces flushed to Langfuse.&quot;)<br>print(&quot;  Open https://cloud.langfuse.com → Tracing to see your results.&quot;)</pre><p><strong>Expected Output:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A64U9Y2LEqwvkwgnDhsklQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GRz3Jgxh9m6N15_pGc0yOA.png" /></figure><h4>Streamlit interface</h4><p>For a better user experience, we can add a simple Streamlit interface on top of the Document Q&amp;A application. This provides a lightweight web-based UI for uploading documents, entering questions, and viewing the generated answers.</p><p>The complete Streamlit implementation is shown below.</p><p>Create a streamlit_app.py and add the following code to it.</p><pre>&quot;&quot;&quot;<br>Streamlit UI for the Langfuse Document Q&amp;A system.<br><br>Run:  uv run streamlit run streamlit_app.py<br>Pre-requisite: .env with OPENAI_API_KEY, LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY<br>               navigating-security-landscape-genai.pdf in the same directory (or upload one).<br>&quot;&quot;&quot;<br><br>import os<br>import uuid<br>import tempfile<br><br>import streamlit as st<br>from dotenv import load_dotenv<br><br>from langchain_community.document_loaders import PyPDFLoader<br>from langchain_text_splitters import RecursiveCharacterTextSplitter<br>from langchain_openai import OpenAIEmbeddings, ChatOpenAI<br>from langchain_core.vectorstores import InMemoryVectorStore<br>from langchain_core.prompts import ChatPromptTemplate<br>from langchain_core.runnables import RunnablePassthrough<br>from langchain_core.output_parsers import StrOutputParser<br><br>from langfuse import Langfuse<br>from langfuse.langchain import CallbackHandler<br><br>load_dotenv()<br><br><br># ── Cached resources (built once per Streamlit process) ──────────────────────<br><br>@st.cache_resource<br>def get_langfuse() -&gt; Langfuse:<br>    return Langfuse()<br><br><br>@st.cache_resource<br>def get_managed_prompt():<br>    &quot;&quot;&quot;Fetch the production prompt from Langfuse; fall back to an inline one.&quot;&quot;&quot;<br>    lf = get_langfuse()<br>    try:<br>        lf.create_prompt(<br>            name=&quot;document-qa-prompt&quot;,<br>            type=&quot;chat&quot;,<br>            prompt=[<br>                {<br>                    &quot;role&quot;: &quot;system&quot;,<br>                    &quot;content&quot;: (<br>                        &quot;Use the given context to answer the question. &quot;<br>                        &quot;If you don&#39;t know the answer, say you don&#39;t know. &quot;<br>                        &quot;Keep the answer concise.\n\n{{context}}&quot;<br>                    ),<br>                },<br>                {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;{{input}}&quot;},<br>            ],<br>            config={&quot;model&quot;: &quot;gpt-4o-mini&quot;, &quot;temperature&quot;: 0.0},<br>            labels=[&quot;production&quot;],<br>        )<br>        managed = lf.get_prompt(&quot;document-qa-prompt&quot;, type=&quot;chat&quot;)<br>        compiled = managed.compile(context=&quot;{context}&quot;, input=&quot;{input}&quot;)<br>        prompt = ChatPromptTemplate.from_messages(<br>            [(m[&quot;role&quot;], m[&quot;content&quot;]) for m in compiled]<br>        )<br>        return prompt, managed.version<br>    except Exception:<br>        prompt = ChatPromptTemplate.from_messages([<br>            (<br>                &quot;system&quot;,<br>                &quot;Use the given context to answer the question. &quot;<br>                &quot;If you don&#39;t know the answer, say you don&#39;t know. &quot;<br>                &quot;Keep the answer concise.\n\n{context}&quot;,<br>            ),<br>            (&quot;human&quot;, &quot;{input}&quot;),<br>        ])<br>        return prompt, &quot;inline&quot;<br><br><br>@st.cache_resource<br>def build_chain(pdf_path: str):<br>    &quot;&quot;&quot;Load PDF, embed chunks, and return the RAG chain.&quot;&quot;&quot;<br>    loader = PyPDFLoader(pdf_path)<br>    docs = loader.load()<br><br>    splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)<br>    splits = splitter.split_documents(docs)<br><br>    embeddings = OpenAIEmbeddings()<br>    vectorstore = InMemoryVectorStore.from_documents(documents=splits, embedding=embeddings)<br>    retriever = vectorstore.as_retriever()<br><br>    prompt, _ = get_managed_prompt()<br>    llm = ChatOpenAI(model=&quot;gpt-4o-mini&quot;)<br><br>    chain = (<br>        RunnablePassthrough.assign(<br>            context=lambda x: &quot;\n\n&quot;.join(<br>                doc.page_content for doc in retriever.invoke(x[&quot;input&quot;])<br>            )<br>        )<br>        | RunnablePassthrough.assign(answer=prompt | llm | StrOutputParser())<br>    )<br>    return chain, len(splits)<br><br><br># ── Helpers ───────────────────────────────────────────────────────────────────<br><br>def _render_trace_links(links: dict):<br>    &quot;&quot;&quot;Render Trace / Session / User deep-links as a small expander.&quot;&quot;&quot;<br>    with st.expander(&quot;🔍 View in Langfuse&quot;, expanded=False):<br>        cols = st.columns(len(links))<br>        icons = {&quot;Trace&quot;: &quot;🔗&quot;, &quot;Session&quot;: &quot;💬&quot;, &quot;User&quot;: &quot;👤&quot;}<br>        for col, (label, url) in zip(cols, links.items()):<br>            col.link_button(f&quot;{icons.get(label, &#39;&#39;)} {label}&quot;, url, use_container_width=True)<br><br><br># ── Page config ───────────────────────────────────────────────────────────────<br><br>st.set_page_config(<br>    page_title=&quot;Document Q&amp;A&quot;,<br>    page_icon=&quot;📄&quot;,<br>    layout=&quot;wide&quot;,<br>)<br><br>st.title(&quot;📄 Document Q&amp;A&quot;)<br>st.caption(&quot;Powered by LangChain · OpenAI · Langfuse&quot;)<br><br># ── Sidebar ───────────────────────────────────────────────────────────────────<br><br>with st.sidebar:<br>    st.header(&quot;Settings&quot;)<br><br>    # PDF source<br>    st.subheader(&quot;Document&quot;)<br>    default_pdf = os.path.join(os.path.dirname(__file__), &quot;navigating-security-landscape-genai.pdf&quot;)<br>    uploaded_file = st.file_uploader(&quot;Upload a PDF (optional)&quot;, type=&quot;pdf&quot;)<br><br>    if uploaded_file:<br>        # Write uploaded bytes to a temp file so PyPDFLoader can read it<br>        suffix = f&quot;_{uploaded_file.name}&quot;<br>        tmp = tempfile.NamedTemporaryFile(delete=False, suffix=suffix)<br>        tmp.write(uploaded_file.read())<br>        tmp.flush()<br>        pdf_path = tmp.name<br>        st.success(f&quot;Using uploaded: {uploaded_file.name}&quot;)<br>    elif os.path.exists(default_pdf):<br>        pdf_path = default_pdf<br>        st.info(&quot;Using default PDF&quot;)<br>    else:<br>        pdf_path = None<br>        st.warning(&quot;No PDF found. Upload one to get started.&quot;)<br><br>    st.divider()<br><br>    # User identity<br>    st.subheader(&quot;Identity&quot;)<br>    user_id = st.text_input(<br>        &quot;User ID&quot;,<br>        value=st.session_state.get(&quot;user_id&quot;, f&quot;user-{uuid.uuid4().hex[:6]}&quot;),<br>        help=&quot;Identifies you in Langfuse → Users&quot;,<br>    )<br>    st.session_state[&quot;user_id&quot;] = user_id<br><br>    # Session controls<br>    st.subheader(&quot;Session&quot;)<br>    if &quot;session_id&quot; not in st.session_state:<br>        st.session_state[&quot;session_id&quot;] = f&quot;session-{uuid.uuid4().hex[:8]}&quot;<br><br>    st.code(st.session_state[&quot;session_id&quot;], language=None)<br><br>    if st.button(&quot;New session&quot;, use_container_width=True):<br>        st.session_state[&quot;session_id&quot;] = f&quot;session-{uuid.uuid4().hex[:8]}&quot;<br>        st.session_state[&quot;messages&quot;] = []<br>        st.rerun()<br><br>    st.divider()<br><br>    # Prompt version indicator<br>    _, prompt_version = get_managed_prompt()<br>    st.caption(f&quot;Prompt version: **{prompt_version}**&quot;)<br>    langfuse_host = os.getenv(&quot;LANGFUSE_HOST&quot;, &quot;https://cloud.langfuse.com&quot;).rstrip(&quot;/&quot;)<br>    st.caption(f&quot;Open [{langfuse_host}]({langfuse_host}) → Tracing to inspect traces.&quot;)<br><br># ── Build the chain (or show an error) ───────────────────────────────────────<br><br>chain = None<br>chunk_count = 0<br>if pdf_path:<br>    with st.spinner(&quot;Loading document and building index…&quot;):<br>        try:<br>            chain, chunk_count = build_chain(pdf_path)<br>        except Exception as e:<br>            st.error(f&quot;Failed to build chain: {e}&quot;)<br><br># ── Chat history ──────────────────────────────────────────────────────────────<br><br>if &quot;messages&quot; not in st.session_state:<br>    st.session_state[&quot;messages&quot;] = []<br><br>if chain and chunk_count:<br>    st.caption(f&quot;Index ready · {chunk_count} chunks&quot;)<br><br>for msg in st.session_state[&quot;messages&quot;]:<br>    with st.chat_message(msg[&quot;role&quot;]):<br>        st.markdown(msg[&quot;content&quot;])<br>        if msg[&quot;role&quot;] == &quot;assistant&quot; and msg.get(&quot;links&quot;):<br>            _render_trace_links(msg[&quot;links&quot;])<br><br># ── Chat input ────────────────────────────────────────────────────────────────<br><br>if question := st.chat_input(<br>    &quot;Ask a question about the document…&quot;,<br>    disabled=chain is None,<br>):<br>    # Show user message immediately<br>    st.session_state[&quot;messages&quot;].append({&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: question})<br>    with st.chat_message(&quot;user&quot;):<br>        st.markdown(question)<br><br>    # Run chain with Langfuse instrumentation<br>    with st.chat_message(&quot;assistant&quot;):<br>        with st.spinner(&quot;Thinking…&quot;):<br>            links = None<br>            try:<br>                _, pv = get_managed_prompt()<br>                handler = CallbackHandler()<br>                session_id = st.session_state[&quot;session_id&quot;]<br>                uid = st.session_state[&quot;user_id&quot;]<br>                result = chain.invoke(<br>                    {&quot;input&quot;: question},<br>                    config={<br>                        &quot;callbacks&quot;: [handler],<br>                        &quot;metadata&quot;: {<br>                            &quot;langfuse_session_id&quot;: session_id,<br>                            &quot;langfuse_user_id&quot;: uid,<br>                            &quot;prompt_version&quot;: pv,<br>                        },<br>                    },<br>                )<br>                answer = result[&quot;answer&quot;]<br>                get_langfuse().flush()<br><br>                trace_id = handler.last_trace_id<br>                if trace_id:<br>                    lf = get_langfuse()<br>                    trace_url = lf.get_trace_url(trace_id=trace_id)<br>                    if trace_url:<br>                        proj_base = trace_url.rsplit(&quot;/traces/&quot;, 1)[0]<br>                        links = {<br>                            &quot;Trace&quot;: trace_url,<br>                            &quot;Session&quot;: f&quot;{proj_base}/sessions/{session_id}&quot;,<br>                            &quot;User&quot;: f&quot;{proj_base}/users/{uid}&quot;,<br>                        }<br>            except Exception as e:<br>                answer = f&quot;Error: {e}&quot;<br><br>        st.markdown(answer)<br>        if links:<br>            _render_trace_links(links)<br><br>    st.session_state[&quot;messages&quot;].append(<br>        {&quot;role&quot;: &quot;assistant&quot;, &quot;content&quot;: answer, &quot;links&quot;: links}<br>    )</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*909nQGls1oi5R-3kl8b9MA.gif" /></figure><p><em>Thanks for reading this article !!</em></p><p>If you enjoyed this article, please click on the clap button 👏 and share to help others find it!</p><p>The full source code for this tutorial can be found here,</p><p><a href="https://github.com/codemaker2015/langfuse-experiments">GitHub - codemaker2015/langfuse-experiments: Sample Python projects for learning Langfuse observability with OpenAI and LangChain.</a></p><h4>References</h4><ul><li><a href="https://langfuse.com/docs">Open Source AI Engineering Platform - Langfuse</a></li><li><a href="https://github.com/langfuse/langfuse">GitHub - langfuse/langfuse: 🪢 Open source AI engineering platform: AI Gateway, agent observability, evals, and prompt/skill management, built on OpenTelemetry. One open platform for every AI call and every agent. Self-host or use Langfuse Cloud.</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=85644f35a4ca" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Grafana k6: A Complete Guide to Modern Performance Testing]]></title>
            <link>https://codemaker2016.medium.com/grafana-k6-a-complete-guide-to-modern-performance-testing-55b165cd9bd8?source=rss-b7a60e472dfc------2</link>
            <guid isPermaLink="false">https://medium.com/p/55b165cd9bd8</guid>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[grafana]]></category>
            <category><![CDATA[load-testing]]></category>
            <dc:creator><![CDATA[Vishnu Sivan]]></dc:creator>
            <pubDate>Sun, 26 Jul 2026 16:34:10 GMT</pubDate>
            <atom:updated>2026-07-26T16:34:10.754Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vPk0QENnL6189B5VcZW8xA.png" /></figure><p>If you’ve shipped a feature that worked perfectly in dev, sailed through QA, and then buckled the moment real traffic hit it, you already understand why performance testing exists. Functional tests tell you whether something works. Performance tests tell you whether it keeps working when 50, 500, or 5,000 users hit it at once.</p><p>Grafana k6 has become one of the most popular tools for this job. It’s open source, scriptable in JavaScript, lightweight enough to run from a laptop, and built to slot naturally into CI/CD pipelines.</p><p>This guide covers what k6 is and how it thinks, then walks through a complete hands-on: a small real app — a Task Manager with a React frontend and a Node + SQLite backend — and load-test it with four different k6 scripts, from a basic smoke test up to a full CRUD user-journey scenario.</p><h3>Getting Started</h3><h4>Table of Contents</h4><ul><li>What Is Grafana k6?</li><li>k6 Workflow</li><li>Why teams pick k6 over older tools</li><li>Experimenting with K6</li><li>Installation</li><li>Writing Your First Load Test</li><li>Anatomy of a k6 Script</li><li>1. The Default Function</li><li>2. Test Options</li><li>3. Groups, Tags, Checks, and Thresholds</li><li>Checks vs. Thresholds vs. Custom Metrics</li><li>4. Executors — Modeling Different Traffic Shapes</li><li>Putting It Together: Multiple Scenarios in One Test</li><li>Quick Reference (k6 options)</li><li>Handson: Full stack app load testing using k6</li><li>Step 1: Building the Backend</li><li>Step 2: Building the Frontend</li><li>Step 3: The k6 Hands-On</li><li>1. Smoke test</li><li>2. Staged load test</li><li>3. Full CRUD user-journey test</li><li>4. Spike test</li><li>5. Bringing It Into CI/CD</li><li>6. Export to Grafana via Prometheus</li><li>Quick References (Commands)</li></ul><h4>What Is Grafana k6?</h4><p>Grafana k6 is an open-source, developer-friendly, and extensible performance testing tool that helps you catch performance issues early and proactively improve reliability. Using k6, you can test the reliability and performance of your application and infrastructure, helping engineering teams prevent errors and SLO breaches so they can build resilient, high-performing applications that scale.</p><p>It’s commonly used by developers, QA engineers, SDETs, and SREs for:</p><ul><li>Load and performance testing — k6 is optimized for minimal resource consumption and designed for running high-load performance tests such as spike, stress, or soak tests.</li><li>Browser performance testing — through the k6 browser API, collecting browser metrics, and mixing browser tests with other performance tests for a comprehensive view of a website’s performance.</li><li>Synthetic monitoring — scheduling tests to run with minimal load frequently, to continuously validate the performance and availability of a production environment.</li><li>CI/CD automation — k6 integrates seamlessly with CI/CD and automation tools, letting engineering teams automate performance testing as part of the release cycle.</li><li>Chaos and resilience testing, and infrastructure testing via extensions.</li></ul><h4>k6 Key concepts</h4><p>k6 introduces a few core concepts that are essential for creating and understanding performance tests. Knowing these terms will make it easier to write, execute, and analyze your tests.</p><ul><li><strong>Virtual Users (VUs)</strong>: Virtual Users (VUs) are simulated users that execute your test script independently. Each VU continuously performs the defined actions, just like a real user. Increasing the number of VUs increases the concurrent load on your application.</li><li><strong>Iterations</strong>: An iteration is one complete execution of a test script by a single VU. Each VU repeats iterations throughout the test duration. The total number of iterations helps measure your application’s throughput.</li><li><strong>Stages</strong>: Stages define how the test load changes over time. They allow you to gradually increase, maintain, and decrease the number of VUs, creating realistic traffic patterns instead of applying the full load instantly.</li><li><strong>Thresholds</strong>: Thresholds define the pass or fail conditions for a performance test. You can specify limits for response time, error rate, or other metrics. If the conditions are not met, the test is marked as failed.</li><li><strong>Checks</strong>: Checks validate responses during test execution, similar to assertions in functional testing. They verify conditions such as HTTP status codes or response content but do not fail the test unless combined with thresholds.</li><li><strong>Metrics</strong>: k6 automatically collects performance metrics such as http_req_duration, http_req_failed, vus, and iterations. These metrics help you evaluate application performance and identify bottlenecks under load.</li></ul><h4>k6 Workflow</h4><p>Every k6 test consists of three main parts. The first is the options block, where you configure the test by specifying the number of Virtual Users (VUs), test duration, load stages, and thresholds. It defines how the test should execute. The second is the default function, which contains the test scenario executed by every VU. This includes actions such as logging in, calling APIs, uploading files, or performing any other user interaction. The final part includes checks and thresholds, which validate responses during execution and define the pass or fail conditions. Together, these components create a complete performance test that defines the workload, execution flow, and success criteria.</p><p>After a test finishes, k6 displays a summary of the collected metrics. The http_req_duration metric shows request latency, including the average, minimum, maximum, and percentile values. The <strong>p(90)</strong> and <strong>p(95)</strong> percentiles are especially important because they reflect the response times experienced by most users, providing a more accurate view than the average alone. The http_req_failed metric reports the percentage of failed requests, helping identify issues such as timeouts or server errors. Finally, the vus and vus_max metrics confirm that the expected number of Virtual Users were active, ensuring the intended load was applied successfully.</p><h4>Why teams pick k6 over older tools</h4><p>Apache JMeter has been the default performance-testing tool for years, but it’s Java-based, GUI-heavy, and can take a while to configure before you run a single test. k6 takes a different approach: tests are plain JavaScript files, everything runs from the command line, and results are readable the moment a run finishes. That “test as code” model is why k6 fits naturally into a Git repo and a CI pipeline the same way unit tests do.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1012/1*NWUAECkxcbrihLHVfd38cA.png" /></figure><p><strong>The four main load test types</strong></p><ul><li><strong>Load testing</strong> — realistic expected concurrent traffic, to confirm the system handles normal load without degrading.</li><li><strong>Stress testing</strong> — pushes load beyond normal limits to find the breaking point.</li><li><strong>Spike testing</strong> — a sudden, massive burst of traffic, measuring how gracefully the system handles and recovers from it.</li><li><strong>Soak testing</strong> — a moderate, sustained load for hours, surfacing slow-building problems like memory leaks or connection-pool exhaustion.</li></ul><h3>Experimenting with K6</h3><h4>Install k6</h4><p>k6 is available for Linux, macOS, and Windows. You can also run it using a Docker container or download the standalone binary.</p><p><strong>Linux:</strong></p><pre>curl -fsSL https://dl.k6.io/key.gpg | sudo gpg --dearmor -o /usr/share/keyrings/k6-archive-keyring.gpg<br>echo &quot;deb [signed-by=/usr/share/keyrings/k6-archive-keyring.gpg] https://dl.k6.io/deb stable main&quot; | sudo tee /etc/apt/sources.list.d/k6.list<br>sudo apt-get update<br>sudo apt-get install k6</pre><p><strong>macOS:</strong></p><pre>brew install k6</pre><p><strong>Windows:</strong></p><pre>winget install k6 --source winget</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0yNwFLIXIiAxeYBmiKw3-Q.png" /></figure><p>Alternatively, you can download and run the latest <a href="https://dl.k6.io/msi/k6-latest-amd64.msi">official k6 installer</a>.</p><p><a href="https://dl.k6.io/msi/k6-latest-amd64.msi">https://dl.k6.io/msi/k6-latest-amd64.msi</a></p><p><strong>Docker:</strong></p><pre>docker pull grafana/k6</pre><p>To execute browser-based tests, use the Docker image that includes Chromium:</p><pre>docker pull grafana/k6:master-with-browser</pre><h4>Writing Your First Load Test</h4><p>Let’s create a basic load test for the sample website (https://test.k6.io). Create a file named example-test.js with the following code:</p><pre>import http from &#39;k6/http&#39;;<br>import { sleep } from &#39;k6&#39;;<br> <br>// Executed by each Virtual User (VU)<br>export default function () {<br>  // Send a GET request<br>  http.get(&#39;https://test.k6.io&#39;);<br>  <br>  // Wait for 1 second<br>  sleep(1);<br>}</pre><p><strong>Explanation</strong></p><ul><li>import http from &#39;k6/http&#39;: Imports the HTTP module.</li><li>export default function(): Defines the test scenario for each Virtual User.</li><li>http.get(): Sends a GET request to the target URL.</li><li>sleep(1): Waits for one second before the next iteration.</li></ul><h4>Running the Test</h4><p>Run the following command:</p><pre>k6 run test.js</pre><h4>Output Overview</h4><p>k6 displays key performance metrics, including:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tANfq9CPzbK9AySk1E8uFA.png" /></figure><ul><li><strong>vus</strong>: Number of active Virtual Users.</li><li><strong>http_req_duration</strong>: Request response time statistics.</li><li><strong>iterations</strong>: Total completed test iterations.</li></ul><h4>Anatomy of a k6 Script</h4><p>Every k6 test is built from up to four exported pieces:</p><pre>import http from &#39;k6/http&#39;;<br>import { check, sleep } from &#39;k6&#39;;<br>export const options = { /* test configuration */ };<br>export function setup() { /* runs once, before the test */ }<br>export default function (data) { /* runs once per VU, per iteration */ }<br>export function teardown(data) { /* runs once, after the test */ }</pre><p>k6 executes these in a fixed order: setup() once → the default function repeatedly (once per iteration, per VU, per whatever the options dictate) → teardown() once, at the very end</p><h4>1. The Default Function</h4><p>This is the only required piece. Whatever you put in the default (unnamed) exported function is what each virtual user (VU) actually runs, over and over, for the life of the test. &lt;cite index=”5–1&quot;&gt;The default function acts as the entry point for each VU, and each VU repeats the code inside it for the configured duration or number of iterations.&lt;/cite&gt;</p><pre>import http from &#39;k6/http&#39;;<br>import { sleep } from &#39;k6&#39;;<br><br>export default function () {<br>  http.get(&#39;http://localhost:3001/api/tasks&#39;);<br>  sleep(1); // &quot;think time&quot; between requests, just like a real user pausing<br>}</pre><p>Every statement inside this function — every http.get, http.post, check, sleep — happens once per iteration. If you configure 20 VUs, k6 runs 20 concurrent copies of this function in a loop.</p><p><strong>setup() — run once, before load starts</strong></p><p>Use setup() for anything that should happen exactly once, outside the timed test: logging in to get an auth token, seeding test data, hitting a &quot;reset&quot; endpoint. Whatever setup() returns gets passed into the default function and teardown() as data.</p><pre>export function setup() {<br>  // Reset the Task Manager&#39;s database to a known state before the run<br>  const res = http.post(&#39;http://localhost:3001/api/reset&#39;);<br>  return { resetCount: res.json(&#39;count&#39;) };<br>}<br>export default function (data) {<br>  console.log(`Started with ${data.resetCount} seeded tasks`);<br>  http.get(&#39;http://localhost:3001/api/tasks&#39;);<br>}</pre><p><strong>teardown() — run once, after load ends</strong></p><p>The mirror image of setup() — cleanup, final logging, or notifying another system that the run finished. It also receives the object setup() returned.</p><pre>export function teardown(data) {<br>  console.log(`Test complete. Started with ${data.resetCount} tasks.`);<br>}</pre><p><strong>handleSummary() — custom end-of-test report</strong></p><p>An optional fourth function that lets you control exactly what k6 prints or writes when the test finishes — a custom JSON file, an HTML report, a Slack-friendly summary, instead of (or alongside) the default terminal output.</p><pre>export function handleSummary(data) {<br>  return {<br>    &#39;summary.json&#39;: JSON.stringify(data, null, 2), // write full results to a file<br>    stdout: `Requests: ${data.metrics.http_reqs.values.count}\n`, // still print something to the terminal<br>  };<br>}</pre><h4>2. The options Object — Test Configuration</h4><p>Everything about how a test runs — not what it does — lives in options. Configure the test using export const options = {}. The most commonly used fields:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/932/1*V_cuVSMP9_M5-6-EO759RQ.png" /></figure><p><strong>The simplest form: </strong><strong>vus + </strong><strong>duration</strong></p><pre>export const options = {<br>  vus: 10,<br>  duration: &#39;30s&#39;,<br>};</pre><p>10 virtual users, each looping the default function continuously for 30 seconds.</p><p><strong>Ramping load with </strong><strong>stages</strong></p><pre>export const options = {<br>  stages: [<br>    { duration: &#39;30s&#39;, target: 20 }, // ramp up to 20 VUs<br>    { duration: &#39;1m&#39;,  target: 20 }, // hold at 20 VUs<br>    { duration: &#39;20s&#39;, target: 0  }, // ramp down<br>  ],<br>};</pre><p>stages is really shorthand for a ramping-vus executor (see below) — it&#39;s the most common way people write their first non-trivial test.</p><p><strong>Thresholds — turning results into pass/fail</strong></p><p>Thresholds are what make a test result binary. Without them, a run just produces numbers; with them, k6 exits non-zero the moment an SLO is breached — which is exactly what a CI pipeline needs to gate a merge.</p><pre>export const options = {<br>  vus: 10,<br>  duration: &#39;1m&#39;,<br>  thresholds: {<br>    http_req_failed: [&#39;rate&lt;0.01&#39;],               // fewer than 1% of requests may fail<br>    http_req_duration: [&#39;p(95)&lt;400&#39;, &#39;avg&lt;250&#39;],  // p95 under 400ms AND average under 250ms<br>    checks: [&#39;rate&gt;0.99&#39;],                         // at least 99% of checks must pass<br>  },<br>};</pre><p>Each metric can have multiple threshold expressions; if anyone fails, the whole test is marked failed.</p><h4>3. Groups, Tags, Checks, and Thresholds</h4><p>Checks vs. Thresholds vs. Custom Metrics</p><ul><li><strong>Groups:</strong> Organize related test steps into logical workflows.</li><li><strong>Tags:</strong> Add metadata for filtering and analysis.</li><li><strong>Checks:</strong> Validate response status, content, or timing.</li><li><strong>Thresholds:</strong> Define acceptable performance limits.</li></ul><p><strong>Groups</strong></p><p>Use group() to organize related actions into a single workflow.</p><pre>import { group } from &#39;k6&#39;;<br> <br>export default function () {<br>  group(&#39;User Login Flow&#39;, () =&gt; {<br>    // Step 1: Login<br>    const loginRes = http.post(&#39;https://test.k6.io/login&#39;, {<br>      username: &#39;test&#39;,<br>      password: &#39;test&#39;,<br>    });<br>    check(loginRes, { &#39;Login successful&#39;: (r) =&gt; r.status === 200 });<br> <br>    // Step 2: Navigate to dashboard<br>    const dashRes = http.get(&#39;https://test.k6.io/dashboard&#39;, {<br>      headers: { &#39;Authorization&#39;: `Bearer ${loginRes.json(&#39;token&#39;)}` },<br>    });<br>    check(dashRes, { &#39;Dashboard loads&#39;: (r) =&gt; r.status === 200 });<br>  });<br> <br>  sleep(1);<br>}</pre><p><strong>Tags</strong></p><p>Tags help categorize requests and metrics. They are useful when analyzing specific API calls or user flows.</p><pre>const res = http.get(&#39;https://test.k6.io&#39;, {<br> tags: { name: &#39;Homepage_Load&#39; },<br>});</pre><h4>Checks vs. Thresholds vs. Custom Metrics</h4><p>These three get confused constantly, so it’s worth being precise:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/930/1*7E6b6mevcT4EbxVVY4_Stw.png" /></figure><p><strong>Example:</strong></p><pre>import http from &#39;k6/http&#39;;<br>import { check } from &#39;k6&#39;;<br>import { Trend, Counter } from &#39;k6/metrics&#39;;<br><br>const queueDelay = new Trend(&#39;queue_delay_ms&#39;);<br>const authFailures = new Counter(&#39;auth_failures&#39;);<br><br>export const options = {<br>  thresholds: {<br>    // At least 95% of all checks should pass<br>    checks: [&#39;rate&gt;0.95&#39;],<br>    // Average queue delay should be less than 500 ms<br>    queue_delay_ms: [&#39;avg&lt;500&#39;],<br>    // 95% of queue delays should be less than 1000 ms<br>    queue_delay_ms: [&#39;p(95)&lt;1000&#39;],<br>    // Total authentication failures should be fewer than 5<br>    auth_failures: [&#39;count&lt;5&#39;],<br>  },<br>};<br><br>export default function () {<br>  const res = http.get(&#39;http://localhost:3001/api/tasks&#39;);<br><br>  check(res, { &#39;status is 200&#39;: (r) =&gt; r.status === 200 }); // a check<br>  if (res.status !== 200) authFailures.add(1);              // a custom metric<br><br>  queueDelay.add(res.timings.waiting); // record a custom Trend value<br>}</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M5z3cxFNxMZS9D2EQdJ3Kg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yw-corUdXnZQWTBivhHdtw.png" /></figure><h4>4. Executors — Modeling Different Traffic Shapes</h4><p>An executor is the algorithm k6 uses to decide <em>how</em> to schedule VUs and iterations over time. vus/duration/stages at the top of options are really just a convenient shorthand for the two most common executors. For anything more nuanced, you configure scenarios directly, where each scenario names its own executor.</p><pre>export const options = {<br>  scenarios: {<br>    my_scenario_name: {<br>      executor: &#39;ramping-vus&#39;, // or constant-arrival-rate, per-vu-iterations, etc.<br>      // ...executor-specific settings<br>    },<br>  },<br>};</pre><p>Multiple scenarios can run simultaneously, each simulating a different kind of traffic against the same target — e.g. steady background browsing plus a login spike, in the same test run.</p><p><strong>shared-iterations</strong></p><p>Runs a fixed total number of iterations, shared across a pool of VUs — whichever VU is free next picks up the next iteration. Good for “run this exact number of test cases, split across N workers,” where timing doesn’t matter.</p><pre>scenarios: {<br>  bulk_reads: {<br>    executor: &#39;shared-iterations&#39;,<br>    vus: 10,<br>    iterations: 200,       // 200 total iterations, shared across 10 VUs<br>    maxDuration: &#39;1m&#39;,<br>  },<br>},</pre><p><strong>per-vu-iterations</strong></p><p>Each VU independently runs a fixed number of iterations (not shared — every VU does its own full count). Because VUs typically start together, this is the classic instant burst / spike pattern: everyone hits the app at once.</p><p><strong>Use it for</strong>: login storms, push-notification bursts, flash-sale “everyone clicks buy at once” scenarios.</p><pre>scenarios: {<br>  login_storm: {<br>    executor: &#39;per-vu-iterations&#39;,<br>    vus: 100,<br>    iterations: 1,          // each of the 100 VUs runs the function exactly once<br>    gracefulStop: &#39;30s&#39;,<br>    exec: &#39;login&#39;,<br>  },<br>},<br><br>export function login() {<br>  const res = http.post(&#39;http://localhost:3001/api/tasks&#39;,<br>    JSON.stringify({ title: &#39;burst task&#39; }),<br>    { headers: { &#39;Content-Type&#39;: &#39;application/json&#39; } });<br>  check(res, { &#39;created&#39;: (r) =&gt; r.status === 201 });<br>}</pre><p><strong>constant-vus</strong></p><p>A fixed number of VUs, looping continuously for a fixed duration — essentially the explicit form of the plain vus + duration shorthand.</p><pre>scenarios: {<br>  steady_browse: {<br>    executor: &#39;constant-vus&#39;,<br>    vus: 20,<br>    duration: &#39;2m&#39;,<br>  },<br>},</pre><p><strong>ramping-vus</strong></p><p>Controls the number of concurrent virtual users over time, via stages. This is the executor behind the top-level stages shorthand shown earlier.</p><p><strong>Use it for:</strong> simulating organic user growth, cache/connection-pool warm-up, general “what happens as concurrent users increase” testing.</p><pre>scenarios: {<br>  ramp_up: {<br>    executor: &#39;ramping-vus&#39;,<br>    startVUs: 0,<br>    stages: [<br>      { duration: &#39;1m&#39;,  target: 50 }, // gradually grow to 50 users<br>      { duration: &#39;2m&#39;,  target: 50 }, // hold steady<br>      { duration: &#39;30s&#39;, target: 0  }, // ramp down<br>    ],<br>    gracefulRampDown: &#39;30s&#39;, // let in-flight iterations finish before killing a VU<br>  },<br>},</pre><p><strong>constant-arrival-rate</strong></p><p>Instead of controlling VUs directly, this executor targets a fixed requests-per-second (or iterations-per-second) rate, and lets k6 add or remove VUs behind the scenes as needed to hit that rate — even if response times increase.</p><p><strong>Use it for:</strong> SLA validation (“can we sustain exactly 100 req/s?”), capacity planning, any test where the <em>question</em> is about a specific throughput number rather than a specific user count.</p><pre>scenarios: {<br>  sustained_api: {<br>    executor: &#39;constant-arrival-rate&#39;,<br>    rate: 100,              // 100 iterations per timeUnit<br>    timeUnit: &#39;1s&#39;,         // → 100 iterations/second<br>    duration: &#39;3m&#39;,<br>    preAllocatedVUs: 60,    // VU pool k6 starts with<br>    maxVUs: 120,            // ceiling k6 can grow the pool to, if needed<br>  },<br>},</pre><p><strong>ramping-arrival-rate</strong></p><p>The throughput equivalent of ramping-vus — instead of ramping the VU count, it ramps the target rate itself through stages, while k6 manages the underlying VU pool automatically.</p><p><strong>Use it for:</strong> simulating a realistic ramp to peak traffic — a product launch, a Black Friday-style curve — where you want a guaranteed, gradually increasing throughput rather than a raw user count, and you want to find the point where the system can no longer keep up.</p><pre>scenarios: {<br>  ramp_to_peak: {<br>    executor: &#39;ramping-arrival-rate&#39;,<br>    startRate: 10,<br>    timeUnit: &#39;1s&#39;,<br>    preAllocatedVUs: 50,<br>    maxVUs: 300,<br>    stages: [<br>      { duration: &#39;1m&#39;, target: 50  },  // warm up to 50 req/s<br>      { duration: &#39;2m&#39;, target: 100 },  // scale to 100 req/s<br>      { duration: &#39;3m&#39;, target: 200 },  // push to 200 req/s<br>      { duration: &#39;2m&#39;, target: 200 },  // hold peak<br>      { duration: &#39;1m&#39;, target: 0   },  // ramp down<br>    ],<br>  },<br>},</pre><p><strong>externally-controlled</strong></p><p>A less commonly used executor that lets you control VUs and duration while the test is running, via the k6 REST API — useful for building custom orchestration on top of k6, but not something most day-to-day tests need.</p><h4>Putting It Together: Multiple Scenarios in One Test</h4><p>Real load rarely comes from a single traffic pattern — production traffic is browsing users <em>plus</em> a steady API stream <em>plus</em> the occasional burst, all at once. scenarios lets you run several executors concurrently, each pointed at a different exported function via exec:</p><pre>import http from &#39;k6/http&#39;;<br>import { check } from &#39;k6&#39;;<br><br>export const options = {<br>  scenarios: {<br>    browsing: {<br>      executor: &#39;ramping-vus&#39;,<br>      startVUs: 0,<br>      stages: [<br>        { duration: &#39;30s&#39;, target: 20 },<br>        { duration: &#39;1m&#39;,  target: 20 },<br>        { duration: &#39;20s&#39;, target: 0  },<br>      ],<br>      exec: &#39;browse&#39;,<br>    },<br>    steady_api: {<br>      executor: &#39;constant-arrival-rate&#39;,<br>      rate: 50,<br>      timeUnit: &#39;1s&#39;,<br>      duration: &#39;1m30s&#39;,<br>      preAllocatedVUs: 30,<br>      maxVUs: 60,<br>      exec: &#39;listTasks&#39;,<br>    },<br>    burst: {<br>      executor: &#39;per-vu-iterations&#39;,<br>      vus: 30,<br>      iterations: 1,<br>      exec: &#39;createTask&#39;,<br>    },<br>  },<br>  thresholds: {<br>    &#39;http_req_duration{scenario:steady_api}&#39;: [&#39;p(95)&lt;300&#39;],<br>    &#39;checks{scenario:burst}&#39;: [&#39;rate&gt;0.95&#39;],<br>  },<br>};<br><br>const BASE_URL = &#39;http://localhost:3001&#39;;<br><br>export function browse() {<br>  http.get(`${BASE_URL}/api/tasks`);<br>}<br><br>export function listTasks() {<br>  const res = http.get(`${BASE_URL}/api/tasks`);<br>  check(res, { &#39;200&#39;: (r) =&gt; r.status === 200 });<br>}<br><br>export function createTask() {<br>  const res = http.post(`${BASE_URL}/api/tasks`,<br>    JSON.stringify({ title: &#39;burst-created task&#39; }),<br>    { headers: { &#39;Content-Type&#39;: &#39;application/json&#39; } });<br>  check(res, { &#39;created&#39;: (r) =&gt; r.status === 201 });<br>}</pre><p>Notice the thresholds syntax &#39;http_req_duration{scenario:steady_api}&#39; — tagging lets you set thresholds (and later, filter dashboards) <em>per scenario</em>, not just globally across the whole test.</p><h3>Quick Reference (k6 options)</h3><pre>export const options = {<br>  vus: 10,                          // simple: fixed VUs<br>  duration: &#39;30s&#39;,                  //         + fixed duration<br>  // — or —<br>  stages: [ { duration, target } ], // simple: ramping VUs<br>  // — or, for full control —<br>  scenarios: {<br>    name: {<br>      executor: &#39;shared-iterations&#39; | &#39;per-vu-iterations&#39; | &#39;constant-vus&#39;<br>              | &#39;ramping-vus&#39; | &#39;constant-arrival-rate&#39; | &#39;ramping-arrival-rate&#39;<br>              | &#39;externally-controlled&#39;,<br>      exec: &#39;functionName&#39;, // which exported function this scenario runs<br>      // ...executor-specific settings (vus, rate, stages, preAllocatedVUs, maxVUs, ...)<br>    },<br>  },<br>  thresholds: {<br>    http_req_duration: [&#39;p(95)&lt;500&#39;],<br>    http_req_failed: [&#39;rate&lt;0.01&#39;],<br>  },<br>};<br><br>export function setup() { /* once, before */ }<br>export default function (data) { /* per iteration, per VU */ }<br>export function teardown(data) { /* once, after */ }<br>export function handleSummary(data) { /* custom report output */ }</pre><p><strong>Choosing the Right Executor</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/927/1*LoAtOlG0L2vqpUhXLII0Ng.png" /></figure><h4><strong>4. Data Parameterization</strong></h4><p>Incorporate external datasets (e.g., CSV, JSON files) to inject dynamic values into tests (such as individual user credentials or product identifiers):</p><p>Create users.csv:</p><pre>username,password<br>user1,pass1<br>user2,pass2<br>user3,pass3</pre><p>In your test script:</p><pre>import papaparse from &#39;https://jslib.k6.io/papaparse/5.1.1/index.js&#39;;<br>import { SharedArray } from &#39;k6/data&#39;;<br>import http from &#39;k6/http&#39;;<br><br>const users = new SharedArray(&#39;users&#39;, function () {<br>    const csv = open(&#39;./users.csv&#39;);<br>    return papaparse.parse(csv, {<br>        header: true<br>    }).data;<br>});<br><br>export const options = {<br>    vus: 10,<br>    duration: &#39;30s&#39;,<br>};<br><br>export default function () {<br>    const user = users[Math.floor(Math.random() * users.length)];<br><br>    http.get(`https://example.com/api/user/${user.id}`);<br>}</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MEQSPdZWKhaMoqhmuyYwUA.png" /></figure><h4><strong>5. Handling Dynamic Responses</strong></h4><p>Capture values such as auth tokens, session markers, or generated URLs from server responses to reuse in subsequent steps:</p><pre>import http from &#39;k6/http&#39;;<br>import { check } from &#39;k6&#39;;<br><br>// Authenticate and capture access token<br>export default function () {<br><br>    const loginRes = http.post(&#39;https://test.k6.io/login&#39;, {<br>        username: &#39;test&#39;,<br>        password: &#39;test&#39;,<br>    });<br><br>    const token = loginRes.json(&#39;access_token&#39;);<br><br>    // Include extracted token within request headers<br>    const headers = {<br>        Authorization: `Bearer ${token}`,<br>    };<br><br>    const secureRes = http.get(<br>        &#39;https://test.k6.io/secure&#39;,<br>        { headers }<br>    );<br><br>    check(secureRes, {<br>        &#39;Secure endpoint access&#39;: (r) =&gt; r.status === 200,<br>    });<br>}</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LLjB_XRiHoj-P2MmKEOnMA.png" /></figure><h3>Handson: Full stack app load testing using k6</h3><p>Instead of relying on Grafana’s public test.k6.io demo application, we will build and test a real-world, full-stack Task Manager application. This approach provides a more practical understanding of performance testing by evaluating an application that closely resembles production environments.</p><ul><li><strong>Backend:</strong> Node.js + Express + SQLite (using Node’s built-in node:sqlite module — no native compilation, nothing extra to install)</li><li><strong>Frontend:</strong> React + Vite</li><li><strong>Tests:</strong> k6, against the backend’s REST API</li></ul><p><strong>Project layout</strong></p><pre>task-manager-app/<br>├── backend/<br>│   ├── package.json<br>│   ├── db.js           # SQLite setup + seed data<br>│   └── server.js       # Express REST API<br>├── frontend/<br>│   ├── package.json<br>│   ├── vite.config.js<br>│   └── src/<br>│       ├── main.jsx<br>│       ├── App.jsx     # Task list UI<br>│       └── App.css<br>└── k6-tests/<br>    ├── smoke-test.js<br>    ├── load-test.js<br>    ├── crud-scenario.js<br>    └── spike-test.js</pre><h4>Step 1: Building the Backend</h4><p>Lets begin by setting up the backend for our Task Manager application. The backend will be built using Node.js, Express, and SQLite.</p><p>Start with initializing a new Node.js project and installing the required dependencies.</p><pre>cd backend<br>npm init<br>npm install express cors</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AhdrSYvbWC_K9YWckbe3Ow.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/925/1*P0NYK3lGM7_JHrYtDwIyFg.png" /></figure><p>Create a db.js file under the backend folder and add the following code to it.</p><pre>import { DatabaseSync } from &#39;node:sqlite&#39;;<br>import path from &#39;node:path&#39;;<br>import { fileURLToPath } from &#39;node:url&#39;;<br><br>const __dirname = path.dirname(fileURLToPath(import.meta.url));<br>const DB_PATH = path.join(__dirname, &#39;tasks.db&#39;);<br><br>export const db = new DatabaseSync(DB_PATH);<br><br>db.exec(`<br>  CREATE TABLE IF NOT EXISTS tasks (<br>    id INTEGER PRIMARY KEY AUTOINCREMENT,<br>    title TEXT NOT NULL,<br>    description TEXT DEFAULT &#39;&#39;,<br>    completed INTEGER NOT NULL DEFAULT 0,<br>    created_at TEXT NOT NULL DEFAULT (datetime(&#39;now&#39;))<br>  );<br>`);<br><br>const SEED_TASKS = [<br>  { title: &#39;Set up project repo&#39;, description: &#39;Initialize git and CI&#39;, completed: 1 },<br>  { title: &#39;Design database schema&#39;, description: &#39;Tasks table with completed flag&#39;, completed: 1 },<br>  { title: &#39;Build REST API&#39;, description: &#39;CRUD endpoints for tasks&#39;, completed: 0 },<br>  { title: &#39;Build React UI&#39;, description: &#39;List, add, toggle, delete tasks&#39;, completed: 0 },<br>  { title: &#39;Load test with k6&#39;, description: &#39;Smoke, load, spike, and CRUD scenario tests&#39;, completed: 0 },<br>];<br><br>export function seed({ reset = false } = {}) {<br>  if (reset) {<br>    db.exec(&#39;DELETE FROM tasks;&#39;);<br>    db.exec(&quot;DELETE FROM sqlite_sequence WHERE name=&#39;tasks&#39;;&quot;);<br>  }<br><br>  const countRow = db.prepare(&#39;SELECT COUNT(*) AS count FROM tasks&#39;).get();<br>  if (countRow.count &gt; 0) return;<br><br>  const insert = db.prepare(<br>    &#39;INSERT INTO tasks (title, description, completed) VALUES (?, ?, ?)&#39;<br>  );<br>  for (const t of SEED_TASKS) {<br>    insert.run(t.title, t.description, t.completed);<br>  }<br>}<br><br>seed();</pre><p>Create a server.js file under the backend folder and add the following code to it.</p><pre>import express from &#39;express&#39;;<br>import cors from &#39;cors&#39;;<br>import { db, seed } from &#39;./db.js&#39;;<br><br>const app = express();<br>const PORT = process.env.PORT || 3001;<br><br>app.use(cors());<br>app.use(express.json());<br><br>// Small artificial delay + occasional jitter, so load tests have something<br>// interesting to measure (real APIs are never instant). Comment out the<br>// setTimeout in each route if you want raw, unthrottled numbers.<br>const THINK_MS = Number(process.env.SIMULATED_LATENCY_MS ?? 15);<br>function delay(ms = THINK_MS) {<br>  return new Promise((resolve) =&gt; setTimeout(resolve, ms));<br>}<br><br>// ---------- Health check ----------<br>app.get(&#39;/health&#39;, (req, res) =&gt; {<br>  res.json({ status: &#39;ok&#39;, uptime: process.uptime() });<br>});<br><br>// ---------- List all tasks ----------<br>app.get(&#39;/api/tasks&#39;, async (req, res) =&gt; {<br>  await delay();<br>  const rows = db.prepare(&#39;SELECT * FROM tasks ORDER BY id DESC&#39;).all();<br>  res.json(rows);<br>});<br><br>// ---------- Get a single task ----------<br>app.get(&#39;/api/tasks/:id&#39;, async (req, res) =&gt; {<br>  await delay();<br>  const row = db.prepare(&#39;SELECT * FROM tasks WHERE id = ?&#39;).get(req.params.id);<br>  if (!row) return res.status(404).json({ error: &#39;Task not found&#39; });<br>  res.json(row);<br>});<br><br>// ---------- Create a task ----------<br>app.post(&#39;/api/tasks&#39;, async (req, res) =&gt; {<br>  await delay();<br>  const { title, description = &#39;&#39; } = req.body ?? {};<br>  if (!title || typeof title !== &#39;string&#39; || !title.trim()) {<br>    return res.status(400).json({ error: &#39;title is required&#39; });<br>  }<br>  const info = db<br>    .prepare(&#39;INSERT INTO tasks (title, description, completed) VALUES (?, ?, 0)&#39;)<br>    .run(title.trim(), description);<br>  const created = db.prepare(&#39;SELECT * FROM tasks WHERE id = ?&#39;).get(info.lastInsertRowid);<br>  res.status(201).json(created);<br>});<br><br>// ---------- Update a task ----------<br>app.put(&#39;/api/tasks/:id&#39;, async (req, res) =&gt; {<br>  await delay();<br>  const existing = db.prepare(&#39;SELECT * FROM tasks WHERE id = ?&#39;).get(req.params.id);<br>  if (!existing) return res.status(404).json({ error: &#39;Task not found&#39; });<br><br>  const title = req.body?.title ?? existing.title;<br>  const description = req.body?.description ?? existing.description;<br>  const completed = req.body?.completed !== undefined<br>    ? (req.body.completed ? 1 : 0)<br>    : existing.completed;<br><br>  db.prepare(&#39;UPDATE tasks SET title = ?, description = ?, completed = ? WHERE id = ?&#39;)<br>    .run(title, description, completed, req.params.id);<br><br>  const updated = db.prepare(&#39;SELECT * FROM tasks WHERE id = ?&#39;).get(req.params.id);<br>  res.json(updated);<br>});<br><br>// ---------- Toggle completed flag ----------<br>app.post(&#39;/api/tasks/:id/toggle&#39;, async (req, res) =&gt; {<br>  await delay();<br>  const existing = db.prepare(&#39;SELECT * FROM tasks WHERE id = ?&#39;).get(req.params.id);<br>  if (!existing) return res.status(404).json({ error: &#39;Task not found&#39; });<br><br>  const completed = existing.completed ? 0 : 1;<br>  db.prepare(&#39;UPDATE tasks SET completed = ? WHERE id = ?&#39;).run(completed, req.params.id);<br><br>  const updated = db.prepare(&#39;SELECT * FROM tasks WHERE id = ?&#39;).get(req.params.id);<br>  res.json(updated);<br>});<br><br>// ---------- Delete a task ----------<br>app.delete(&#39;/api/tasks/:id&#39;, async (req, res) =&gt; {<br>  await delay();<br>  const info = db.prepare(&#39;DELETE FROM tasks WHERE id = ?&#39;).run(req.params.id);<br>  if (info.changes === 0) return res.status(404).json({ error: &#39;Task not found&#39; });<br>  res.status(204).send();<br>});<br><br>// ---------- Reset/reseed (handy for repeatable k6 runs) ----------<br>app.post(&#39;/api/reset&#39;, (req, res) =&gt; {<br>  seed({ reset: true });<br>  const rows = db.prepare(&#39;SELECT * FROM tasks&#39;).all();<br>  res.json({ message: &#39;Database reset&#39;, count: rows.length });<br>});<br><br>app.listen(PORT, () =&gt; {<br>  console.log(`Task Manager API listening on http://localhost:${PORT}`);<br>});</pre><h4>Step 2: Building the Frontend</h4><p>Now let’s build the React frontend for our Task Manager application using Vite.</p><p>Initialize the React project by running the following command.</p><pre>npm create vite@latest frontend</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/930/1*xF_t5yjw7hstIv85ix2kLw.png" /></figure><p>Replace the contents of the App.jsx file with the following code.</p><pre>import { useEffect, useState } from &#39;react&#39;;<br>import &#39;./App.css&#39;;<br><br>const API_BASE = &#39;/api/tasks&#39;;<br><br>export default function App() {<br>  const [tasks, setTasks] = useState([]);<br>  const [title, setTitle] = useState(&#39;&#39;);<br>  const [description, setDescription] = useState(&#39;&#39;);<br>  const [loading, setLoading] = useState(true);<br>  const [error, setError] = useState(&#39;&#39;);<br><br>  async function loadTasks() {<br>    setLoading(true);<br>    try {<br>      const res = await fetch(API_BASE);<br>      if (!res.ok) throw new Error(`Failed to load tasks (${res.status})`);<br>      setTasks(await res.json());<br>      setError(&#39;&#39;);<br>    } catch (err) {<br>      setError(err.message);<br>    } finally {<br>      setLoading(false);<br>    }<br>  }<br><br>  useEffect(() =&gt; {<br>    loadTasks();<br>  }, []);<br><br>  async function addTask(e) {<br>    e.preventDefault();<br>    if (!title.trim()) return;<br>    const res = await fetch(API_BASE, {<br>      method: &#39;POST&#39;,<br>      headers: { &#39;Content-Type&#39;: &#39;application/json&#39; },<br>      body: JSON.stringify({ title, description }),<br>    });<br>    if (res.ok) {<br>      setTitle(&#39;&#39;);<br>      setDescription(&#39;&#39;);<br>      loadTasks();<br>    }<br>  }<br><br>  async function toggleTask(id) {<br>    await fetch(`${API_BASE}/${id}/toggle`, { method: &#39;POST&#39; });<br>    loadTasks();<br>  }<br><br>  async function deleteTask(id) {<br>    await fetch(`${API_BASE}/${id}`, { method: &#39;DELETE&#39; });<br>    loadTasks();<br>  }<br><br>  const remaining = tasks.filter((t) =&gt; !t.completed).length;<br><br>  return (<br>    &lt;div className=&quot;app&quot;&gt;<br>      &lt;h1&gt;Task Manager&lt;/h1&gt;<br>      &lt;p className=&quot;subtitle&quot;&gt;<br>        {loading ? &#39;Loading…&#39; : `${remaining} of ${tasks.length} tasks remaining`}<br>      &lt;/p&gt;<br><br>      &lt;form className=&quot;add-form&quot; onSubmit={addTask}&gt;<br>        &lt;input<br>          type=&quot;text&quot;<br>          placeholder=&quot;Task title&quot;<br>          value={title}<br>          onChange={(e) =&gt; setTitle(e.target.value)}<br>        /&gt;<br>        &lt;input<br>          type=&quot;text&quot;<br>          placeholder=&quot;Description (optional)&quot;<br>          value={description}<br>          onChange={(e) =&gt; setDescription(e.target.value)}<br>        /&gt;<br>        &lt;button type=&quot;submit&quot;&gt;Add Task&lt;/button&gt;<br>      &lt;/form&gt;<br><br>      {error &amp;&amp; &lt;p className=&quot;error&quot;&gt;{error}&lt;/p&gt;}<br><br>      &lt;ul className=&quot;task-list&quot;&gt;<br>        {tasks.map((task) =&gt; (<br>          &lt;li key={task.id} className={task.completed ? &#39;completed&#39; : &#39;&#39;}&gt;<br>            &lt;label&gt;<br>              &lt;input<br>                type=&quot;checkbox&quot;<br>                checked={!!task.completed}<br>                onChange={() =&gt; toggleTask(task.id)}<br>              /&gt;<br>              &lt;div&gt;<br>                &lt;span className=&quot;task-title&quot;&gt;{task.title}&lt;/span&gt;<br>                {task.description &amp;&amp; (<br>                  &lt;span className=&quot;task-desc&quot;&gt;{task.description}&lt;/span&gt;<br>                )}<br>              &lt;/div&gt;<br>            &lt;/label&gt;<br>            &lt;button className=&quot;delete-btn&quot; onClick={() =&gt; deleteTask(task.id)}&gt;<br>              Delete<br>            &lt;/button&gt;<br>          &lt;/li&gt;<br>        ))}<br>      &lt;/ul&gt;<br>    &lt;/div&gt;<br>  );<br>}</pre><p>If you would like your application to have a cleaner and more visually appealing interface, replace the contents of the App.css file with the following code. Also, remove index.css file styles.</p><pre>* { box-sizing: border-box; }<br><br>body {<br>  margin: 0;<br>  font-family: -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Roboto, sans-serif;<br>  background: #f4f5f7;<br>  color: #1a1a1a;<br>}<br><br>.app {<br>  max-width: 560px;<br>  margin: 48px auto;<br>  padding: 32px;<br>  background: #fff;<br>  border-radius: 12px;<br>  box-shadow: 0 1px 3px rgba(0, 0, 0, 0.08);<br>}<br><br>h1 {<br>  margin: 0 0 4px;<br>  font-size: 24px;<br>}<br><br>.subtitle {<br>  margin: 0 0 24px;<br>  color: #6b7280;<br>  font-size: 14px;<br>}<br><br>.add-form {<br>  display: flex;<br>  gap: 8px;<br>  margin-bottom: 24px;<br>  flex-wrap: wrap;<br>}<br><br>.add-form input {<br>  flex: 1;<br>  min-width: 140px;<br>  padding: 10px 12px;<br>  border: 1px solid #d1d5db;<br>  border-radius: 8px;<br>  font-size: 14px;<br>}<br><br>.add-form button {<br>  padding: 10px 16px;<br>  background: #4f46e5;<br>  color: white;<br>  border: none;<br>  border-radius: 8px;<br>  cursor: pointer;<br>  font-size: 14px;<br>}<br><br>.add-form button:hover {<br>  background: #4338ca;<br>}<br><br>.error {<br>  color: #dc2626;<br>  font-size: 14px;<br>}<br><br>.task-list {<br>  list-style: none;<br>  margin: 0;<br>  padding: 0;<br>  display: flex;<br>  flex-direction: column;<br>  gap: 8px;<br>}<br><br>.task-list li {<br>  display: flex;<br>  align-items: center;<br>  justify-content: space-between;<br>  padding: 12px;<br>  border: 1px solid #e5e7eb;<br>  border-radius: 8px;<br>}<br><br>.task-list li.completed .task-title {<br>  text-decoration: line-through;<br>  color: #9ca3af;<br>}<br><br>.task-list label {<br>  display: flex;<br>  align-items: flex-start;<br>  gap: 10px;<br>  cursor: pointer;<br>}<br><br>.task-title {<br>  display: block;<br>  font-size: 15px;<br>}<br><br>.task-desc {<br>  display: block;<br>  font-size: 13px;<br>  color: #6b7280;<br>}<br><br>.delete-btn {<br>  background: none;<br>  border: none;<br>  color: #dc2626;<br>  cursor: pointer;<br>  font-size: 13px;<br>}<br><br>.delete-btn:hover {<br>  text-decoration: underline;<br>}</pre><p>To allow the React frontend to communicate with the Express backend during development, we need to configure a proxy in the Vite configuration file. This enables the frontend to make API requests using paths like /api/tasks without directly specifying the backend URL. The Vite development server will automatically forward these API requests to the Express backend running on port 3001.</p><p>Update the vite.config.js file with the following configuration.</p><pre>import { defineConfig } from &#39;vite&#39;;<br>import react from &#39;@vitejs/plugin-react&#39;;<br><br>// https://vitejs.dev/config/<br>export default defineConfig({<br>  plugins: [react()],<br>  server: {<br>    port: 5173,<br>    proxy: {<br>      // Lets the React app call fetch(&#39;/api/tasks&#39;) and have Vite forward<br>      // it to the Express backend on :3001 during development.<br>      &#39;/api&#39;: {<br>        target: &#39;http://localhost:3001&#39;,<br>        changeOrigin: true,<br>      },<br>    },<br>  },<br>});</pre><h4>Running the Application</h4><p>Now that both the backend and frontend have been configured, it’s time to start the application. The backend and frontend run as separate processes, so they need to be started in two different terminal windows.</p><pre># for backend<br>cd backend<br>npm start<br><br># for frontend<br>cd frontend<br>npm run dev</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bjuY3oSrsmv05zeL1sTwSQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/775/1*cIBMdhxYssH74TkUL92APg.png" /></figure><p>Once both servers are running successfully, open your browser and navigate to the URL displayed by the Vite development server (typically <a href="http://localhost:5173)">http://localhost:5173</a> ).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZLLiLyLhk1w37Rs7ZmI-NQ.png" /></figure><h4>Step 3: The k6 Hands-On</h4><p>With the backend running on http://localhost:3001, here are four k6 scripts, from simplest to most complete, all verified against the live app above.</p><h4>1. Smoke test</h4><p>Run this first, always: 1 VU, a handful of iterations, just confirming the API is alive before anything heavier.</p><pre>// smoke-test.js<br>// Run this first, always. 1 VU, a handful of iterations, just confirming<br>// the API is up and every endpoint responds the way it should before you<br>// run anything heavier.<br>//<br>// Usage:<br>//   k6 run smoke-test.js<br>//   k6 run -e BASE_URL=http://localhost:3001 smoke-test.js<br><br>import http from &#39;k6/http&#39;;<br>import { check, sleep } from &#39;k6&#39;;<br><br>const BASE_URL = __ENV.BASE_URL || &#39;http://localhost:3001&#39;;<br><br>export const options = {<br>  vus: 1,<br>  iterations: 5,<br>  thresholds: {<br>    http_req_duration: [&#39;p(95)&lt;300&#39;],<br>    http_req_failed: [&#39;rate==0&#39;],<br>  },<br>};<br><br>export default function () {<br>  const health = http.get(`${BASE_URL}/health`);<br>  check(health, { &#39;health check is 200&#39;: (r) =&gt; r.status === 200 });<br><br>  const list = http.get(`${BASE_URL}/api/tasks`);<br>  check(list, {<br>    &#39;GET /api/tasks is 200&#39;: (r) =&gt; r.status === 200,<br>    &#39;GET /api/tasks returns an array&#39;: (r) =&gt; Array.isArray(r.json()),<br>  });<br><br>  sleep(1);<br>}</pre><p><strong>Executing the test:</strong></p><pre>k6 run smoke-test.js</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sAjXw7FmzeX3wcEfMq13AA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SJOigWkFH5ehRuGm-Iu9MA.png" /></figure><h4>2. Staged load test</h4><p>Real traffic on a task app is mostly people browsing the list and occasionally opening a task. This ramps VUs up, holds, and ramps down — with thresholds that fail the run if performance drops below the bar.</p><pre>// load-test.js<br>// A realistic &quot;browsing&quot; load test: most traffic on a task app is people<br>// loading the list, occasionally fetching a single task. Ramps up, holds,<br>// ramps down, and fails the build if performance drops below the bar.<br>//<br>// Usage:<br>//   k6 run load-test.js<br>//   K6_WEB_DASHBOARD=true k6 run load-test.js   # with live local dashboard<br><br>import http from &#39;k6/http&#39;;<br>import { check, sleep } from &#39;k6&#39;;<br><br>const BASE_URL = __ENV.BASE_URL || &#39;http://localhost:3001&#39;;<br><br>export const options = {<br>  stages: [<br>    { duration: &#39;20s&#39;, target: 20 },  // ramp up to 20 VUs<br>    { duration: &#39;40s&#39;, target: 20 },  // hold at 20 VUs<br>    { duration: &#39;20s&#39;, target: 50 },  // ramp up to 50 VUs<br>    { duration: &#39;40s&#39;, target: 50 },  // hold at 50 VUs<br>    { duration: &#39;20s&#39;, target: 0 },   // ramp down<br>  ],<br>  thresholds: {<br>    http_req_duration: [&#39;p(90)&lt;400&#39;, &#39;p(95)&lt;700&#39;],<br>    http_req_failed: [&#39;rate&lt;0.01&#39;],   // fewer than 1% of requests may fail<br>    checks: [&#39;rate&gt;0.99&#39;],<br>  },<br>};<br><br>export default function () {<br>  const listRes = http.get(`${BASE_URL}/api/tasks`, {<br>    tags: { name: &#39;ListTasks&#39; },<br>  });<br><br>  const ok = check(listRes, {<br>    &#39;list status is 200&#39;: (r) =&gt; r.status === 200,<br>    &#39;list body is an array&#39;: (r) =&gt; Array.isArray(r.json()),<br>  });<br><br>  if (ok) {<br>    const tasks = listRes.json();<br>    if (tasks.length &gt; 0) {<br>      const randomTask = tasks[Math.floor(Math.random() * tasks.length)];<br>      const detailRes = http.get(`${BASE_URL}/api/tasks/${randomTask.id}`, {<br>        tags: { name: &#39;GetTaskById&#39; },<br>      });<br>      check(detailRes, { &#39;detail status is 200&#39;: (r) =&gt; r.status === 200 });<br>    }<br>  }<br><br>  sleep(Math.random() * 2 + 1); // 1-3s &quot;think time&quot; between actions<br>}</pre><p><strong>Executing the test:</strong></p><pre>k6 run load-test.js</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eNQIAxVKivMx4w9uuMuteg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XMja_bNKuPUBuPoAhldi_A.png" /></figure><p>Result from a shortened verification run (20 VUs, 15s): all 471 checks passed, p(95) response time was ~34ms, 0% failed requests.</p><h4>3. Full CRUD user-journey test</h4><p>This is the most realistic script: each iteration simulates a real session — view the list, create a task, read it back, toggle it complete, rename it, then delete it — using group() so the summary breaks results down step by step.</p><pre>// crud-scenario.js<br>// Simulates a real user session: view the list, create a task, read it back,<br>// toggle it complete, update its title, then delete it. Uses groups so the<br>// end-of-test summary breaks down timing per step, not just overall.<br>//<br>// Usage:<br>//   k6 run crud-scenario.js<br><br>import http from &#39;k6/http&#39;;<br>import { check, group, sleep } from &#39;k6&#39;;<br><br>const BASE_URL = __ENV.BASE_URL || &#39;http://localhost:3001&#39;;<br><br>// Small local helper so this script has no external dependencies<br>// (no need to reach jslib.k6.io, which some corporate networks block).<br>function randomString(length) {<br>  const chars = &#39;abcdefghijklmnopqrstuvwxyz0123456789&#39;;<br>  let out = &#39;&#39;;<br>  for (let i = 0; i &lt; length; i++) {<br>    out += chars.charAt(Math.floor(Math.random() * chars.length));<br>  }<br>  return out;<br>}<br><br>export const options = {<br>  vus: 10,<br>  duration: &#39;1m&#39;,<br>  thresholds: {<br>    http_req_duration: [&#39;p(95)&lt;500&#39;],<br>    http_req_failed: [&#39;rate&lt;0.01&#39;],<br>    checks: [&#39;rate&gt;0.98&#39;],<br>  },<br>};<br><br>export default function () {<br>  let taskId;<br><br>  group(&#39;1. View task list&#39;, function () {<br>    const res = http.get(`${BASE_URL}/api/tasks`, { tags: { name: &#39;ListTasks&#39; } });<br>    check(res, { &#39;list status 200&#39;: (r) =&gt; r.status === 200 });<br>  });<br><br>  group(&#39;2. Create a task&#39;, function () {<br>    const payload = JSON.stringify({<br>      title: `Load test task ${randomString(6)}`,<br>      description: &#39;Created by k6 crud-scenario.js&#39;,<br>    });<br>    const res = http.post(`${BASE_URL}/api/tasks`, payload, {<br>      headers: { &#39;Content-Type&#39;: &#39;application/json&#39; },<br>      tags: { name: &#39;CreateTask&#39; },<br>    });<br>    const ok = check(res, {<br>      &#39;create status 201&#39;: (r) =&gt; r.status === 201,<br>      &#39;create returns id&#39;: (r) =&gt; r.json(&#39;id&#39;) !== undefined,<br>    });<br>    if (ok) taskId = res.json(&#39;id&#39;);<br>  });<br><br>  if (!taskId) return; // nothing left to do this iteration if creation failed<br><br>  group(&#39;3. Read the created task&#39;, function () {<br>    const res = http.get(`${BASE_URL}/api/tasks/${taskId}`, {<br>      tags: { name: &#39;GetTaskById&#39; },<br>    });<br>    check(res, { &#39;get status 200&#39;: (r) =&gt; r.status === 200 });<br>  });<br><br>  group(&#39;4. Toggle task complete&#39;, function () {<br>    const res = http.post(`${BASE_URL}/api/tasks/${taskId}/toggle`, null, {<br>      tags: { name: &#39;ToggleTask&#39; },<br>    });<br>    check(res, { &#39;toggle status 200&#39;: (r) =&gt; r.status === 200 });<br>  });<br><br>  group(&#39;5. Update task title&#39;, function () {<br>    const payload = JSON.stringify({ title: &#39;Updated by k6&#39; });<br>    const res = http.put(`${BASE_URL}/api/tasks/${taskId}`, payload, {<br>      headers: { &#39;Content-Type&#39;: &#39;application/json&#39; },<br>      tags: { name: &#39;UpdateTask&#39; },<br>    });<br>    check(res, { &#39;update status 200&#39;: (r) =&gt; r.status === 200 });<br>  });<br><br>  group(&#39;6. Delete the task&#39;, function () {<br>    const res = http.del(`${BASE_URL}/api/tasks/${taskId}`, null, {<br>      tags: { name: &#39;DeleteTask&#39; },<br>    });<br>    check(res, { &#39;delete status 204&#39;: (r) =&gt; r.status === 204 });<br>  });<br><br>  sleep(1);<br>}</pre><p><strong>Executing the test:</strong></p><p>let’s execute the load test using k6 with the built-in Web Dashboard enabled. k6 can also generate an HTML report that you can review after the test completes.</p><p>Run the following command to enable the Web Dashboard, export the test results to an HTML report named report.html, and execute the crud-scenario.js test script.</p><pre># linux<br>K6_WEB_DASHBOARD=true K6_WEB_DASHBOARD_EXPORT=report.html k6 run crud-scenario.js<br># windows<br>set K6_WEB_DASHBOARD=true&amp;&amp;set K6_WEB_DASHBOARD_EXPORT=report.html&amp;&amp;k6 run crud-scenario.js</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aa4SjMRZVrWl86227R8zlg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rXRA1vVMhdVXgq6p-WwhMg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7ARHScw8yxw3zydn3GlRMA.png" /></figure><h4>4. Spike test</h4><p>Simulates a sudden burst (a post going viral, a flash sale) and checks recovery once it subsides.</p><pre>// spike-test.js<br>// Simulates a sudden burst of traffic (e.g. app got shared somewhere) and<br>// checks how quickly the API recovers once the burst subsides.<br>//<br>// Usage:<br>//   k6 run spike-test.js<br><br>import http from &#39;k6/http&#39;;<br>import { check, sleep } from &#39;k6&#39;;<br><br>const BASE_URL = __ENV.BASE_URL || &#39;http://localhost:3001&#39;;<br><br>export const options = {<br>  stages: [<br>    { duration: &#39;10s&#39;, target: 10 },   // normal baseline traffic<br>    { duration: &#39;5s&#39;, target: 200 },   // sudden spike to 200 VUs<br>    { duration: &#39;20s&#39;, target: 200 },  // hold the spike<br>    { duration: &#39;10s&#39;, target: 10 },   // drop back to baseline<br>    { duration: &#39;15s&#39;, target: 10 },   // observe recovery<br>    { duration: &#39;5s&#39;, target: 0 },<br>  ],<br>  thresholds: {<br>    http_req_failed: [&#39;rate&lt;0.05&#39;],  // allow a slightly higher error budget during the spike<br>  },<br>};<br><br>export default function () {<br>  const res = http.get(`${BASE_URL}/api/tasks`, { tags: { name: &#39;ListTasks&#39; } });<br>  check(res, { &#39;status is 200&#39;: (r) =&gt; r.status === 200 });<br>  sleep(0.5);<br>}</pre><p><strong>Executing the test:</strong></p><pre>k6 run spike-test.js</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cE4MbcBGvYaa88vdN6apOQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b6V-abaY42X07bAugeBwSQ.png" /></figure><h4>5. Bringing It Into CI/CD</h4><p>Once thresholds are dialed in, the payoff is automating the run. A typical GitHub Actions job installs k6, starts the backend (or points at a staging URL), runs the script, and lets the configured thresholds decide pass/fail — so a change that doubles response times fails the build instead of reaching production. Start with generous thresholds, gather a few baseline runs, then tighten them as you learn what “normal” looks like for your app.</p><pre># .github/workflows/perf.yml (sketch)<br>- name: Start backend<br>  run: cd backend &amp;&amp; npm ci &amp;&amp; npm start &amp;<br>- name: Wait for API<br>  run: npx wait-on http://localhost:3001/health<br>- name: Run k6 load test<br>  uses: grafana/k6-action@v0.3.1<br>  with:<br>    filename: k6-tests/load-test.js</pre><h4>6. Export to Grafana via Prometheus</h4><p>For tracking trends across runs over time rather than one-off summaries. Point Grafana at that Prometheus instance and build (or import) a dashboard to compare response times release over release.</p><pre>k6 run --out experimental-prometheus-rw load-test.js</pre><h4>Quick Reference (Commands)</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/927/1*L-2_9qnKmBTZFsFxZNjWFQ.png" /></figure><p><em>Thanks for reading this article !!</em></p><p>If you enjoyed this article, please click on the clap button 👏 and share to help others find it!</p><p>The full source code for this tutorial can be found here,</p><p><a href="https://github.com/codemaker2015/k6-experiments">GitHub - codemaker2015/k6-experiments</a></p><h4><strong>Resources</strong></h4><p><a href="https://grafana.com/docs/k6/latest/">Grafana k6 | Grafana k6 documentation</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=55b165cd9bd8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building an Intelligent GraphRAG Pipeline with LangExtract, Neo4j, Qdrant, and Ollama]]></title>
            <link>https://codemaker2016.medium.com/building-an-intelligent-graphrag-pipeline-with-langextract-neo4j-qdrant-and-ollama-d8f4ecbb2b75?source=rss-b7a60e472dfc------2</link>
            <guid isPermaLink="false">https://medium.com/p/d8f4ecbb2b75</guid>
            <category><![CDATA[knowledge-graph]]></category>
            <category><![CDATA[graphrag]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[neo4j]]></category>
            <category><![CDATA[generative-ai-use-cases]]></category>
            <dc:creator><![CDATA[Vishnu Sivan]]></dc:creator>
            <pubDate>Mon, 20 Jul 2026 15:28:03 GMT</pubDate>
            <atom:updated>2026-07-20T15:28:03.520Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u_FCI3BPpBLvauNaBiSx6Q.png" /></figure><p>Large Language Models have dramatically changed how we interact with enterprise data. Instead of manually searching through documents, users can simply ask questions in natural language and receive contextual answers. This capability is made possible through Retrieval-Augmented Generation (RAG), which allows an LLM to retrieve relevant information from a knowledge base before generating a response.</p><p>While RAG has become the standard architecture for enterprise AI assistants, the model could retrieve semantically similar documents, but it often failed to understand how pieces of information were related. This became especially evident when answering questions that required reasoning across multiple documents or understanding relationships between entities. To overcome these limitations, Neo4j introduced Neo4J GraphRAG. The results were promising because representing information as a graph significantly improved contextual understanding.</p><p>In this article, I will explain the concepts behind GraphRAG, discuss why it outperforms traditional RAG in many scenarios, introduce each component of the architecture, and finally walk through the implementation and my observations.</p><h3>Getting Started</h3><h4>Table of contents</h4><ul><li>Understanding Traditional RAG</li><li>What is GraphRAG?</li><li>Introducing LangExtract</li><li>Why Neo4j?</li><li>Why Qdrant?</li><li>Running Everything Locally with Ollama</li><li>Experimenting with Neo4j GraphRAG library</li><li>Building the GraphRAG Pipeline — From Documents to Intelligent Question Answering</li></ul><h4>Understanding Traditional RAG</h4><p>Imagine an organization with thousands of PDF documents, technical manuals, meeting notes, and knowledge base articles. Since Large Language Models don’t inherently know about this private information, the documents are first divided into smaller chunks. Each chunk is converted into a numerical representation called an embedding and stored in a vector database.</p><p>Whenever a user asks a question, the same embedding process is applied to the query. The vector database searches for chunks that are semantically similar to the question, and those chunks are provided to the LLM as context for generating an answer.</p><p>As the knowledge base grows larger, several limitations begin to emerge.</p><p>Suppose your organization stores information like this:</p><blockquote><em>Alice joined Company A in 2022.</em></blockquote><blockquote><em>Company A acquired Startup B in 2023.</em></blockquote><blockquote><em>Startup B developed Product X.</em></blockquote><p>Now imagine asking:</p><blockquote><em>Which product is associated with Alice’s company?</em></blockquote><p>The answer requires connecting three separate facts.</p><p>A vector database may retrieve one or two of these documents, but there is no guarantee it will retrieve all the information required to establish the relationship. Since embeddings represent semantic similarity rather than explicit connections, the model may miss important context or even hallucinate an answer.</p><p>This problem becomes even more pronounced in domains such as healthcare, finance, legal documentation, cybersecurity, and enterprise knowledge management, where understanding relationships is just as important as understanding individual facts.</p><p>This is precisely where GraphRAG becomes valuable.</p><h4>What is GraphRAG?</h4><p>GraphRAG extends the traditional Retrieval-Augmented Generation pipeline by introducing a structured knowledge graph alongside semantic retrieval. Instead of viewing documents as isolated chunks of text, GraphRAG identifies important entities — such as people, organizations, products, locations, or events — and stores the relationships between them in a graph database.</p><p>Rather than asking only, “Which document is similar to this question?”, the system can also ask:</p><ul><li>Which entities are connected?</li><li>What relationships exist between them?</li><li>Can multiple relationships be traversed to answer the query?</li><li>Are there additional connected entities that should be included as context?</li></ul><p>In other words, GraphRAG transforms unstructured text into structured knowledge that can be explored and reasoned over. This allows the retrieval process to become significantly more intelligent.</p><p>At a high level, GraphRAG combines two different retrieval strategies.</p><p>The first is semantic search, which retrieves documents based on meaning using vector embeddings.</p><p>The second is graph traversal, which discovers related entities and relationships inside a knowledge graph.</p><p>When a user submits a question, both retrieval mechanisms work together. The semantic search identifies relevant documents, while the graph expands the context by exploring connected entities and relationships. The combined information is then supplied to the language model for answer generation.</p><p>This hybrid retrieval strategy allows the model to answer questions that would otherwise require manually connecting information spread across multiple documents.</p><h4>Introducing LangExtract</h4><p>Most RAG systems simply split documents into chunks and generate embeddings. LangExtract goes a step further by analyzing the document and identifying structured information hidden inside natural language.</p><p>It extracts entities such as people, organizations, products, technologies, and locations. It also identifies relationships between these entities, along with their attributes and supporting evidence from the source text.</p><p>For example, given the sentence:</p><blockquote><em>“John joined Microsoft in 2021 and currently leads the Azure AI team.”</em></blockquote><p>LangExtract can identify:</p><ul><li>John as a Person</li><li>Microsoft as an Organization</li><li>Azure AI as a Team</li></ul><p>and generate relationships such as:</p><ul><li>John WORKS_AT Microsoft</li><li>John LEADS Azure AI</li></ul><p>Instead of storing raw text, we now have structured knowledge that can be inserted directly into a graph database. This structured representation becomes the foundation of GraphRAG.</p><h4>Why Neo4j?</h4><p>Neo4j is one of the most widely used graph databases because it is specifically designed to represent connected information.</p><p>Unlike relational databases, where relationships are computed through joins, Neo4j stores relationships as first-class citizens. Traversing a graph containing millions of nodes remains extremely efficient, making it ideal for GraphRAG applications.</p><p>For example, answering questions like:</p><ul><li>Which customers purchased products manufactured by Company X?</li><li>Which employee manages projects related to Generative AI?</li><li>Which research papers cite papers written by Author Y?</li></ul><p>becomes a matter of traversing relationships rather than performing complex joins or relying solely on semantic similarity.</p><h4>Why Qdrant?</h4><p>Although knowledge graphs are excellent at representing relationships, they are not designed for semantic similarity search. Users typically ask questions in natural language rather than using exact entity names. This is where vector databases continue to play an essential role.</p><p>Qdrant stores embeddings for document chunks and performs high-speed nearest-neighbor searches to retrieve semantically relevant information. Instead of replacing vector search, GraphRAG complements it.</p><p>The architecture retrieves semantically similar chunks from Qdrant while simultaneously exploring related entities in Neo4j. The retrieved information is merged to create a richer and more complete context for the language model.</p><h4>Running Everything Locally with Ollama</h4><p>Ollama simplifies deploying open-source language models such as Qwen, Llama, Gemma, Mistral, and many others without relying on cloud APIs.</p><p>For organizations dealing with sensitive or confidential information, this provides several benefits:</p><ul><li>Data never leaves the local environment.</li><li>There are no API usage costs.</li><li>Models can be switched easily depending on hardware and performance requirements.</li><li>The entire GraphRAG pipeline remains self-hosted.</li></ul><p>This makes the solution particularly attractive for enterprise deployments.</p><h3>Experimenting with Neo4j GraphRAG library</h3><p>Before jumping to the prototype, let’s first explore the Neo4j GraphRAG library to understand how it helps developers build Knowledge Graphs and perform Graph-based Retrieval-Augmented Generation (GraphRAG).</p><p>The official Neo4j GraphRAG package for Python enables developers to build GraphRAG applications using the power of Neo4j and Python. As a first-party library, it offers a robust, feature-rich, and high-performance solution, along with long-term support and maintenance directly from Neo4j.</p><p><strong>Prerequisites</strong></p><ul><li><strong>Python 3.10+</strong></li><li><strong>Neo4j</strong> — either <a href="https://neo4j.com/download/">Neo4j Desktop</a> or a running Neo4j instance (local Docker container or AuraDB)</li><li><strong>Ollama</strong> — installed and running locally (<a href="https://ollama.com">ollama.com</a>)</li><li><strong>Docker </strong>— running qdrant locally. (<a href="https://www.docker.com/products/docker-desktop/">Docker Desktop</a>)</li></ul><p><strong>Step1: Install the Required Libraries</strong></p><p>Install the Neo4j GraphRAG package with the required OpenAI dependencies.</p><pre>pip install &quot;neo4j-graphrag[openai]&quot; python-dotenv</pre><p><strong>Step 2: Create the .env file</strong></p><p>Create a new file named .env in the root directory of your project and add the following configuration:</p><pre># Neo4j Desktop — Default Local Instance<br>NEO4J_URI=bolt://localhost:7687<br>NEO4J_USERNAME=neo4j<br>NEO4J_PASSWORD=your_neo4j_password<br><br># OpenAI API Key<br>OPENAI_API_KEY=your_openai_api_key</pre><p><strong>Step 3: Knowledge Graph Construction</strong></p><p>The Pipeline class provides extensive customization options, making it suitable for advanced use cases. For simpler implementations, the SimpleKGPipeline class offers an abstraction over the Pipeline class, making it easier to build Knowledge Graphs.</p><pre>import asyncio<br>import os<br><br>from neo4j import GraphDatabase<br>from neo4j_graphrag.embeddings import OpenAIEmbeddings<br>from neo4j_graphrag.experimental.pipeline.kg_builder import SimpleKGPipeline<br>from neo4j_graphrag.llm import OpenAILLM<br>from neo4j_graphrag.indexes import create_vector_index, upsert_vectors<br>from neo4j_graphrag.types import EntityType<br>from neo4j_graphrag.generation import GraphRAG<br>from neo4j_graphrag.retrievers import VectorRetriever<br><br>from dotenv import load_dotenv, find_dotenv<br>load_dotenv(find_dotenv())<br><br><br>NEO4J_URI = os.getenv(&quot;NEO4J_URI&quot;)<br>NEO4J_USERNAME = os.getenv(&quot;NEO4J_USERNAME&quot;)<br>NEO4J_PASSWORD = os.getenv(&quot;NEO4J_PASSWORD&quot;)<br><br># Connect to the Neo4j database<br>driver = GraphDatabase.driver(NEO4J_URI, auth=(NEO4J_USERNAME, NEO4J_PASSWORD))<br><br># List the entities and relations the LLM should look for in the text<br>node_types = [&quot;Person&quot;, &quot;House&quot;, &quot;Planet&quot;]<br>relationship_types = [&quot;PARENT_OF&quot;, &quot;HEIR_OF&quot;, &quot;RULES&quot;]<br>patterns = [<br>    (&quot;Person&quot;, &quot;PARENT_OF&quot;, &quot;Person&quot;),<br>    (&quot;Person&quot;, &quot;HEIR_OF&quot;, &quot;House&quot;),<br>    (&quot;House&quot;, &quot;RULES&quot;, &quot;Planet&quot;),<br>]<br><br># Create an Embedder object<br>embedder = OpenAIEmbeddings(model=&quot;text-embedding-3-large&quot;)<br><br># Instantiate the LLM<br>llm = OpenAILLM(<br>    model_name=&quot;gpt-4.1&quot;,<br>    model_params={<br>        &quot;max_completion_tokens&quot;: 2000,<br>    },<br>)<br><br># Instantiate the SimpleKGPipeline<br>kg_builder = SimpleKGPipeline(<br>    llm=llm,<br>    driver=driver,<br>    embedder=embedder,<br>    schema={<br>        &quot;node_types&quot;: node_types,<br>        &quot;relationship_types&quot;: relationship_types,<br>        &quot;patterns&quot;: patterns,<br>    },<br>    on_error=&quot;IGNORE&quot;,<br>    from_file=False,<br>)<br><br># Run the pipeline on a piece of text<br>text = (<br>    &quot;The son of Duke Leto Atreides and the Lady Jessica, Paul is the heir of House &quot;<br>    &quot;Atreides, an aristocratic family that rules the planet Caladan.&quot;<br>)<br>asyncio.run(kg_builder.run_async(text=text))</pre><p><strong>Step 4: Creating a Vector Index</strong></p><p>After constructing the Knowledge Graph, create a vector index to enable semantic similarity search.</p><pre>INDEX_NAME = &quot;test_vector_index&quot;<br><br># Create the index<br>create_vector_index(<br>    driver,<br>    INDEX_NAME,<br>    label=&quot;Chunk&quot;,<br>    embedding_property=&quot;embedding&quot;,<br>    dimensions=3072,<br>    similarity_fn=&quot;euclidean&quot;,<br>)</pre><p><strong>Step 5: Populating the Vector Index</strong></p><p>Generate embeddings for your content and store them in Neo4j.</p><pre># Generate an embedding for some text<br>text = (<br>    &quot;The son of Duke Leto Atreides and the Lady Jessica, Paul is the heir of House &quot;<br>    &quot;Atreides, an aristocratic family that rules the planet Caladan.&quot;<br>)<br>vector = embedder.embed_query(text)<br><br># Upsert the vector<br>upsert_vectors(<br>    driver,<br>    ids=[&quot;1234&quot;],<br>    embedding_property=&quot;vectorProperty&quot;,<br>    embeddings=[vector],<br>    entity_type=EntityType.NODE,<br>)</pre><p><strong>Step 6: Performing a Similarity Search</strong></p><p>Neo4j uses an Approximate Nearest Neighbor (ANN) algorithm for vector similarity search. The search retrieves semantically similar nodes from the vector index, which can then be passed to an LLM for Retrieval-Augmented Generation.</p><pre>retriever = VectorRetriever(driver, INDEX_NAME, embedder)<br>rag = GraphRAG(retriever=retriever, llm=llm)<br><br># Query the graph<br>query_text = &quot;Who is Paul Atreides?&quot;<br>response = rag.search(query_text=query_text, retriever_config={&quot;top_k&quot;: 5})<br>print(response.answer)<br>driver.close()</pre><p><strong>Combine everything together</strong></p><pre># Step 1: Knowledge Graph Construction<br><br>import asyncio<br>import os<br><br>from neo4j import GraphDatabase<br>from neo4j_graphrag.embeddings import OpenAIEmbeddings<br>from neo4j_graphrag.experimental.pipeline.kg_builder import SimpleKGPipeline<br>from neo4j_graphrag.llm import OpenAILLM<br>from neo4j_graphrag.indexes import create_vector_index, upsert_vectors<br>from neo4j_graphrag.types import EntityType<br>from neo4j_graphrag.generation import GraphRAG<br>from neo4j_graphrag.retrievers import VectorRetriever<br><br>from dotenv import load_dotenv, find_dotenv<br>load_dotenv(find_dotenv())<br><br><br>NEO4J_URI = os.getenv(&quot;NEO4J_URI&quot;)<br>NEO4J_USERNAME = os.getenv(&quot;NEO4J_USERNAME&quot;)<br>NEO4J_PASSWORD = os.getenv(&quot;NEO4J_PASSWORD&quot;)<br><br># Connect to the Neo4j database<br>driver = GraphDatabase.driver(NEO4J_URI, auth=(NEO4J_USERNAME, NEO4J_PASSWORD))<br><br># List the entities and relations the LLM should look for in the text<br>node_types = [&quot;Person&quot;, &quot;House&quot;, &quot;Planet&quot;]<br>relationship_types = [&quot;PARENT_OF&quot;, &quot;HEIR_OF&quot;, &quot;RULES&quot;]<br>patterns = [<br>    (&quot;Person&quot;, &quot;PARENT_OF&quot;, &quot;Person&quot;),<br>    (&quot;Person&quot;, &quot;HEIR_OF&quot;, &quot;House&quot;),<br>    (&quot;House&quot;, &quot;RULES&quot;, &quot;Planet&quot;),<br>]<br><br># Create an Embedder object<br>embedder = OpenAIEmbeddings(model=&quot;text-embedding-3-large&quot;)<br><br># Instantiate the LLM<br>llm = OpenAILLM(<br>    model_name=&quot;gpt-4.1&quot;,<br>    model_params={<br>        &quot;max_completion_tokens&quot;: 2000,<br>    },<br>)<br><br># Instantiate the SimpleKGPipeline<br>kg_builder = SimpleKGPipeline(<br>    llm=llm,<br>    driver=driver,<br>    embedder=embedder,<br>    schema={<br>        &quot;node_types&quot;: node_types,<br>        &quot;relationship_types&quot;: relationship_types,<br>        &quot;patterns&quot;: patterns,<br>    },<br>    on_error=&quot;IGNORE&quot;,<br>    from_file=False,<br>)<br><br># Run the pipeline on a piece of text<br>text = (<br>    &quot;The son of Duke Leto Atreides and the Lady Jessica, Paul is the heir of House &quot;<br>    &quot;Atreides, an aristocratic family that rules the planet Caladan.&quot;<br>)<br>asyncio.run(kg_builder.run_async(text=text))<br><br><br># Step 2: Creating a Vector Index<br><br>INDEX_NAME = &quot;test_vector_index&quot;<br><br># Create the index<br>create_vector_index(<br>    driver,<br>    INDEX_NAME,<br>    label=&quot;Chunk&quot;,<br>    embedding_property=&quot;embedding&quot;,<br>    dimensions=3072,<br>    similarity_fn=&quot;euclidean&quot;,<br>)<br><br><br># Step 3: Populating a Vector Index<br><br># Generate an embedding for some text<br>text = (<br>    &quot;The son of Duke Leto Atreides and the Lady Jessica, Paul is the heir of House &quot;<br>    &quot;Atreides, an aristocratic family that rules the planet Caladan.&quot;<br>)<br>vector = embedder.embed_query(text)<br><br># Upsert the vector<br>upsert_vectors(<br>    driver,<br>    ids=[&quot;1234&quot;],<br>    embedding_property=&quot;vectorProperty&quot;,<br>    embeddings=[vector],<br>    entity_type=EntityType.NODE,<br>)<br><br><br># Step 4: Performing a Similarity Search<br><br>retriever = VectorRetriever(driver, INDEX_NAME, embedder)<br>rag = GraphRAG(retriever=retriever, llm=llm)<br><br># Query the graph<br>query_text = &quot;Who is Paul Atreides?&quot;<br>response = rag.search(query_text=query_text, retriever_config={&quot;top_k&quot;: 5})<br>print(response.answer)<br>driver.close()</pre><p><strong>Step 7: Execute a GraphRAG Query</strong></p><p>Run the GraphRAG retrieval code to see the output as below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xlOtIDlVKn3xOPNqRFE1Mg.png" /></figure><p>Open <strong>Neo4j Desktop</strong> and run the following Cypher query to view the generated graph.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/712/1*3oaBXyoB3bkDirivQB4bBw.png" /></figure><h3>Building the GraphRAG Pipeline — From Documents to Intelligent Question Answering</h3><p>Having explored the concepts behind GraphRAG and the technologies that power it, the next step is understanding how these components come together to form a complete knowledge retrieval pipeline. This section walks through the overall implementation flow of the project, explaining how each component contributes to the final question-answering process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tFF69ULSNtJO7ntafCcwpQ.png" /></figure><h4>Pull the required Ollama models</h4><p>The app uses gemma3:latest for extraction/answering and embeddinggemma:latest for embeddings by default:</p><pre>ollama pull gemma3:latest<br>ollama pull embeddinggemma:latest</pre><p>Then make sure the Ollama server is running:</p><pre>ollama serve</pre><h4>Run Qdrant using Docker</h4><p>Execute the following command to run the qdrant locally.</p><pre>docker run -p 6333:6333 -p 6334:6334 qdrant/qdrant</pre><p>This gives you QDRANT_URL=http://localhost:6333 with no API key needed for local use.</p><p><strong>Or use Qdrant Cloud</strong> (managed, free tier available) — sign up at <a href="https://cloud.qdrant.io">cloud.qdrant.io</a>, create a cluster, and grab the URL + API key from the dashboard.</p><p>Then add to your .env:</p><pre>QDRANT_URL=http://localhost:6333<br>QDRANT_KEY=your_api_key_here_or_blank_for_local</pre><h4>Installing dependencies</h4><p>Install the python dependencies required for the handson.</p><pre>pip install neo4j neo4j_graphrag ollama langextract qdrant-client streamlit python-dotenv</pre><h4><strong>Importing libraries</strong></h4><p>Import the necessary library helpers, the Ollama and Neo4j clients, Neo4j’s GraphRAG toolkit, and LangExtract for LLM-based extraction.</p><pre>import os<br>import uuid<br>import textwrap<br><br>import ollama<br>from dotenv import load_dotenv, find_dotenv<br>from ollama import chat<br>from neo4j import GraphDatabase<br>from qdrant_client import QdrantClient, models<br>from neo4j_graphrag.retrievers import QdrantNeo4jRetriever<br><br>import langextract as lx<br>from langextract.providers import ollama as lx_ollama<br><br>load_dotenv(find_dotenv())</pre><h4><strong>__init__</strong></h4><p>This constructor wires up all the pipeline’s dependencies in one place, so the rest of the class can assume Neo4j, Ollama, and model config are all ready to go.</p><pre>  def __init__(self,env_path: str = &quot;.env&quot;,ollama_model_extract: str = &quot;gemma3:latest&quot;,<br>          ollama_model_answer: str = &quot;gemma3:latest&quot;,ollama_embedding_model: str = &quot;embeddinggemma:latest&quot;,<br>          ollama_host: str | None = None,vector_dimension: int = 768,<br>  ):<br><br>      load_dotenv(env_path)<br><br>      self.qdrant_key = os.getenv(&quot;QDRANT_KEY&quot;)<br>      self.qdrant_url = os.getenv(&quot;QDRANT_URL&quot;)<br>      self.neo4j_uri = os.getenv(&quot;NEO4J_URI&quot;)<br>      self.neo4j_username = os.getenv(&quot;NEO4J_USERNAME&quot;)<br>      self.neo4j_password = os.getenv(&quot;NEO4J_PASSWORD&quot;)<br><br>      self.neo4j_driver = GraphDatabase.driver(<br>          self.neo4j_uri, auth=(self.neo4j_username, self.neo4j_password)<br>      )<br><br>      self.qdrant_client = QdrantClient(<br>          url=self.qdrant_url,<br>          api_key=self.qdrant_key,<br>      )<br><br>      # Ollama client for local embeddings (embeddinggemma:latest)<br>      self.ollama_client = ollama.Client(host=ollama_host) if ollama_host else ollama.Client()<br><br>      # langextract needs a plain URL string, not a client object<br>      self.ollama_url = ollama_host or os.environ.get(&quot;OLLAMA_HOST&quot;, &quot;http://localhost:11434&quot;)<br><br>      # Model / config knobs<br>      self.ollama_model_extract = ollama_model_extract<br>      self.ollama_model_answer = ollama_model_answer<br>      self.ollama_embedding_model = ollama_embedding_model<br>      self.vector_dimension = vector_dimension</pre><h4><strong>extract_graph_components</strong></h4><p>This is the “graph construction” step — turning unstructured clinical text into structured medication data via an LLM, using LangExtract’s few-shot extraction framework.</p><pre>  def extract_graph_components(self, raw_data: str):<br>      &quot;&quot;&quot;Extract medication entities and relationships using langextract + Ollama.&quot;&quot;&quot;<br><br>      prompt_description = textwrap.dedent(&quot;&quot;&quot;<br>          Extract medications with their details, using attributes to group related information:<br>          1. Extract entities in the order they appear in the text<br>          2. Each entity must have a &#39;medication_group&#39; attribute linking it to its medication<br>          3. All details about a medication should share the same medication_group value<br>          &quot;&quot;&quot;).strip()<br><br>      examples = [<br>          lx.data.ExampleData(<br>              text=(<br>                  &quot;Patient takes Aspirin 100mg daily for heart health and&quot;<br>                  &quot; Simvastatin 20mg at bedtime.&quot;<br>              ),<br>              extractions=[<br>                  lx.data.Extraction(<br>                      extraction_class=&quot;medication&quot;,<br>                      extraction_text=&quot;Aspirin&quot;,<br>                      attributes={&quot;medication_group&quot;: &quot;Aspirin&quot;},<br>                  ),<br>                  lx.data.Extraction(<br>                      extraction_class=&quot;dosage&quot;,<br>                      extraction_text=&quot;100mg&quot;,<br>                      attributes={&quot;medication_group&quot;: &quot;Aspirin&quot;},<br>                  ),<br>                  lx.data.Extraction(<br>                      extraction_class=&quot;frequency&quot;,<br>                      extraction_text=&quot;daily&quot;,<br>                      attributes={&quot;medication_group&quot;: &quot;Aspirin&quot;},<br>                  ),<br>                  lx.data.Extraction(<br>                      extraction_class=&quot;condition&quot;,<br>                      extraction_text=&quot;heart health&quot;,<br>                      attributes={&quot;medication_group&quot;: &quot;Aspirin&quot;},<br>                  ),<br>                  lx.data.Extraction(<br>                      extraction_class=&quot;medication&quot;,<br>                      extraction_text=&quot;Simvastatin&quot;,<br>                      attributes={&quot;medication_group&quot;: &quot;Simvastatin&quot;},<br>                  ),<br>                  lx.data.Extraction(<br>                      extraction_class=&quot;dosage&quot;,<br>                      extraction_text=&quot;20mg&quot;,<br>                      attributes={&quot;medication_group&quot;: &quot;Simvastatin&quot;},<br>                  ),<br>                  lx.data.Extraction(<br>                      extraction_class=&quot;frequency&quot;,<br>                      extraction_text=&quot;at bedtime&quot;,<br>                      attributes={&quot;medication_group&quot;: &quot;Simvastatin&quot;},<br>                  ),<br>              ],<br>          )<br>      ]<br><br>      result = lx.extract(<br>          text_or_documents=raw_data,<br>          prompt_description=prompt_description,<br>          examples=examples,<br>          model_id=self.ollama_model_extract,<br>          model_url=self.ollama_url,<br>          resolver_params={&quot;format_handler&quot;: lx_ollama.OLLAMA_FORMAT_HANDLER},<br>          max_char_buffer=4000,<br>          show_progress=True,<br>      )<br><br>      return self._convert_extractions_to_graph(result.extractions)</pre><h4><strong>_convert_extractions_to_graph</strong></h4><p>It groups extractions by medication_group. Within each group, the extraction tagged &quot;medication&quot; becomes the anchor node (e.g., &quot;Lisinopril&quot;). Every other extraction in that group (dosage, frequency, condition) becomes its own node, connected to the anchor via a relationship labeled with its extraction class (e.g., Lisinopril —dosage→ 10mg). It returns nodes: {name: uuid} and relationships: [{source, target, type}, ...].</p><pre>    def _convert_extractions_to_graph(self, extractions: list):<br>      &quot;&quot;&quot;Convert langextract&#39;s flat, grouped extractions into (nodes, relationships).&quot;&quot;&quot;<br><br>      groups: dict[str, list] = {}<br>      for ext in extractions:<br>          if not ext.attributes or &quot;medication_group&quot; not in ext.attributes:<br>              continue<br>          group_name = ext.attributes[&quot;medication_group&quot;]<br>          groups.setdefault(group_name, []).append(ext)<br><br>      nodes: dict[str, str] = {}<br>      relationships: list[dict] = []<br><br>      for group_name, group_extractions in groups.items():<br>          anchor_ext = next(<br>              (e for e in group_extractions if e.extraction_class == &quot;medication&quot;),<br>              None,<br>          )<br><br>          # Fall back to the group name itself if no explicit &quot;medication&quot;<br>          # extraction was found in this group, so we still get an anchor.<br>          anchor_text = anchor_ext.extraction_text if anchor_ext else group_name<br><br>          if anchor_text not in nodes:<br>              nodes[anchor_text] = str(uuid.uuid4())<br><br>          for ext in group_extractions:<br>              if ext is anchor_ext:<br>                  continue<br><br>              target_text = ext.extraction_text<br>              if target_text not in nodes:<br>                  nodes[target_text] = str(uuid.uuid4())<br><br>              relationships.append(<br>                  {<br>                      &quot;source&quot;: nodes[anchor_text],<br>                      &quot;target&quot;: nodes[target_text],<br>                      &quot;type&quot;: ext.extraction_class,<br>                  }<br>              )<br><br>      return nodes, relationships</pre><h4><strong>ingest_to_neo4j</strong></h4><p>For each node, it computes an embedding via self.embedder.embed_query(name) and writes it into Neo4j with a Cypher CREATE. For each relationship, it runs a MATCH ... CREATE to link the two nodes by ID. The relationship type is inserted via an f-string because Cypher doesn&#39;t allow relationship types to be passed as query parameters — _sanitize_relationship_type strips it to safe alphanumeric/underscore uppercase text first, specifically to avoid Cypher injection.</p><pre>  def ingest_to_neo4j(self, nodes: dict, relationships: list):<br>      with self.neo4j_driver.session() as session:<br>          for name, node_id in nodes.items():<br>              embedding = self.embedder.embed_query(name)<br>              session.run(<br>                  &quot;CREATE (n:Entity {id: $id, name: $name, embedding: $embedding})&quot;,<br>                  id=node_id,<br>                  name=name,<br>                  embedding=embedding,<br>              )<br><br>          for relationship in relationships:<br>              rel_type = self._sanitize_relationship_type(relationship[&quot;type&quot;])<br>              session.run(<br>                  &quot;MATCH (a:Entity {id: $source_id}), (b:Entity {id: $target_id}) &quot;<br>                  f&quot;CREATE (a)-[:{rel_type} {{type: $type}}]-&gt;(b)&quot;,<br>                  source_id=relationship[&quot;source&quot;],<br>                  target_id=relationship[&quot;target&quot;],<br>                  type=relationship[&quot;type&quot;],<br>              )<br><br>      return nodes<br><br>@staticmethod<br>def _sanitize_relationship_type(raw_type: str) -&gt; str:<br>    safe = &quot;&quot;.join(ch if ch.isalnum() else &quot;_&quot; for ch in raw_type.strip())<br>    safe = safe.upper().strip(&quot;_&quot;) or &quot;RELATIONSHIP&quot;<br>    if safe[0].isdigit():<br>        safe = f&quot;REL_{safe}&quot;<br>    return safe</pre><h4>create_collection</h4><p>This is Qdrant’s equivalent of create_vector_index from the Neo4j-only version — it provisions the vector storage, just in a different database.</p><p>It tries to fetch the collection by name; if that fails with a “not found”-style error, it creates a new Qdrant collection sized to vector_dimension (768, matching embeddinggemma) using cosine distance.</p><pre>    def create_collection(self, collection_name: str, vector_dimension: int = None):<br>        vector_dimension = vector_dimension or self.vector_dimension<br><br>        try:<br>            # Try to fetch the collection status<br>            self.qdrant_client.get_collection(collection_name)<br>            print(f&quot;Skipping creating collection; &#39;{collection_name}&#39; already exists.&quot;)<br>        except Exception as e:<br>            # If collection does not exist, an error will be thrown, so we create the collection<br>            if &quot;Not found: Collection&quot; in str(e) or &quot;doesn&#39;t exist&quot; in str(e) or &quot;404&quot; in str(e):<br>                print(f&quot;Collection &#39;{collection_name}&#39; not found. Creating it now...&quot;)<br><br>                self.qdrant_client.create_collection(<br>                    collection_name=collection_name,<br>                    vectors_config=models.VectorParams(<br>                        size=vector_dimension, distance=models.Distance.COSINE<br>                    ),<br>                )<br><br>                print(f&quot;Collection &#39;{collection_name}&#39; created successfully.&quot;)<br>            else:<br>                print(f&quot;Error while checking collection: {e}&quot;)</pre><h4>ingest_to_qdrant and ollama_embeddings</h4><p>A small standalone method replacing the OllamaEmbedder class from the Neo4j-only version — since Qdrant&#39;s client and retriever don&#39;t require the neo4j_graphrag Embedder interface, a plain function is enough.</p><p>For every node name, it computes an embedding via ollama_embeddings, then upserts all of them into the Qdrant collection as points, each carrying a payload of {&quot;id&quot;: &lt;neo4j_node_id&gt;, &quot;name&quot;: &lt;node_name&gt;} — this payload is what lets the retriever map a Qdrant hit back to its Neo4j node.</p><pre>  def ollama_embeddings(self, text: str) -&gt; list[float]:<br>      response = self.ollama_client.embeddings(<br>          model=self.ollama_embedding_model,<br>          prompt=text,<br>      )<br>      return response[&quot;embedding&quot;]<br><br>  def ingest_to_qdrant(self, collection_name: str, raw_data: str, node_id_mapping: dict):<br>      names = list(node_id_mapping.keys())<br>      embeddings = [self.ollama_embeddings(name) for name in names]<br><br>      self.qdrant_client.upsert(<br>          collection_name=collection_name,<br>          points=[<br>              {<br>                  &quot;id&quot;: str(uuid.uuid4()),<br>                  &quot;vector&quot;: embedding,<br>                  &quot;payload&quot;: {&quot;id&quot;: node_id_mapping[name], &quot;name&quot;: name},<br>              }<br>              for name, embedding in zip(names, embeddings)<br>          ],<br>      )</pre><h4>retriever_search</h4><p>It builds a QdrantNeo4jRetriever, telling it which property (id) links records between the two databases (id_property_external for Qdrant, id_property_neo4j for Neo4j). Unlike the Neo4j-only retriever, which embeds the query text internally, here the query text is embedded manually via ollama_embeddings(query) and passed in as query_vector.</p><pre>  def retriever_search(self, collection_name: str, query: str, top_k: int = 5):<br>      retriever = QdrantNeo4jRetriever(<br>          driver=self.neo4j_driver,<br>          client=self.qdrant_client,<br>          collection_name=collection_name,<br>          id_property_external=&quot;id&quot;,<br>          id_property_neo4j=&quot;id&quot;,<br>      )<br><br>      results = retriever.search(<br>          query_vector=self.ollama_embeddings(query), top_k=top_k<br>      )<br><br>      return results</pre><h4>fetch_related_graph / format_graph_context</h4><p>Once relevant nodes are found, these two methods expand that into a richer subgraph and flatten it into something an LLM can read as plain text.</p><pre>    def fetch_related_graph(self, entity_ids: list):<br>      query = &quot;&quot;&quot;<br>      MATCH (e:Entity)-[r1]-(n1)-[r2]-(n2)<br>      WHERE e.id IN $entity_ids<br>      RETURN e, r1 as r, n1 as related, r2, n2<br>      UNION<br>      MATCH (e:Entity)-[r]-(related)<br>      WHERE e.id IN $entity_ids<br>      RETURN e, r, related, null as r2, null as n2<br>      &quot;&quot;&quot;<br>      with self.neo4j_driver.session() as session:<br>          result = session.run(query, entity_ids=entity_ids)<br>          subgraph = []<br>          for record in result:<br>              subgraph.append(<br>                  {<br>                      &quot;entity&quot;: record[&quot;e&quot;],<br>                      &quot;relationship&quot;: record[&quot;r&quot;],<br>                      &quot;related_node&quot;: record[&quot;related&quot;],<br>                  }<br>              )<br>              if record[&quot;r2&quot;] and record[&quot;n2&quot;]:<br>                  subgraph.append(<br>                      {<br>                          &quot;entity&quot;: record[&quot;related&quot;],<br>                          &quot;relationship&quot;: record[&quot;r2&quot;],<br>                          &quot;related_node&quot;: record[&quot;n2&quot;],<br>                      }<br>                  )<br>      return subgraph<br><br>  def format_graph_context(self, subgraph: list):<br>      nodes = set()<br>      edges = []<br><br>      for entry in subgraph:<br>          entity = entry[&quot;entity&quot;]<br>          related = entry[&quot;related_node&quot;]<br>          relationship = entry[&quot;relationship&quot;]<br><br>          nodes.add(entity[&quot;name&quot;])<br>          nodes.add(related[&quot;name&quot;])<br><br>          edges.append(f&quot;{entity[&#39;name&#39;]} {relationship[&#39;type&#39;]} {related[&#39;name&#39;]}&quot;)<br><br>      return {&quot;nodes&quot;: list(nodes), &quot;edges&quot;: edges}</pre><h4>graphRAG_run</h4><p>It builds a prompt embedding the flattened graph context (nodes_str, edges_str) plus the user&#39;s question, and calls ollama.chat(...) with the answer model to produce the response.</p><pre> def graphRAG_run(self, graph_context: dict, user_query: str):<br>    nodes_str = &quot;, &quot;.join(graph_context[&quot;nodes&quot;])<br>    edges_str = &quot;; &quot;.join(graph_context[&quot;edges&quot;])<br>    prompt = f&quot;&quot;&quot;<br>    You are an intelligent assistant with access to the following knowledge graph:<br><br>    Nodes: {nodes_str}<br><br>    Edges: {edges_str}<br><br>    Using this graph, Answer the following question:<br><br>    User Query: &quot;{user_query}&quot;<br>    &quot;&quot;&quot;<br><br>    try:<br>        response = chat(<br>            model=self.ollama_model_answer,<br>            messages=[<br>                {<br>                    &quot;role&quot;: &quot;system&quot;,<br>                    &quot;content&quot;: &quot;Provide the answer for the following question:&quot;,<br>                },<br>                {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: prompt},<br>            ],<br>        )<br>        return response.message.content<br><br>    except Exception as e:<br>        return f&quot;Error querying LLM: {str(e)}&quot;</pre><h4>create_and_ingest / run_pipeline / close</h4><p>These are orchestration helpers that chain the lower-level methods into complete workflows, so callers don’t have to sequence everything manually.</p><pre>  def create_and_ingest(self, raw_data: str, query: str, collection_name: str = &quot;medicationGraphRAGstore&quot;):<br>      print(&quot;Creating collection...&quot;)<br>      self.create_collection(collection_name, self.vector_dimension)<br>      print(&quot;Collection created/verified&quot;)<br><br>      print(&quot;Extracting graph components...&quot;)<br>      nodes, relationships = self.extract_graph_components(raw_data)<br>      print(&quot;Nodes:&quot;, nodes)<br>      print(&quot;Relationships:&quot;, relationships)<br><br>      print(&quot;Ingesting to Neo4j...&quot;)<br>      node_id_mapping = self.ingest_to_neo4j(nodes, relationships)<br>      print(&quot;Neo4j ingestion complete&quot;)<br><br>      print(&quot;Ingesting to Qdrant...&quot;)<br>      self.ingest_to_qdrant(collection_name, raw_data, node_id_mapping)<br>      print(&quot;Qdrant ingestion complete&quot;)<br><br>  def run_pipeline(self, raw_data: str, query: str, collection_name: str = &quot;medicationGraphRAGstore&quot;):<br><br>      # run only the first time, comment this for subsequent runs<br>      self.create_and_ingest(raw_data, query, collection_name)<br><br>      print(&quot;Starting retriever search...&quot;)<br>      retriever_result = self.retriever_search(collection_name, query)<br>      print(&quot;Retriever results:&quot;, retriever_result)<br><br>      print(&quot;Extracting entity IDs...&quot;)<br>      entity_ids = [<br>          item.content.split(&quot;&#39;id&#39;: &#39;&quot;)[1].split(&quot;&#39;&quot;)[0]<br>          for item in retriever_result.items<br>      ]<br>      print(&quot;Entity IDs:&quot;, entity_ids)<br><br>      print(&quot;Fetching related graph...&quot;)<br>      subgraph = self.fetch_related_graph(entity_ids)<br>      print(&quot;Subgraph:&quot;, subgraph)<br><br>      print(&quot;Formatting graph context...&quot;)<br>      graph_context = self.format_graph_context(subgraph)<br>      print(&quot;Graph context:&quot;, graph_context)<br><br>      print(&quot;Running GraphRAG...&quot;)<br>      answer = self.graphRAG_run(graph_context, query)<br>      print(&quot;Final Answer:&quot;, answer)<br><br>      return answer<br><br>  def close(self):<br>      self.neo4j_driver.close()</pre><h4>if __name__ == &quot;__main__&quot;:</h4><p>This block is a standalone CLI smoke test, letting you exercise the whole pipeline without going through Streamlit.</p><pre>if __name__ == &quot;__main__&quot;:<br>    print(&quot;Script started&quot;)<br><br>    graph_rag = MedicationGraphRAG()<br><br>    raw_data = textwrap.dedent(&quot;&quot;&quot;<br>    The patient is a 54-year-old man with a history of multiple chronic conditions<br>    being managed through an extensive medication regimen. He was prescribed<br>    Lisinopril, Metformin, Atorvastatin, Aspirin, Levothyroxine, and Sertraline<br>    over the course of the past year, with his treatment plan adjusted several<br>    times based on follow-up visits.<br>    <br>    He takes Lisinopril 10mg daily for hypertension, but often misses his<br>    Metformin 500mg dose which should be taken twice daily for diabetes. His<br>    cardiologist also started him on Atorvastatin 40mg at bedtime for high<br>    cholesterol after his last lipid panel showed elevated LDL levels. To reduce<br>    his risk of cardiovascular events, he was additionally prescribed Aspirin<br>    81mg daily for heart disease prevention, which he takes alongside his<br>    breakfast each morning.<br>    <br>    Following a routine thyroid screening, he was found to have an underactive<br>    thyroid and was started on Levothyroxine 75mcg every morning for<br>    hypothyroidism, to be taken on an empty stomach before any other medications.<br>    More recently, after reporting persistent low mood and difficulty sleeping<br>    during a wellness visit, his primary care physician added Sertraline 50mg<br>    daily for depression, with plans to reassess the dosage after eight weeks.<br>    <br>    Despite the number of prescriptions, the patient has had difficulty<br>    maintaining consistency with his Metformin and occasionally forgets his<br>    evening Atorvastatin dose, which his care team is now addressing through a<br>    simplified pill organizer and reminder system.<br>    &quot;&quot;&quot;).strip()<br><br>    # Sample Questions<br>    #1. &quot;What is the dosage and frequency for Lisinopril?&quot;<br>    #2. &quot;What is the dosage and frequency for Metformin?&quot;<br>    #3. &quot;Which medications does the patient take once daily versus twice daily?&quot;<br>    #4. &quot;What medication is prescribed for hypothyroidism, and at what dose?&quot;<br>    #5. &quot;List all medications related to cardiovascular conditions and their dosages.&quot;<br>    #6. &quot;How often does the patient take Aspirin?&quot;<br>    #7. &quot;What condition is Levothyroxine prescribed for?&quot;<br>    #8. &quot;What time of day should Levothyroxine be taken, and why?&quot;<br>    #9. &quot;Which medications does the patient have trouble taking consistently?&quot;<br>    #10. &quot;What is the dosage and frequency for Sertraline?&quot;<br><br>    query = &quot;List all medications related to cardiovascular conditions and their dosages.&quot;<br><br>    answer = graph_rag.run_pipeline(raw_data, query, collection_name=&quot;medicationGraphRAGstore&quot;)<br><br>    graph_rag.close()</pre><h4>Building streamlit interface</h4><p>MedGraphRAG’s Streamlit app provides the web interface for the medication Graph RAG pipeline defined in app.py. It lets a user configure Neo4j and Ollama model settings from a sidebar, ingest clinical text into a knowledge graph with live progress feedback, and then query that graph — retrieving relevant nodes, expanding them into a subgraph, and generating an LLM answer grounded in that context, all without touching code.</p><pre>#streamlit_app.py<br><br>import streamlit as st<br>from app import MedicationGraphRAG<br><br>st.set_page_config(page_title=&quot;MedGraphRAG&quot;, page_icon=&quot;💊&quot;, layout=&quot;wide&quot;)<br><br>SAMPLE_TEXT = &quot;&quot;&quot;The patient is a 54-year-old man with a history of multiple chronic conditions being managed through an extensive medication regimen. He was prescribed<br>Lisinopril, Metformin, Atorvastatin, Aspirin, Levothyroxine, and Sertraline over the course of the past year, with his treatment plan adjusted several<br>times based on follow-up visits.<br><br>He takes Lisinopril 10mg daily for hypertension, but often misses his Metformin 500mg dose which should be taken twice daily for diabetes. His<br>cardiologist also started him on Atorvastatin 40mg at bedtime for high cholesterol after his last lipid panel showed elevated LDL levels. To reduce<br>his risk of cardiovascular events, he was additionally prescribed Aspirin 81mg daily for heart disease prevention, which he takes alongside his<br>breakfast each morning.<br><br>Following a routine thyroid screening, he was found to have an underactive thyroid and was started on Levothyroxine 75mcg every morning for<br>hypothyroidism, to be taken on an empty stomach before any other medications. More recently, after reporting persistent low mood and difficulty sleeping<br>during a wellness visit, his primary care physician added Sertraline 50mg daily for depression, with plans to reassess the dosage after eight weeks.&quot;&quot;&quot;<br><br>SAMPLE_QUESTIONS = [<br>    &quot;List all medications related to cardiovascular conditions and their dosages.&quot;,<br>    &quot;What is the dosage and frequency for Lisinopril?&quot;,<br>    &quot;Which medications does the patient take once daily versus twice daily?&quot;,<br>    &quot;What medication is prescribed for hypothyroidism, and at what dose?&quot;,<br>    &quot;Which medications does the patient have trouble taking consistently?&quot;,<br>    &quot;What time of day should Levothyroxine be taken, and why?&quot;,<br>]<br><br>COLLECTION_NAME = &quot;medicationGraphRAGstore&quot;<br><br># ── sidebar ───────────────────────────────────────────────────────────────────<br><br>with st.sidebar:<br>    st.title(&quot;⚙️ Configuration&quot;)<br><br>    collection_name = st.text_input(&quot;Qdrant collection name&quot;, value=COLLECTION_NAME)<br><br>    st.subheader(&quot;Ollama models&quot;)<br>    model_extract = st.text_input(&quot;Extraction model&quot;, value=&quot;gemma3:latest&quot;)<br>    model_answer  = st.text_input(&quot;Answer model&quot;,     value=&quot;gemma3:latest&quot;)<br>    model_embed   = st.text_input(&quot;Embedding model&quot;,  value=&quot;embeddinggemma:latest&quot;)<br><br>    st.divider()<br>    st.caption(&quot;Neo4j + Qdrant config loaded from `.env`&quot;)<br><br><br># ── cached RAG instance ───────────────────────────────────────────────────────<br><br>@st.cache_resource(show_spinner=&quot;Connecting to Neo4j and Qdrant...&quot;)<br>def get_rag(extract_model, answer_model, embed_model):<br>    return MedicationGraphRAG(<br>        env_path=&quot;.env&quot;,<br>        ollama_model_extract=extract_model,<br>        ollama_model_answer=answer_model,<br>        ollama_embedding_model=embed_model,<br>    )<br><br><br>rag = get_rag(model_extract, model_answer, model_embed)<br><br># ── main ──────────────────────────────────────────────────────────────────────<br><br>st.title(&quot;💊 MedGraphRAG&quot;)<br>st.caption(&quot;Graph RAG pipeline for medication data — Neo4j + Qdrant + Ollama&quot;)<br><br>ingest_tab, query_tab = st.tabs([&quot;📥 Ingest&quot;, &quot;🔍 Query&quot;])<br><br># ── ingest tab ────────────────────────────────────────────────────────────────<br><br>with ingest_tab:<br>    st.subheader(&quot;Ingest clinical text&quot;)<br>    st.info(&quot;Paste clinical notes and click **Ingest**. Skip if data already ingested.&quot;, icon=&quot;ℹ️&quot;)<br><br>    clinical_text = st.text_area(<br>        &quot;Clinical text&quot;,<br>        value=SAMPLE_TEXT,<br>        height=300,<br>        label_visibility=&quot;collapsed&quot;,<br>    )<br><br>    if st.button(&quot;Ingest&quot;, type=&quot;primary&quot;, use_container_width=True):<br>        if not clinical_text.strip():<br>            st.error(&quot;Clinical text is empty.&quot;)<br>        else:<br>            with st.status(&quot;Ingesting...&quot;, expanded=True) as status:<br>                try:<br>                    st.write(&quot;Creating Qdrant collection...&quot;)<br>                    rag.create_collection(collection_name, rag.vector_dimension)<br><br>                    st.write(&quot;Extracting graph components via LLM...&quot;)<br>                    nodes, relationships = rag.extract_graph_components(clinical_text)<br>                    st.write(f&quot;Extracted {len(nodes)} nodes, {len(relationships)} relationships&quot;)<br><br>                    st.write(&quot;Ingesting nodes to Neo4j...&quot;)<br>                    node_id_mapping = rag.ingest_to_neo4j(nodes, relationships)<br><br>                    st.write(&quot;Ingesting embeddings to Qdrant...&quot;)<br>                    rag.ingest_to_qdrant(collection_name, clinical_text, node_id_mapping)<br><br>                    status.update(label=&quot;Ingestion complete!&quot;, state=&quot;complete&quot;)<br><br>                    with st.expander(&quot;Nodes extracted&quot;):<br>                        st.write(list(nodes.keys()))<br>                    with st.expander(&quot;Relationships extracted&quot;):<br>                        st.write(relationships)<br><br>                    st.session_state[&quot;ingested&quot;] = True<br><br>                except Exception as e:<br>                    status.update(label=&quot;Ingestion failed&quot;, state=&quot;error&quot;)<br>                    st.exception(e)<br><br># ── query tab ─────────────────────────────────────────────────────────────────<br><br>with query_tab:<br>    st.subheader(&quot;Ask a question&quot;)<br><br>    col1, col2 = st.columns([3, 1])<br><br>    with col2:<br>        st.caption(&quot;Sample questions&quot;)<br>        for q in SAMPLE_QUESTIONS:<br>            label = q[:55] + &quot;…&quot; if len(q) &gt; 55 else q<br>            if st.button(label, use_container_width=True, key=q):<br>                st.session_state[&quot;selected_query&quot;] = q<br><br>    with col1:<br>        default_query = st.session_state.get(&quot;selected_query&quot;, SAMPLE_QUESTIONS[0])<br>        query = st.text_input(&quot;Query&quot;, value=default_query)<br><br>        if st.button(&quot;Ask&quot;, type=&quot;primary&quot;, use_container_width=True):<br>            if not query.strip():<br>                st.error(&quot;Query is empty.&quot;)<br>            else:<br>                with st.spinner(&quot;Searching graph...&quot;):<br>                    try:<br>                        retriever_result = rag.retriever_search(collection_name, query)<br>                        entity_ids = [<br>                            item.content.split(&quot;&#39;id&#39;: &#39;&quot;)[1].split(&quot;&#39;&quot;)[0]<br>                            for item in retriever_result.items<br>                            if &quot;&#39;id&#39;: &#39;&quot; in item.content<br>                        ]<br>                        subgraph = rag.fetch_related_graph(entity_ids)<br>                        graph_context = rag.format_graph_context(subgraph)<br>                    except Exception as e:<br>                        st.error(&quot;Retrieval failed.&quot;)<br>                        st.exception(e)<br>                        st.stop()<br><br>                with st.spinner(&quot;Generating answer...&quot;):<br>                    try:<br>                        answer = rag.graphRAG_run(graph_context, query)<br>                    except Exception as e:<br>                        st.error(&quot;LLM call failed.&quot;)<br>                        st.exception(e)<br>                        st.stop()<br><br>                st.subheader(&quot;Answer&quot;)<br>                st.markdown(answer)<br><br>                with st.expander(&quot;Graph context used&quot;):<br>                    c1, c2 = st.columns(2)<br>                    with c1:<br>                        st.markdown(&quot;**Nodes**&quot;)<br>                        for node in graph_context[&quot;nodes&quot;]:<br>                            st.markdown(f&quot;- {node}&quot;)<br>                    with c2:<br>                        st.markdown(&quot;**Edges**&quot;)<br>                        for edge in graph_context[&quot;edges&quot;]:<br>                            st.markdown(f&quot;- {edge}&quot;)</pre><h4>Run the app</h4><p>Execute the following command to start the streamlit application. For first time, go to the Ingest tab, click Ingest to populate the graph (only needed once per dataset), then switch to the Query tab and ask a question or pick one of the sample buttons.</p><pre>streamlit run streamlit_app.py</pre><p>This opens the UI in your browser (default: <a href="http://localhost:8501).">http://localhost:8501).</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wEtQhx7tLsk_XiQGdK6r6A.gif" /></figure><p><em>Thanks for reading this article !!</em></p><p>If you enjoyed this article, please click on the clap button 👏 and share to help others find it!</p><p>The full source code for this tutorial can be found here,</p><p><a href="https://github.com/codemaker2015/med-graph-rag">GitHub - codemaker2015/med-graph-rag: Graph RAG pipeline for medication data. Extracts medication entities and relationships from clinical text, stores them in Neo4j (graph + vector index), then answers queries using retrieved subgraph context.</a></p><p>Resources:</p><ul><li><a href="https://github.com/neo4j/neo4j-graphrag-python">GitHub - neo4j/neo4j-graphrag-python: Neo4j GraphRAG for Python</a></li><li><a href="https://github.com/google/langextract">GitHub - google/langextract: A Python library for extracting structured information from unstructured text using LLMs with precise source grounding and interactive visualization.</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d8f4ecbb2b75" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[PySIFT: GPU Accelerated SIFT for Modern Era]]></title>
            <link>https://codemaker2016.medium.com/pysift-gpu-accelerated-sift-for-modern-era-9de6791a181c?source=rss-b7a60e472dfc------2</link>
            <guid isPermaLink="false">https://medium.com/p/9de6791a181c</guid>
            <category><![CDATA[opencv]]></category>
            <category><![CDATA[python-for-beginners]]></category>
            <category><![CDATA[feature-extraction]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Vishnu Sivan]]></dc:creator>
            <pubDate>Sun, 31 May 2026 10:53:20 GMT</pubDate>
            <atom:updated>2026-05-31T10:53:20.117Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1rD6pV7SS0ImFaSwu8nsag.png" /></figure><p>In computer vision, the Scale-Invariant Feature Transform (SIFT) algorithm remains a classic foundational standard for keypoint detection and local feature description. Despite the massive shift toward deep learning-based matchers, SIFT remains a highly robust, reliable fallback and a standard component across traditional 3D reconstruction pipelines (like Structure-from-Motion) and hybrid image retrieval architectures.</p><p>However, standard implementations like OpenCV’s SIFT present an expensive bottleneck in modern deep learning workflows: Host-to-Device (CPU-to-GPU) memory transfers<strong>.</strong> When your pipeline feeds extracted local keypoints directly into neural matchers (such as SuperGlue or LightGlue), copying raw images or intermediate tensors back and forth across the PCIe bus severely caps your execution speed.</p><p>To address this structural hurdle, a team of researchers from IIT Madras developed PySIFT (recently published in arXiv:2605.17869). PySIFT is a Python-native, fully GPU-resident, and deterministic SIFT implementation built specifically to match OpenCV’s precision while outperforming standard pipelines by keeping data entirely on the GPU via zero-copy interop.</p><h3>Getting Started</h3><h4>Table of Contents</h4><ul><li><a href="#f5cd">What is PySIFT</a></li><li><a href="#5044">Architecture of pySIFT</a></li><li><a href="#e2d7">Why PySIFT</a></li><li>OpenCV SIFT vs PySIFT</li><li>Benchmarks</li><li>Usecases</li><li><a href="#fca4">Getting Started with PySIFT</a></li><li>Installation</li><li>1. Feature Extraction</li><li>2. Feature Matching</li><li>3. PyTorch / DLPack Integration</li><li>4. Determinism</li><li>Running PySIFT in a Docker Container</li><li>References</li></ul><h4>What is PySIFT</h4><p>PySIFT is the first fully GPU-resident SIFT implementation. Unlike OpenCV SIFT, which round-trips data between CPU and GPU memory, PySIFT keeps every stage — detection, description, matching, RANSAC, and blending — on the GPU via CuPy and Numba CUDA kernels. Tensors hand off to PyTorch via DLPack without any memory copies.</p><p>A key finding from the accompanying paper is that classical SIFT with DSP multi-scale pooling outperforms learned neural descriptors (HardNet, OriNet) on every accuracy metric across HPatches, ROxford5K, IMC Phototourism, and MegaDepth — while running 2–18× faster. The conclusion: compose SIFT with learned matchers like LightGlue rather than replacing it entirely.</p><h4>Architecture of pySIFT</h4><p>PySIFT introduces three structural improvements to keypoint extraction architectures:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HtwIpMDcEgF0TzOp.png" /></figure><ul><li><strong>GPU-Resident Execution &amp; Zero-Copy Interop</strong>: PySIFT relies on standard deep learning data sharing via DLPack data structures. This means you can ingest a PyTorch CUDA tensor directly into PySIFT, execute scale-space construction, keypoint localization, and descriptor computation entirely in GPU memory, and yield PyTorch CUDA output tensors. Zero PCIe round-trips.</li><li><strong>Deterministic Output</strong>: Traditional parallelized GPU SIFT variants often suffer from slight non-determinism during keypoint reduction or sorting phases due to atomic thread operations. PySIFT guarantees deterministic execution, ensuring reproducibility — a crucial requirement for production ML pipelines.</li><li><strong>Massive End-to-End Speedups</strong>: Benchmarked against OpenCV, PySIFT achieves up to a 26% faster end-to-end execution speedup for typical vision pipelines, coupled with an incredible 4x speedup during feature matching stages.</li></ul><h4>Why PySIFT</h4><p>When you use the standard cv2.SIFT_create() in OpenCV, the algorithm executes on your CPU. If you are building a modern AI vision pipeline (e.g., feeding SIFT features into a deep learning neural matcher like LightGlue, or utilizing a neural network for object recognition), your pipeline suffers from a massive architectural bottleneck known as PCIe bus latency.</p><p>Here is what the data path looks like with standard OpenCV SIFT:</p><pre>[GPU VRAM] ──(Deep Learning Model / Tensor Ingestion)──┐<br>                                                         │  [EXPENSIVE COPY]<br>                                                         ▼  via PCIe Bus<br>[Host RAM] ──(OpenCV SIFT Feature Extraction on CPU)───┘<br>                                                         │  [EXPENSIVE COPY]<br>                                                         ▼  via PCIe Bus<br>[GPU VRAM] ──(Neural Matcher / Next Deep Learning Stage)</pre><p>Every frame requires you to ship the raw image data down to the CPU host memory via the PCIe bus, wait for the CPU to compute the keypoints and descriptors, and then synchronize and copy those tensors <em>back</em> up to the GPU VRAM so your neural network can use them. For real-time applications (like video tracking, SLAM, or robotics), these Host-to-Device (H2D) and Device-to-Host (D2H) memory copies destroy your throughput.</p><p>PySIFT is a fully GPU-resident, deterministic implementation of SIFT. Because it accepts GPU data structures (via CuPy/DLPack interop), the data path becomes entirely unified inside your graphics card’s VRAM:</p><pre>[GPU VRAM] ──&gt; [PyTorch CUDA Tensor] ──&gt; [PySIFT Extraction (CuPy)] ──&gt; [Neural Matcher Tensors]</pre><p>Classical SIFT implementations were never designed for GPU-native AI pipelines.</p><p>Traditional workflows typically look like this:</p><pre>Image → CPU SIFT → CPU Descriptor → Transfer to GPU → Deep Learning Model</pre><p>PySIFT changes this to:</p><pre>Image → GPU SIFT → GPU Descriptor → GPU Matcher → GPU Deep Learning Model</pre><p>This may sound like a small architectural improvement, but in practice it dramatically reduces latency and improves throughput.</p><h4>OpenCV SIFT vs PySIFT</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/712/1*3gBF_L_vc8PLInDG1LNKQw.png" /></figure><h4>Benchmarks</h4><p>Measured on a laptop-grade NVIDIA RTX 3050 (4 GB VRAM) versus OpenCV SIFT on CPU.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/817/1*t--2oVFLmf1LPb016esfMQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/825/1*gMuya7sC2xfc-iubVgChcQ.png" /></figure><h4>Real-World Use Cases</h4><ul><li><strong>Visual SLAM</strong>: PySIFT can significantly improve Visual SLAM pipelines by accelerating feature extraction and matching operations. Its GPU-native architecture helps speed up keyframe matching, loop closure detection, and pose estimation, making it well-suited for robotics and autonomous navigation applications.</li><li><strong>Image Retrieval</strong>: PySIFT enables fast and accurate descriptor matching, making it useful for visual search engines, product similarity platforms, and reverse image search applications that need to process large image collections efficiently.</li><li><strong>Autonomous Driving: </strong>PySIFT provides reliable feature descriptors that can complement deep learning models, particularly in challenging real-world environments where robust geometric information is required.</li><li><strong>Edge AI and Robotics</strong>: The efficiency and determinism of PySIFT make it a strong candidate for edge computing and robotics applications. Its ability to perform feature extraction entirely on the GPU can help reduce latency and improve performance in resource-constrained environments where real-time processing is critical.</li></ul><h3>Getting Started with PySIFT</h3><h4>Installation</h4><p>PySIFT needs an NVIDIA GPU with CUDA. Two dependencies are version-specific and must be installed manually before the package itself.</p><p>PySIFT needs an NVIDIA GPU with CUDA. Two dependencies are version-specific and must be installed manually before the package itself.</p><ul><li><strong>Step 1: Check your CUDA version</strong></li></ul><pre># Find out which CUDA toolkit is installed<br>nvcc --version</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/812/1*ZEssAz4DaTbsju1YOMaiYA.png" /></figure><p>If CUDA is not already installed on your system, download the appropriate installer from the official NVIDIA CUDA Toolkit website. Follow the installation instructions provided in the documentation to complete the setup.</p><p><a href="https://developer.nvidia.com/cuda-12-4-0-download-archive">CUDA Toolkit 12.4 Downloads</a></p><ul><li><strong>Step 2: Install CuPy (CUDA-version-specific)</strong></li></ul><pre># Pick ONE line that matches your CUDA version<br>pip install cupy-cuda12x   # CUDA 12.x<br>pip install cupy-cuda11x   # CUDA 11.x</pre><ul><li><strong>Step 3: Install PyTorch with CUDA support</strong></li></ul><blockquote>A plain pip install torch gives you a CPU-only build. Use the index URLs below to get a CUDA-enabled wheel.</blockquote><pre>pip install torch --index-url https://download.pytorch.org/whl/cu124</pre><ul><li><strong>Step 4: Install PySIFT</strong></li></ul><pre># From PyPI (package name is staysift)<br>pip install staysift<br># Or directly from GitHub<br>pip install git+https://github.com/SivaIITM/PySIFT.git<br># Or editable from source<br>git clone https://github.com/SivaIITM/PySIFT.git<br>cd PySIFT<br>pip install -e .</pre><ul><li><strong>Full single-command install</strong></li></ul><pre>pip install cupy-cuda12x<br>pip install git+https://github.com/SivaIITM/PySIFT.git</pre><h4>1. Feature Extraction</h4><p>Feature extraction is a fundamental step in many computer vision applications, including image matching, object recognition, image stitching, and visual localization.</p><p>PySIFT is designed as a drop-in replacement for OpenCV’s SIFT implementation. Similar to cv2.SIFT_create(), it accepts a grayscale NumPy image as input and returns a set of detected keypoints along with their corresponding feature descriptors.</p><p><strong>OpenCV SIFT</strong></p><pre>import cv2<br>import matplotlib.pyplot as plt<br><br># Load image and convert to grayscale numpy array<br>image_background = cv2.imread(&quot;input.jpg&quot;)<br>gray = cv2.cvtColor(image_background, cv2.COLOR_BGR2GRAY)<br><br># Instantiate OpenCV SIFT detector<br>sift = cv2.SIFT_create()<br><br># detectAndCompute returns (keypoints, descriptors)<br>keypoints, descriptors = sift.detectAndCompute(gray, None)<br><br>print(f&quot;Found {len(keypoints)} keypoints&quot;)<br>print(f&quot;Descriptor shape: {descriptors.shape}&quot;)  # (N, 128)<br><br># Draw keypoints using OpenCV<br>vis = cv2.drawKeypoints(<br>    image_background, keypoints, None,<br>    flags=cv2.DRAW_MATCHES_FLAGS_DRAW_RICH_KEYPOINTS<br>)<br><br>plt.figure(figsize=(12, 8))<br>plt.imshow(cv2.cvtColor(vis, cv2.COLOR_BGR2RGB))<br>plt.axis(&quot;off&quot;)<br>plt.title(f&quot;OpenCV SIFT — {len(keypoints)} keypoints detected&quot;)<br>plt.show()</pre><p>Run the code to visualize the output produced by the OpenCV SIFT feature extraction process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zbU1RMzRQWpZhxZmUedezA.png" /></figure><p><strong>PySIFT implementation</strong></p><pre>import cv2<br>import matplotlib.pyplot as plt<br>from pysift import PySIFT<br><br># Load image and convert to grayscale numpy array<br>image_background = cv2.imread(&quot;input.jpg&quot;)<br>gray = cv2.cvtColor(image_background, cv2.COLOR_BGR2GRAY)<br><br># Instantiate PySIFT detector<br>sift = PySIFT()<br><br>keypoints, descriptors = sift.detectAndCompute(gray)<br><br>print(f&quot;Found {len(keypoints)} keypoints&quot;)<br>print(f&quot;Descriptor shape: {descriptors.shape}&quot;)  # (N, 128)<br><br># Draw keypoints using OpenCV (kps are standard cv2.KeyPoint objects)<br>vis = cv2.drawKeypoints(<br>    image_background, keypoints, None,<br>    flags=cv2.DRAW_MATCHES_FLAGS_DRAW_RICH_KEYPOINTS<br>)<br><br>plt.figure(figsize=(12, 8))<br>plt.imshow(cv2.cvtColor(vis, cv2.COLOR_BGR2RGB))<br>plt.axis(&quot;off&quot;)<br>plt.title(f&quot;PySIFT — {len(keypoints)} keypoints detected&quot;)<br>plt.show()</pre><p>Run the code to visualize the output produced by the PySIFT feature extraction process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pVGyFrBnY3G7va4L3zp_Uw.png" /></figure><h4>2. Feature Matching</h4><p>Feature matching is the process of identifying corresponding keypoints between two images based on the similarity of their feature descriptors. It plays a crucial role in applications such as image registration, panorama stitching, visual localization, object recognition, and 3D reconstruction.</p><p><strong>OpenCV SIFT Implementation</strong></p><pre>import cv2<br>import numpy as np<br>import matplotlib.pyplot as plt<br><br># Load images — colour copy for display, grayscale for detection<br>img1_bgr  = cv2.imread(&quot;left.jpg&quot;)<br>img2_bgr  = cv2.imread(&quot;right.jpg&quot;)<br>img1_gray = cv2.cvtColor(img1_bgr, cv2.COLOR_BGR2GRAY).astype(np.uint8)<br>img2_gray = cv2.cvtColor(img2_bgr, cv2.COLOR_BGR2GRAY).astype(np.uint8)<br><br># ── Detector ──<br>sift = cv2.SIFT_create(nfeatures=8000)<br><br># ── Extract features ──<br>kps1, desc1 = sift.detectAndCompute(img1_gray, None)<br>kps2, desc2 = sift.detectAndCompute(img2_gray, None)<br><br>print(f&quot;Keypoints — left: {len(kps1)}  right: {len(kps2)}&quot;)<br><br># ── Match with BFMatcher + Lowe ratio test (mirrors PySIFT&#39;s ratio matcher) ──<br>bf      = cv2.BFMatcher(cv2.NORM_L2)<br>knn_matches = bf.knnMatch(desc1, desc2, k=2)<br><br># Lowe ratio test — keep only matches where best is clearly better than second<br>matches = []<br>for m, n in knn_matches:<br>    if m.distance &lt; 0.75 * n.distance:<br>        matches.append(m)<br><br># Sort by distance — best matches first<br>matches_sorted = sorted(matches, key=lambda m: m.distance)<br>print(f&quot;Total matches after ratio test: {len(matches_sorted)}&quot;)<br><br>vis_all = cv2.drawMatches(<br>    img1_bgr, kps1,<br>    img2_bgr, kps2,<br>    matches_sorted,<br>    None,<br>    matchColor       = (0, 200, 255),<br>    singlePointColor = (100, 100, 100),<br>    flags=cv2.DrawMatchesFlags_NOT_DRAW_SINGLE_POINTS<br>)<br><br>plt.figure(figsize=(18, 7))<br>plt.imshow(cv2.cvtColor(vis_all, cv2.COLOR_BGR2RGB))<br>plt.axis(&quot;off&quot;)<br>plt.title(f&quot;OpenCV SIFT — All {len(matches_sorted)} matches&quot;)<br>plt.tight_layout()<br>plt.savefig(&quot;opencv_matches_all.png&quot;, dpi=150)<br>plt.show()</pre><p>Run the code to visualize the output produced by the OpenCV SIFT feature matching process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0tHxBiEX0SusXb7PZ_oO0A.png" /></figure><p><strong>PySIFT Implementation</strong></p><p>PySIFT’s matcher runs entirely on-GPU with batched matrix operations. Two modes are available: classical Lowe ratio test or learned LightGlue.</p><pre>import cv2<br>import numpy as np<br>import matplotlib.pyplot as plt<br>from pysift import GPUPyStitch<br><br># Load images — keep a colour copy for display, grayscale for detection<br>img1_bgr = cv2.imread(&quot;left.jpg&quot;)<br>img2_bgr = cv2.imread(&quot;right.jpg&quot;)<br>img1_gray = cv2.cvtColor(img1_bgr, cv2.COLOR_BGR2GRAY).astype(np.uint8)<br>img2_gray = cv2.cvtColor(img2_bgr, cv2.COLOR_BGR2GRAY).astype(np.uint8)<br><br><br># Build stitcher (owns both detector and matcher)<br>stitcher = GPUPyStitch(match_ratio=0.75, match_batch=512,<br>                       max_keypoints=8000, matcher=&quot;ratio&quot;)<br><br># Extract features<br>kps1, desc1 = stitcher._detector.detectAndCompute(img1_gray)<br>kps2, desc2 = stitcher._detector.detectAndCompute(img2_gray)<br><br># Match<br>matches = stitcher._match_features(desc1, desc2)<br>print(f&quot;Total matches: {len(matches)}&quot;)<br><br># ── Sort by distance (best matches first) ──<br>matches_sorted = sorted(matches, key=lambda m: m.distance)<br><br>vis_all = cv2.drawMatches(<br>    img1_bgr, kps1,<br>    img2_bgr, kps2,<br>    matches_sorted,<br>    None,<br>    matchColor      = (0, 200, 255),<br>    singlePointColor= (100, 100, 100),<br>    flags=cv2.DrawMatchesFlags_NOT_DRAW_SINGLE_POINTS<br>)<br><br>plt.figure(figsize=(18, 7))<br>plt.imshow(cv2.cvtColor(vis_all, cv2.COLOR_BGR2RGB))<br>plt.axis(&quot;off&quot;)<br>plt.title(f&quot;All {len(matches)} matches&quot;)<br>plt.tight_layout()<br>plt.savefig(&quot;matches_all.png&quot;, dpi=150)<br>plt.show()</pre><p>Run the code to visualize the output produced by the PySIFT feature matching process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xxCaVCt4hOM3FTZ41yd8gQ.png" /></figure><h4>3. PyTorch / DLPack Integration</h4><p>Because PySIFT keeps everything in VRAM, descriptors can be handed to a PyTorch model in sub-millisecond time with zero copies.</p><pre>import os<br>os.environ[&quot;CUDA_PATH&quot;] = r&quot;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4&quot;<br>os.environ[&quot;CUDA_HOME&quot;] = r&quot;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4&quot;<br><br>import cv2<br>import torch<br>import cupy as cp<br>import numpy as np<br>from pysift import PySIFT<br><br># ── 1. Load image ──<br>gray = cv2.imread(&quot;input.jpg&quot;, cv2.IMREAD_GRAYSCALE)<br>assert gray is not None, &quot;Could not load input.jpg&quot;<br>gray = gray.astype(np.uint8)<br><br># ── 2. Extract features ──<br>sift  = PySIFT()<br>kps, descs = sift.detectAndCompute(gray, gpu_output=True)<br><br>on_gpu = isinstance(descs, cp.ndarray)<br><br>print(&quot;=&quot; * 50)<br>print(f&quot;  Keypoints detected : {len(kps)}&quot;)<br>print(f&quot;  Descriptor type    : {type(descs).__name__}&quot;)<br>print(f&quot;  Descriptor shape   : {descs.shape}&quot;)<br>print(f&quot;  Descriptor dtype   : {descs.dtype}&quot;)<br>print(f&quot;  On GPU (CuPy)      : {on_gpu}&quot;)<br>print(&quot;=&quot; * 50)<br><br># ── Always work with a CuPy array from here on ──<br># If PySIFT returned NumPy (CUDA mismatch fallback), upload once explicitly<br>if on_gpu:<br>    descs_gpu = descs                  # already on GPU — no copy<br>    print(&quot;\n  ✓ Descriptors already on GPU — all transfers below are zero-copy&quot;)<br>else:<br>    descs_gpu = cp.asarray(descs)      # one explicit CPU→GPU upload<br>    print(&quot;\n  ⚠  Descriptors were on CPU — uploaded once to GPU&quot;)<br><br># ── 3. Method A — torch.as_tensor via __cuda_array_interface__ ──<br>desc_tensor_a = torch.as_tensor(descs_gpu, device=&quot;cuda&quot;)<br><br>print(&quot;\n[Method A] torch.as_tensor  (via __cuda_array_interface__)&quot;)<br>print(f&quot;  Shape  : {desc_tensor_a.shape}&quot;)<br>print(f&quot;  Device : {desc_tensor_a.device}&quot;)<br>print(f&quot;  Dtype  : {desc_tensor_a.dtype}&quot;)<br>print(f&quot;  Zero-copy (same GPU buffer)? &quot;<br>      f&quot;{desc_tensor_a.data_ptr() == descs_gpu.data.ptr}&quot;)<br><br># ── 4. Method B — modern DLPack via __dlpack__() protocol ──<br>desc_tensor_b = torch.from_dlpack(descs_gpu)   # CuPy 14.x modern API<br><br>print(&quot;\n[Method B] torch.from_dlpack  (modern __dlpack__() protocol)&quot;)<br>print(f&quot;  Shape  : {desc_tensor_b.shape}&quot;)<br>print(f&quot;  Device : {desc_tensor_b.device}&quot;)<br>print(f&quot;  Dtype  : {desc_tensor_b.dtype}&quot;)<br><br># ── 5. Round-trip: PyTorch tensor → back to CuPy ──<br>#    cp.from_dlpack() is the modern replacement for cp.fromDlpack()<br>cp_back = cp.from_dlpack(desc_tensor_b)<br><br>print(&quot;\n[Round-trip] PyTorch → CuPy  (cp.from_dlpack)&quot;)<br>print(f&quot;  Shape  : {cp_back.shape}&quot;)<br>print(f&quot;  Type   : {type(cp_back).__name__}&quot;)<br>print(f&quot;  Matches original? {cp.array_equal(descs_gpu, cp_back)}&quot;)<br><br># ── 6. Downstream PyTorch model ──<br>class DummyModel(torch.nn.Module):<br>    def __init__(self, in_dim=128, out_dim=64):<br>        super().__init__()<br>        self.net = torch.nn.Sequential(<br>            torch.nn.Linear(in_dim, 256),<br>            torch.nn.ReLU(),<br>            torch.nn.Linear(256, out_dim),<br>        )<br>    def forward(self, x):<br>        return self.net(x)<br><br>model = DummyModel().cuda()<br><br>with torch.no_grad():<br>    embeddings = model(desc_tensor_a)<br><br>print(&quot;\n[Downstream model]&quot;)<br>print(f&quot;  Input  shape : {desc_tensor_a.shape}&quot;)<br>print(f&quot;  Output shape : {embeddings.shape}&quot;)<br>print(f&quot;  Output device: {embeddings.device}&quot;)<br><br># ── 7. Summary ──<br>print(&quot;\n&quot; + &quot;=&quot; * 50)<br>print(&quot;  Pipeline summary&quot;)<br>print(&quot;=&quot; * 50)<br>print(f&quot;  input.jpg         →  CPU  numpy  uint8   {gray.shape}&quot;)<br>print(f&quot;  detectAndCompute  →  {&#39;GPU  CuPy &#39; if on_gpu else &#39;CPU  NumPy&#39;} float32 {descs.shape}&quot;)<br>if not on_gpu:<br>    print(f&quot;  cp.asarray()      →  GPU  CuPy  float32 {descs_gpu.shape}  (1 upload)&quot;)<br>print(f&quot;  from_dlpack       →  GPU  torch float32 {desc_tensor_a.shape}  (0 bytes copied)&quot;)<br>print(f&quot;  DummyModel        →  GPU  torch float32 {embeddings.shape}&quot;)<br>print(&quot;=&quot; * 50)</pre><p>Run the code to see the output generated.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/952/1*DuVqBeVQgHRwRRzmnSv5Lg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/952/1*Ivn6Q24OFL1l7wbdCpZ3sQ.png" /></figure><h4>4. Determinism</h4><p>PySIFT uses warp-shuffle reductions instead of atomicAdd to eliminate non-deterministic floating-point accumulation. This gives bitwise-identical output across runs and — uniquely — across different GPU architectures.</p><pre>import cv2<br>import numpy as np<br>import cupy as cp<br>from pysift import PySIFT<br><br># ── 1. Load image ──<br>gray = cv2.imread(&quot;input.jpg&quot;, cv2.IMREAD_GRAYSCALE)<br>assert gray is not None, &quot;Could not load input.jpg&quot;<br>gray = gray.astype(np.uint8)<br><br>print(&quot;=&quot; * 50)<br>print(f&quot;  Image shape : {gray.shape}&quot;)<br>print(f&quot;  Image dtype : {gray.dtype}&quot;)<br>print(&quot;=&quot; * 50)<br><br># ── 2. Run detectAndCompute three times on the same image ──<br>sift = PySIFT()<br><br>print(&quot;\nRunning detectAndCompute 3 times on the same image...&quot;)<br><br>kps1, desc1 = sift.detectAndCompute(gray)<br>kps2, desc2 = sift.detectAndCompute(gray)<br>kps3, desc3 = sift.detectAndCompute(gray)<br><br>print(f&quot;  Run 1 — keypoints: {len(kps1)},  descriptor shape: {desc1.shape}&quot;)<br>print(f&quot;  Run 2 — keypoints: {len(kps2)},  descriptor shape: {desc2.shape}&quot;)<br>print(f&quot;  Run 3 — keypoints: {len(kps3)},  descriptor shape: {desc3.shape}&quot;)<br><br># ── 3. Convert to NumPy for comparison ──<br># .get() moves CuPy array to CPU — use np.array() if desc is already NumPy<br>def to_numpy(d):<br>    return d.get() if isinstance(d, cp.ndarray) else np.array(d)<br><br>arr1 = to_numpy(desc1)<br>arr2 = to_numpy(desc2)<br>arr3 = to_numpy(desc3)<br><br># ── 4. Bitwise determinism checks ──<br>print(&quot;\n── Determinism checks ──&quot;)<br><br>match_12 = np.array_equal(arr1, arr2)<br>match_13 = np.array_equal(arr1, arr3)<br>match_23 = np.array_equal(arr2, arr3)<br><br>print(f&quot;  Run 1 == Run 2 : {&#39;✓ identical&#39; if match_12 else &#39;✗ different&#39;}&quot;)<br>print(f&quot;  Run 1 == Run 3 : {&#39;✓ identical&#39; if match_13 else &#39;✗ different&#39;}&quot;)<br>print(f&quot;  Run 2 == Run 3 : {&#39;✓ identical&#39; if match_23 else &#39;✗ different&#39;}&quot;)<br><br># ── 5. Numeric diff (should be all zeros) ──<br>diff_12 = np.abs(arr1.astype(np.float64) - arr2.astype(np.float64))<br>diff_13 = np.abs(arr1.astype(np.float64) - arr3.astype(np.float64))<br><br>print(f&quot;\n── Numeric diff (should be 0.0 everywhere) ──&quot;)<br>print(f&quot;  Run1 vs Run2 — max: {diff_12.max():.6f}  mean: {diff_12.mean():.6f}&quot;)<br>print(f&quot;  Run1 vs Run3 — max: {diff_13.max():.6f}  mean: {diff_13.mean():.6f}&quot;)<br><br># ── 6. Keypoint position determinism ──<br>print(f&quot;\n── Keypoint position checks ──&quot;)<br>kp1_pts = np.array([k.pt for k in kps1])<br>kp2_pts = np.array([k.pt for k in kps2])<br>kp3_pts = np.array([k.pt for k in kps3])<br><br>pos_12 = np.allclose(kp1_pts, kp2_pts, atol=0)<br>pos_13 = np.allclose(kp1_pts, kp3_pts, atol=0)<br>print(f&quot;  Keypoint positions Run1 == Run2 : {&#39;✓ identical&#39; if pos_12 else &#39;✗ different&#39;}&quot;)<br>print(f&quot;  Keypoint positions Run1 == Run3 : {&#39;✓ identical&#39; if pos_13 else &#39;✗ different&#39;}&quot;)<br><br># ── 7. Final assert ──<br>print()<br>assert match_12 and match_13 and match_23, &quot;✗ Determinism check FAILED&quot;<br>print(&quot;✓ Bitwise determinism confirmed across all 3 runs&quot;)<br><br># ── 8. Summary ──<br>print(&quot;\n&quot; + &quot;=&quot; * 50)<br>print(&quot;  Summary&quot;)<br>print(&quot;=&quot; * 50)<br>print(f&quot;  Image              : input.jpg  {gray.shape}&quot;)<br>print(f&quot;  Keypoints per run  : {len(kps1)}&quot;)<br>print(f&quot;  Descriptor shape   : {arr1.shape}&quot;)<br>print(f&quot;  Max descriptor diff: {diff_12.max():.6f}  (0.0 = bitwise identical)&quot;)<br>print(f&quot;  Deterministic      : {&#39;Yes ✓&#39; if match_12 and match_13 else &#39;No ✗&#39;}&quot;)<br>print(&quot;=&quot; * 50)</pre><p>Run the code to see the output generated.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/833/1*54aHaHBngpuc3DtAr5b9Kw.png" /></figure><h4>Running PySIFT in a Docker Container</h4><p>Containerization is an effective way to ensure a consistent and reproducible development environment. This makes it particularly useful for machine learning and computer vision projects that depend on specific versions of CUDA, Python libraries, and GPU drivers.</p><p>In this section, we will create a Docker-based environment for PySIFT and verify its functionality inside a container.</p><h4>Creating the Dockerfile</h4><p>Create a Dockerfile containing the required dependencies and environment configuration needed to run PySIFT within a container.</p><pre>FROM nvidia/cuda:12.1.1-devel-ubuntu22.04<br><br># Prevent interactive prompts<br>ENV DEBIAN_FRONTEND=noninteractive<br><br># Install core system packages<br>RUN apt-get update &amp;&amp; apt-get install -y \<br>    python3-pip python3-dev git wget libgl1-mesa-glx libglib2.0-0 \<br>    &amp;&amp; rm -rf /var/lib/apt/lists/*<br><br># Install PyTorch and Python package management tools<br>RUN pip3 install --no-cache-dir uv<br>RUN uv pip install --system torch --index-url https://download.pytorch.org/whl/cu121<br><br># Install CuPy matching your CUDA version alongside visualization frameworks<br>RUN uv pip install --system cupy-cuda12x opencv-python matplotlib<br><br># Clone and install PySIFT<br>RUN git clone https://github.com/SivaIITM/PySIFT.git /opt/PySIFT<br>WORKDIR /opt/PySIFT<br>RUN uv pip install --system -e .<br><br>CMD [&quot;python3&quot;]</pre><h4>Creating a Basic PySIFT Test Script</h4><p>Next, create a simple Python script that imports PySIFT, loads a sample image, performs feature extraction, and prints the resulting keypoints and descriptors.</p><pre>import cv2<br>import torch<br>import numpy as np<br>import cupy as cp<br>import matplotlib.pyplot as plt<br>from pysift import PySIFT<br><br>def run_gpu_sift(image_path):<br>    # 1. Load image<br>    img_bgr = cv2.imread(image_path)<br>    if img_bgr is None:<br>        raise FileNotFoundError(f&quot;Could not load image at {image_path}&quot;)<br>        <br>    img_gray = cv2.cvtColor(img_bgr, cv2.COLOR_BGR2GRAY)<br>    <br>    # 2. Ship to GPU as a PyTorch Tensor<br>    torch_tensor = torch.from_numpy(img_gray).to(torch.uint8).cuda()<br>    <br>    # 3. Pass the Torch GPU memory pool straight into CuPy via DLPack zero-copy<br>    cupy_array = cp.asarray(torch_tensor)<br>    <br>    print(f&quot;Input Tensor Shape: {torch_tensor.shape} | Device: {torch_tensor.device}&quot;)<br>    print(f&quot;Converted to CuPy Array on GPU: {type(cupy_array)}&quot;)<br>    <br>    # 4. Initialize the PySIFT Engine<br>    sift_engine = PySIFT() <br>    <br>    # 5. Extract Features natively on the GPU using the CuPy array block<br>    keypoints, descriptors = sift_engine.detectAndCompute(cupy_array)<br>    <br>    # 6. Safely cast the returned list object into a NumPy array for visualization processing<br>    kp_cpu = np.array(keypoints)<br>    <br>    print(f&quot;Extracted {len(kp_cpu)} Keypoints safely on GPU memory!&quot;)<br>    if hasattr(descriptors, &#39;shape&#39;):<br>        print(f&quot;Descriptors Tensor Shape: {descriptors.shape}&quot;)<br>    <br>    # 7. Visualizing the Result<br>    plt.figure(figsize=(10, 8))<br>    plt.imshow(cv2.cvtColor(img_bgr, cv2.COLOR_BGR2RGB))<br>    <br>    if len(kp_cpu) &gt; 0:<br>        # Check if output is formatted as a 2D coordinates matrix [N, 4+]<br>        if len(kp_cpu.shape) &gt; 1 and kp_cpu.shape[1] &gt;= 2:<br>            plt.scatter(kp_cpu[:, 0], kp_cpu[:, 1], s=kp_cpu[:, 2]*2 if kp_cpu.shape[1] &gt; 2 else 20, <br>                        c=&#39;red&#39;, alpha=0.5, edgecolors=&#39;black&#39;)<br>        else:<br>            print(&quot;Keypoints returned as an alternate array format structure. Check shape configuration.&quot;)<br>        <br>    plt.title(f&quot;PySIFT GPU-Resident Detections: {len(kp_cpu)} points&quot;)<br>    plt.axis(&#39;off&#39;)<br>    <br>    plt.savefig(&#39;pysift_output.png&#39;, bbox_inches=&#39;tight&#39;)<br>    print(&quot;Saved target visualization to &#39;pysift_output.png&#39;&quot;)<br><br>if __name__ == &quot;__main__&quot;:<br>    # Generate procedural dummy scene with rich geometric contrast patterns<br>    dummy_img = np.zeros((512, 512, 3), dtype=np.uint8)<br>    cv2.rectangle(dummy_img, (100, 100), (400, 400), (255, 255, 255), -1)<br>    cv2.circle(dummy_img, (256, 256), 80, (128, 128, 128), -1)<br>    <br>    # Inject high-frequency texture to maximize SIFT scale space detections<br>    for _ in range(30):<br>        x, y = np.random.randint(120, 380, 2)<br>        cv2.circle(dummy_img, (x, y), np.random.randint(5, 15), (0, 0, 0), -1)<br><br>    cv2.imwrite(&quot;sample_scene.jpg&quot;, dummy_img)<br>    run_gpu_sift(&quot;sample_scene.jpg&quot;)</pre><h4>Creating the Docker Compose Configuration</h4><p>Create a docker-compose.yaml file to automate the image build process and execute the test script.</p><pre>services:<br>  pysift-tests:<br>    build:<br>      context: ./app<br>      dockerfile: Dockerfile<br>    image: pysift-test:latest<br>    gpus: all<br>    environment:<br>      - NVIDIA_VISIBLE_DEVICES=all<br>      - NVIDIA_DRIVER_CAPABILITIES=compute,utility,graphics<br>    volumes:<br>      - ./app:/workspace<br>    working_dir: /workspace<br>    command: bash -lc &quot;python3 test_opencv_sift.py &amp;&amp; python3 test_pysift.py&quot;<br>    tty: true<br>    restart: &quot;no&quot;</pre><h4>Building and Running the Container</h4><p>Once all the required files have been created, execute the following command to build the Docker image and run the test application.</p><pre>docker compose up --build</pre><p>After the container starts successfully, review the console output to confirm that PySIFT has been installed correctly and is able to perform feature extraction as expected.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GD0jE6jW1KASOi9dEkZO8w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vMXM6JWcQR7AxPgnVa_zrQ.png" /></figure><p><em>Thanks for reading this article !!</em></p><p>If you enjoyed this article, please click on the clap button 👏 and share to help others find it!</p><p>The full source code for this tutorial can be found here,</p><p><a href="https://github.com/codemaker2015/pysift-examples">GitHub - codemaker2015/pysift-examples: A small experimental repository for comparing SIFT-based feature extraction and matching between OpenCV and the PySIFT ecosystem.</a></p><p><strong>References</strong></p><ul><li><a href="https://github.com/SivaIITM/PySIFT">GitHub - SivaIITM/PySIFT: A Python, GPU-resident SIFT implementation that matches OpenCV SIFT accuracy while running 26% faster end-to-end with 4x matching speedup. Zero-copy DLPack interop keeps tensors on the GPU across the full pipeline -- no PCIe round-trips.</a></li><li><a href="https://arxiv.org/html/2605.17869v1">PySIFT: GPU-Resident Deterministic SIFT for Deep Learning Vision Pipelines</a></li><li><a href="https://www.kaggle.com/code/sivakumarksce24d040/pysift-tutorial">pysift_tutorial</a></li><li><a href="https://huggingface.co/spaces/sivaIITM/PySIFT">PySIFT Course - a Hugging Face Space by sivaIITM</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9de6791a181c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Practical Guide to Training AI Agents with Microsoft Agent Lightning]]></title>
            <link>https://codemaker2016.medium.com/a-practical-guide-to-training-ai-agents-with-microsoft-agent-lightning-52921b0ed8af?source=rss-b7a60e472dfc------2</link>
            <guid isPermaLink="false">https://medium.com/p/52921b0ed8af</guid>
            <category><![CDATA[reinforcement-learning]]></category>
            <category><![CDATA[ai-agents-in-action]]></category>
            <category><![CDATA[agent-lightning]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[large-language-models]]></category>
            <dc:creator><![CDATA[Vishnu Sivan]]></dc:creator>
            <pubDate>Sun, 22 Mar 2026 13:43:01 GMT</pubDate>
            <atom:updated>2026-03-22T13:52:09.952Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mdOtLEMkO7B0SCbUGK8QEg.png" /></figure><p>Artificial intelligence agents are rapidly transforming the way we interact with software. Powered by large language models (LLMs), these agents can answer questions, automate workflows, and integrate with external tools and data sources. However, training and improving AI agents — especially for complex, multi-step tasks — has traditionally been difficult. Developers often need to modify large portions of code, design custom training loops, and manage complex reinforcement learning pipelines.</p><p>Microsoft’s Agent Lightning framework aims to simplify this process. It introduces a new approach that separates how an agent operates from how it learns, allowing existing AI agents to be trained with reinforcement learning with minimal or even zero code changes. Instead of redesigning the agent architecture, developers can plug Agent Lightning into their current systems and allow the agent to improve through real-world interactions.</p><p>In this article, we explore how to build and train an AI agent using the Agent Lightning framework.</p><h3>Getting Started</h3><h4>Table of contents</h4><ul><li><a href="#8435">What is Agent Lightning</a></li><li><a href="#96fc">Why Agent Lightning Matters</a></li><li><a href="#7dc3">Three-Component Architecture of Agent Lightning</a></li><li><a href="#c55b">How Agent Lightning Works</a></li><li><a href="#822e">Hands-on 1: Manual Prompt Search with AgentLightning and OpenAI</a></li><li><a href="#4184">Hands-on 2: Building a Trainable LLM Agent with AgentLightning</a></li><li><a href="#3830">Hands-on 3: Sentiment Analysis Agent with AgentLightning</a></li><li><a href="#a1da">Hands-on 4: LangGraph SQL Agent with AgentLightning</a></li></ul><h4>What is Agent Lightning?</h4><p>Agent Lightning is an open-source framework developed by Microsoft that enables AI agents to be trained and optimized using reinforcement learning (RL) based on their real-world execution behavior.</p><p>Traditionally, improving an AI agent requires modifying the agent’s internal logic or redesigning parts of its architecture. Agent Lightning solves this problem by introducing an external training layer that observes how an agent behaves during execution. It captures the agent’s actions, states, and outcomes, and uses this data to improve the agent’s performance over time. This approach allows existing agents to become self-learning systems without rewriting their core logic, which is especially valuable in production environments.</p><p>At a conceptual level, Agent Lightning treats an agent’s execution as a Markov Decision Process (MDP). During each step of a task, the agent is in a specific state, generates an action (typically an LLM output), and receives a reward depending on whether the action helps achieve the task goal. These rewards become learning signals that guide the reinforcement learning process.</p><p>One of the biggest advantages of Agent Lightning is its framework-agnostic design. It can be integrated with agents built using popular frameworks such as LangChain, OpenAI Agents SDK, AutoGen, CrewAI, LangGraph, or even custom Python-based agents. In most cases, integration requires minimal or near-zero code changes.</p><p>The framework typically consists of a Python SDK and a training server. Developers wrap their existing agent logic in a lightweight interface (such as a LitAgent class), define how to evaluate the agent’s output using a reward function, and start the training process. Agent Lightning then collects execution traces, processes them through its hierarchical reinforcement learning algorithm (LightningRL), and updates the model or prompt configuration to improve the agent’s performance.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Ffd.xuwubk.eu.org%3A443%2Fhttps%2Fwww.youtube.com%2Fembed%2FPRWvU7DQuZI%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Ffd.xuwubk.eu.org%3A443%2Fhttps%2Fwww.youtube.com%2Fwatch%3Fv%3DPRWvU7DQuZI&amp;image=https%3A%2F%2Ffd.xuwubk.eu.org%3A443%2Fhttps%2Fi.ytimg.com%2Fvi%2FPRWvU7DQuZI%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/d71655f466e7dbf7f9fda5cc1eaee323/href">https://medium.com/media/d71655f466e7dbf7f9fda5cc1eaee323/href</a></iframe><p><strong>Key Features</strong></p><ul><li><strong>Framework Agnostic</strong> — Compatible with major agent frameworks such as LangChain, OpenAI Agents SDK, AutoGen, CrewAI, and custom Python-based agents.</li><li><strong>Reinforcement Learning Optimization</strong> — Improves agent performance over time by applying reinforcement learning to learn from successes, failures, and feedback signals.</li><li><strong>Multiple Training Methods</strong> — Supports reinforcement learning, automatic prompt optimization, and supervised fine-tuning.</li><li><strong>Multi-Agent Coordination</strong> — Enables optimization and collaboration across multiple agents within complex multi-agent systems.</li><li><strong>Execution Monitoring &amp; Error Tracking</strong> — Uses the Lightning Server to monitor agent execution, detect errors, and track performance.</li><li><strong>Selective Optimization</strong> — Allows developers to optimize specific agents or components within a larger system.</li><li><strong>Flexible and Extensible Architecture</strong> — Provides open interfaces for customizing algorithms, reward strategies, and training workflows.</li></ul><p><strong>Core Components</strong></p><ul><li><strong>LightningStore</strong> — Central system that stores tasks, resources, and execution traces.</li><li><strong>Tracer</strong> — Captures structured data such as prompts, tool calls, and reward signals.</li><li><strong>Algorithm Engine</strong> — Processes traces and learns improved strategies or prompts.</li><li><strong>Trainer</strong> — Orchestrates the training workflow and updates the inference system.</li></ul><h4>Why Agent Lightning Matters</h4><p>Many popular agent frameworks such as LangChain, LangGraph, CrewAI, and AutoGen enable developers to build powerful AI agents capable of reasoning step-by-step and interacting with tools. However, most of these systems operate using static prompts, fixed workflows, and unchanged model parameters. As a result, the agents do not learn from their past interactions or improve automatically over time.</p><p>This limitation becomes a major challenge in real-world applications where tasks are complex and environments constantly change. Developers often need to manually refine prompts, adjust logic, or redesign workflows, which becomes difficult to maintain as systems scale.</p><p>Agent Lightning addresses this gap by introducing an automated learning pipeline powered by reinforcement learning. By enabling agents to learn continuously from real-world usage, Agent Lightning transforms traditional static agents into adaptive systems. This capability is particularly valuable for enterprise workflows, long-running automation systems, and multi-step processes, where reliability and accuracy improve only through repeated execution and feedback.</p><h4>Three-Component Architecture of Agent Lightning</h4><p>Agent Lightning consists of two main components: the Lightning Server and the Lightning Client. Together, they act as a lightweight intermediate layer that connects agent frameworks with LLM training systems. The framework exposes an OpenAI-compatible LLM API within the training infrastructure, enabling existing agents to integrate with the training system without modifying their original code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/0*oustd5X0VJq787mU.png" /><figcaption><a href="https://www.microsoft.com/en-us/research/project/agent-lightning/">Agent Lightning — Microsoft Research</a></figcaption></figure><p>The architecture of Agent Lightning is built around three core components that work together to create a continuous training loop for AI agents. These components manage learning, execution, and coordination, allowing agents to improve automatically through real-world interactions.</p><p><strong>1. Algorithm — The Learning Engine<br></strong>The Algorithm component is responsible for training and optimizing the agent. It analyzes how agents perform during task execution and uses that information to improve their behavior over time.</p><p>Its main responsibilities include:</p><ul><li>Assigning tasks (rollouts) for agents to execute</li><li>Analyzing execution traces collected from completed tasks</li><li>Learning from agent actions and outcomes</li><li>Updating resources such as model parameters or prompt templates</li></ul><p>Agent Lightning supports pluggable algorithms, allowing developers to choose or implement different optimization strategies. Built-in options include:</p><ul><li><strong>APO (Automatic Prompt Optimization) </strong>— Improves prompt templates using textual gradients and search strategies</li><li><strong>VERL-based Reinforcement Learning</strong> — Uses RL techniques such as PPO or GRPO to train agent policies</li><li><strong>Custom Algorithms </strong>— Developers can implement their own optimization logic through the Algorithm interface.</li></ul><p><strong>2. Runner — The Execution Worker<br></strong>The Runner is responsible for executing agent tasks and collecting telemetry data. It retrieves tasks from the system, runs the agent using the latest resources, and records detailed execution traces.</p><p>Key responsibilities include:</p><ul><li>Fetching tasks from the queue</li><li>Loading the latest models or prompt configurations</li><li>Running the agent to complete tasks</li><li>Capturing execution traces through a tracer system</li><li>Sending collected data back for training</li></ul><p>The default LitAgentRunner works with agents wrapped as LitAgent instances, but developers can create custom runners for specialized execution environments.</p><p><strong>3. LightningStore — The Coordination Hub<br></strong>The LightningStore acts as the central system that coordinates all components. It manages tasks, stores execution traces, and keeps track of resource versions used during training.</p><p>Its main functions include:</p><ul><li>Managing the task queue for agent rollouts</li><li>Storing detailed execution traces (spans)</li><li>Tracking versions of models, prompts, and resources</li><li>Handling retries and failure management during training</li></ul><p>Agent Lightning provides multiple storage implementations such as InMemoryLightningStore, SqliteLightningStore, and MongoLightningStore, enabling the framework to scale from local experiments to distributed production environments.</p><h4>How Agent Lightning Works</h4><p>Agent Lightning acts as middleware between reinforcement learning (RL) algorithms and agent environments, providing standardized interfaces that allow scalable training and coordination across different system components.</p><ul><li>The Agent Runner manages agents as they execute tasks. It distributes work, monitors progress, and collects execution results and traces.</li><li>The Algorithm component handles the training process and hosts the LLMs used for both inference and optimization.</li><li>The LightningStore serves as the system’s central data repository.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QuTfE2SCNTW_d1to.png" /><figcaption><a href="https://www.microsoft.com/en-us/research/blog/agent-lightning-adding-reinforcement-learning-to-ai-agents-without-code-rewrites/">Agent Lightning: Adding reinforcement learning to AI agents without code rewrites</a></figcaption></figure><p><strong>Execution flow</strong></p><ol><li><strong>Task Execution<br></strong>The Lightning Server retrieves tasks from a task pool and sends them to the agent. The agent then attempts to complete the task using its native workflow, which may include tool usage, multi-turn interactions, or coordination with other agents.</li></ol><p><strong>2. Trace Collection<br></strong>Agent Lightning uses a sidecar-based monitoring approach to capture execution data without interfering with the agent’s internal logic. During each run, the system collects structured telemetry such as execution traces, agent actions, errors and reward signals. These traces are converted into state–action–reward–next state transitions, which represent the agent’s decision-making steps during the task.</p><p><strong>3. Training and Optimization Loop<br></strong>The collected traces are organized into training data and processed by a reinforcement learning framework (such as VERL). RL algorithms like GRPO analyze the agent’s behavior and update resources such as prompt templates or model weights.</p><p>The updated configuration is then used in the next task cycle, creating a continuous feedback loop where the agent improves its performance through repeated execution and learning. Agent Lightning also supports intermediate reward signals, allowing smaller rewards for successful steps within a task to accelerate training.</p><h4>Basic Integration</h4><p>Integrating Agent Lightning into an existing agent is simple and requires minimal changes to the original code. The core idea is to add lightweight hooks that capture the agent’s actions and observations during execution, without modifying its internal logic.</p><p>By using Agent Lightning’s helper functions, the agent can report key events such as inputs and outputs, enabling the training system to collect data for optimization.</p><pre>import agentlightning as agl<br><br># Existing agent logic remains unchanged<br>def your_existing_agent_function(query):<br>    # Capture the input action<br>    agl.emit_action(&quot;user_query&quot;, query)<br>    <br>    # Execute agent logic<br>    response = your_agent.process(query)<br>    <br>    # Capture the output observation<br>    agl.emit_observation(&quot;agent_response&quot;, response)<br>    <br>    return response</pre><h3>Hands-on 1: <strong>Manual Prompt Search with AgentLightning and OpenAI</strong></h3><p>In this hands-on tutorial, we will build and train a simple AI agent using Agent Lightning in Google Colab. The goal is to demonstrate how reinforcement learning can improve an agent’s performance over time through real interactions.</p><p>We will set up both the Lightning Server and Client, create a basic QA agent, and connect it to the training pipeline. As the agent executes tasks, Agent Lightning will collect feedback and continuously refine its behavior, making it progressively smarter.</p><p>Agent Lightning supports multiple learning algorithms, including:</p><ul><li><strong>APO (Automatic Prompt Optimization)</strong> — requires additional libraries such as POML</li><li><strong>VERL (Reinforcement Learning)</strong> — integrates with frameworks like PyTorch and vLLM, and requires GPU support</li></ul><p>Note that VERL setup may take 20–40 minutes, depending on dependencies and environment configuration.</p><h4>Setting Up the Environment</h4><p>For this tutorial, we will use Google Colab with GPU support to ensure efficient training.</p><ol><li>Open Google Colab and sign in with your Google account</li><li>Create a new notebook</li><li>Go to Runtime → Change runtime type</li></ol><ul><li>Set Hardware Accelerator to GPU</li><li>Select T4 GPU (recommended)</li></ul><p>4. Click Save</p><h4>Installing dependencies</h4><ul><li>Install the required libraries using the following command.</li></ul><pre>!pip install agentlightning</pre><ul><li>Build a simple qa agent and enable training using Agent Lightning.</li></ul><pre>import os, asyncio, nest_asyncio, logging<br>from getpass import getpass<br><br>os.environ[&quot;AGENTOPS_DISABLE_AUTO_INSTRUMENTATION&quot;] = &quot;true&quot;<br>os.environ[&quot;OPENAI_API_KEY&quot;] = &quot;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&quot;<br><br>logging.getLogger(&quot;agentlightning.tracer.otel&quot;).setLevel(logging.ERROR)<br>logging.getLogger(&quot;opentelemetry.trace&quot;).setLevel(logging.ERROR)<br><br>from agentlightning import (<br>    LitAgent, LitAgentRunner, OtelTracer,<br>    emit_reward, NamedResources, Rollout,<br>    InMemoryLightningStore,<br>)<br>from agentlightning.types import PromptTemplate<br>from typing import Any, Dict, Optional<br>import openai<br><br>nest_asyncio.apply()<br>MODEL = os.getenv(&quot;MODEL&quot;, &quot;gpt-4o-mini&quot;)<br><br>_reward_log: list[float] = []<br><br>class QAAgent(LitAgent):<br>    def rollout(self, task: Dict[str, Any], resources: NamedResources, rollout: Rollout) -&gt; Optional[float]:<br>        sys_prompt = resources[&quot;system_prompt&quot;].template<br>        user = task[&quot;prompt&quot;]<br>        gold = task.get(&quot;answer&quot;, &quot;&quot;).strip().lower()<br><br>        try:<br>            r = openai.chat.completions.create(<br>                model=MODEL,<br>                messages=[<br>                    {&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: sys_prompt},<br>                    {&quot;role&quot;: &quot;user&quot;,   &quot;content&quot;: user},<br>                ],<br>                temperature=0.2,<br>            )<br>            pred = r.choices[0].message.content.strip()<br>        except Exception as e:<br>            pred = f&quot;[error] {e}&quot;<br><br>        def score(pred: str, gold: str) -&gt; float:<br>            P       = pred.lower()<br>            base    = 1.0 if gold and gold in P else 0.0<br>            gt      = set(gold.split()); pr = set(P.split())<br>            inter   = len(gt &amp; pr); denom = (len(gt) + len(pr)) or 1<br>            overlap = 2 * inter / denom<br>            brevity = 0.2 if base == 1.0 and len(P.split()) &lt;= 8 else 0.0<br>            return max(0.0, min(1.0, 0.7 * base + 0.25 * overlap + brevity))<br><br>        reward = float(score(pred, gold))<br>        emit_reward(reward)<br>        _reward_log.append(reward)<br>        print(f&quot;  Q: {user!r:45s} | Pred: {pred!r:20s} | Gold: {gold!r:12s} | R: {reward:.3f}&quot;)<br>        return reward<br><br>TASKS = [<br>    {&quot;prompt&quot;: &quot;Capital of France?&quot;,             &quot;answer&quot;: &quot;Paris&quot;},<br>    {&quot;prompt&quot;: &quot;Who wrote Pride and Prejudice?&quot;, &quot;answer&quot;: &quot;Jane Austen&quot;},<br>    {&quot;prompt&quot;: &quot;2+2 = ?&quot;,                        &quot;answer&quot;: &quot;4&quot;},<br>]<br><br>PROMPTS = [<br>    &quot;You are a terse expert. Answer with only the final fact, no sentences.&quot;,<br>    &quot;You are a helpful, knowledgeable AI. Prefer concise, correct answers.&quot;,<br>    &quot;Answer as a rigorous evaluator; return only the canonical fact.&quot;,<br>    &quot;Be a friendly tutor. Give the one-word answer if obvious.&quot;,<br>]<br><br>async def run_prompt_search():<br>    store  = InMemoryLightningStore()<br>    agent  = QAAgent()<br>    tracer = OtelTracer()<br>    runner = LitAgentRunner(tracer=tracer)<br><br>    results = []<br><br>    for sp in PROMPTS:<br>        print(f&quot;\n{&#39;=&#39;*60}&quot;)<br>        print(f&quot;Prompt: {sp}&quot;)<br>        print(&#39;=&#39;*60)<br><br>        await store.update_resources(<br>            resources_id=&quot;default&quot;,<br>            resources={&quot;system_prompt&quot;: PromptTemplate(template=sp, engine=&quot;f-string&quot;)}<br>        )<br>        _reward_log.clear()<br><br>        with runner.run_context(agent=agent, store=store):<br>            for t in TASKS:<br>                await runner.step(t)<br><br>        avg = sum(_reward_log) / len(_reward_log) if _reward_log else 0.0<br>        print(f&quot;\n  → Prompt avg: {avg:.3f}&quot;)<br>        results.append((sp, avg))<br><br>    best = max(results, key=lambda x: x[1])<br>    print(f&quot;\n{&#39;=&#39;*60}&quot;)<br>    print(f&quot;BEST PROMPT : {best[0]}&quot;)<br>    print(f&quot;BEST SCORE  : {best[1]:.3f}&quot;)<br><br>asyncio.run(run_prompt_search())</pre><p>Here’s a step-by-step walkthrough of what the code does:</p><p>1. Setup &amp; Imports Configures API keys, suppresses noisy logs, applies nest_asyncio (needed to run asyncio inside Jupyter), and imports AgentLightning components.</p><p>2. QAAgent — the core agent Subclasses LitAgent and implements rollout(), which is called once per task:</p><ul><li>Pulls the current system prompt from resources[&quot;system_prompt&quot;]</li><li>Calls OpenAI with that prompt + the task’s question</li><li>Scores the prediction against the gold answer using a custom score() function that combines exact match, token overlap, and a brevity bonus</li><li>Emits the reward back to AgentLightning via emit_reward() and logs it to _reward_log</li></ul><p>3. TASKS and PROMPTS TASKS is a small 3-question QA dataset. PROMPTS is a list of 4 candidate system prompts with different styles (terse, helpful, rigorous, friendly) — these are what we&#39;re trying to compare.</p><p>4. run_prompt_search() — the search loop This is the main async function that runs a manual prompt search:</p><ul><li>Creates an InMemoryLightningStore, a QAAgent, and a LitAgentRunner</li><li>Loops over each candidate prompt, updates it in the store, then runs all 3 tasks through the agent</li><li>Collects the average reward for each prompt</li><li>At the end, picks and prints the best-scoring prompt</li></ul><p>5. Entry point asyncio.run(run_prompt_search()) kicks everything off. The result is a ranked comparison of which system prompt style gets the highest average reward across the 3 QA tasks.</p><p>6. Run the cell to display the output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Vg4-aUrh-nFZFUbP9592CQ.png" /></figure><h3>Hands-on 2: Building a Trainable LLM Agent with AgentLightning</h3><p>The hands-on walks you through building a minimal but complete trainable LLM agent using the AgentLightning framework. You’ll learn how agents receive tasks, call an LLM, score their own outputs via a reward function, and report results back to a training loop — the core cycle behind reinforcement learning from human/environment feedback (RLHF-style training).</p><p><strong>Part 1 — </strong><strong>compute_reward()</strong></p><pre>def compute_reward(output: str, expected: str) -&gt; float:<br>    &quot;&quot;&quot;<br>    Simple reward logic. +1.0 for correct answer, -1.0 for wrong answer.<br>    &quot;&quot;&quot;<br>    return 1.0 if output.strip().lower() == expected.strip().lower() else -1.0</pre><p>The simplest possible reward function. It does an exact string match between the agent’s output and the expected answer, returning +1.0 for a correct answer and -1.0 for a wrong one. In real-world agents this could be a more nuanced scorer (e.g. F1, BLEU, SQL execution match).</p><p><strong>Part 2 — </strong><strong>SimpleAgent</strong></p><pre>import agentlightning as agl<br>from agentlightning import LitAgent, emit_reward, NamedResources, Rollout<br>from openai import OpenAI<br>from typing import Any, Dict, Optional<br><br>class SimpleAgent(LitAgent):<br>    &quot;&quot;&quot;<br>    A LitAgent that uses an LLM resource (like LitSQLAgent) instead of<br>    hardcoded rule-based logic.<br>    &quot;&quot;&quot;<br><br>    def __init__(self, system_prompt: str = &quot;You are a helpful assistant.&quot;):<br>        super().__init__()<br>        self.system_prompt = system_prompt<br><br>    def rollout(self, task: Dict[str, Any], resources: NamedResources, rollout: Rollout) -&gt; Optional[float]:<br>        # 1. Extract the LLM resource (same as LitSQLAgent)<br>        llm: agl.LLM = resources[&quot;main_llm&quot;]<br><br>        # 2. Build the OpenAI client using the LLM resource endpoint<br>        client = OpenAI(<br>            base_url=llm.endpoint,<br>            api_key=llm.api_key,<br>        )<br><br>        # 3. Call the LLM (instead of hardcoded if/elif logic)<br>        question = task.get(&quot;input&quot;, &quot;&quot;)<br>        response = client.chat.completions.create(<br>            model=llm.model,<br>            messages=[<br>                {&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: self.system_prompt},<br>                {&quot;role&quot;: &quot;user&quot;,   &quot;content&quot;: question},<br>            ],<br>            **(llm.sampling_parameters or {}),<br>        )<br>        output = response.choices[0].message.content.strip()<br><br>        # 4. Compute reward and emit it (same as LitSQLAgent)<br>        expected = task.get(&quot;expected_output&quot;, &quot;&quot;)<br>        reward_value = compute_reward(output, expected)<br>        emit_reward(reward_value)<br><br>        print(f&quot;[Agent] Input:    {question}&quot;)<br>        print(f&quot;[Agent] Output:   {output}&quot;)<br>        print(f&quot;[Agent] Expected: {expected}&quot;)<br>        print(f&quot;[Agent] Reward:   {reward_value}&quot;)<br>        print(&quot;-&quot; * 40)<br><br>        return float(reward_value)</pre><p>The agent subclasses LitAgent and implements one required method: rollout(). This is called once per task during training or evaluation. Inside it:</p><ul><li><strong>Extracts the LLM resource</strong> from resources[&quot;main_llm&quot;] — this is injected by the runner, not hardcoded</li><li><strong>Builds an OpenAI client</strong> using the endpoint and API key from that resource</li><li><strong>Calls the LLM</strong> with a system prompt + the task’s question</li><li><strong>Scores the response</strong> via compute_reward() and calls emit_reward() to report it back to AgentLightning&#39;s training loop</li><li><strong>Returns the reward</strong> as a float for immediate use</li></ul><p><strong>Part 3 — Training Data and LLM Resource</strong></p><pre>import os<br>os.environ[&quot;AGENTOPS_DISABLE_AUTO_INSTRUMENTATION&quot;] = &quot;true&quot;<br><br>import asyncio<br>import nest_asyncio<br>nest_asyncio.apply()<br><br>import agentlightning as agl<br>from agentlightning import (<br>    LitAgentRunner,<br>    InMemoryLightningStore,<br>    OtelTracer,<br>    LLM,<br>    Trainer,<br>)<br><br># Training data: list of task inputs your agent will be tested on<br>TRAINING_DATA = [<br>    {&quot;input&quot;: &quot;What is 2+2?&quot;,   &quot;expected_output&quot;: &quot;2 + 2 equals 4.&quot;},<br>    {&quot;input&quot;: &quot;Who is Newton?&quot;, &quot;expected_output&quot;: &quot;Newton is the father of classical mechanics.&quot;},<br>    {&quot;input&quot;: &quot;Calculate 2+2&quot;,  &quot;expected_output&quot;: &quot;2 + 2 equals 4.&quot;},<br>]<br><br># Dummy LLM resource — required by the runner even for rule-based agents.<br># Replace endpoint/model/api_key with real values if you use an actual LLM.<br><br>def get_llm_resource(temperature: float = 0.7) -&gt; agl.LLM:<br>    import os<br>    os.environ[&quot;OPENAI_API_KEY&quot;] = &quot;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&quot;<br><br>    return agl.LLM(<br>        endpoint=&quot;https://api.openai.com/v1&quot;,<br>        model=&quot;gpt-4o-mini&quot;,<br>        api_key=os.environ.get(&quot;OPENAI_API_KEY&quot;, &quot;&quot;),<br>        sampling_parameters={&quot;temperature&quot;: temperature},<br>    )</pre><p>TRAINING_DATA is a list of task dictionaries — each has an input (what the agent sees) and expected_output (what it should produce, used by the reward function).</p><p>get_llm_resource() creates an agl.LLM object that wraps the model config (endpoint, model name, API key, sampling parameters). This is passed into the store so the agent can access it at runtime without hardcoding credentials inside the agent class.</p><p><strong>Part 4 — </strong><strong>run_step_test() and </strong><strong>run_trainer()</strong></p><pre>def run_step_test():<br>    &quot;&quot;&quot;<br>    Directly tests the agent by calling runner.step() for each training sample.<br>    This bypasses the full Trainer loop and is the easiest way to verify<br>    that agent.py and reward.py are connected and working correctly.<br>    &quot;&quot;&quot;<br>    print(&quot;=&quot; * 40)<br>    print(&quot;Running step-by-step test...&quot;)<br>    print(&quot;=&quot; * 40)<br><br>    tracer = OtelTracer()<br>    store  = InMemoryLightningStore()<br>    agent  = SimpleAgent()<br>    runner = LitAgentRunner(tracer=tracer)<br><br>    async def _run():<br>        # Seed the store with resources before running - the runner requires this<br>        await store.update_resources(<br>            resources_id=&quot;default&quot;, <br>            resources={&quot;main_llm&quot;: get_llm_resource(temperature=0.7)}<br>        )<br><br>        with runner.run_context(agent=agent, store=store):<br>            for task in TRAINING_DATA:<br>                rollout = await runner.step(task)<br>                print(f&quot;[Rollout status] {rollout.status}\n&quot;)<br><br>    asyncio.run(_run())<br>    print(&quot;Step test completed successfully!&quot;)<br><br>def run_trainer():<br>    &quot;&quot;&quot;<br>    Full Trainer-based training loop.<br>    Uses dev=True for a dry-run. Switch to dev=False and trainer.fit()<br>    for real training with an algorithm.<br>    &quot;&quot;&quot;<br>    print(&quot;=&quot; * 40)<br>    print(&quot;Running Trainer (dev mode)...&quot;)<br>    print(&quot;=&quot; * 40)<br><br>    trainer = Trainer(<br>        runner=LitAgentRunner,<br>        dev=True,<br>        initial_resources={&quot;main_llm&quot;: get_llm_resource(temperature=0.7)},<br>    )<br>    # Instantiate the agent<br>    agent = SimpleAgent()<br>    trainer.dev(agent=agent, train_dataset=TRAINING_DATA, val_dataset=TRAINING_DATA)<br>    print(&quot;Trainer dev run completed successfully!&quot;)<br><br>if __name__ == &quot;__main__&quot;:<br>    # Step 1: Test agent + reward wiring directly (recommended first)<br>    run_step_test()<br><br>    # Step 2: Uncomment below to test via full Trainer loop<br>    run_trainer()</pre><p><strong>run_step_test </strong>is the recommended first test — it bypasses the full Trainer and directly calls runner.step() for each task. This lets you verify that the agent, LLM, and reward function are all wired correctly before introducing the complexity of a training algorithm. The flow is:</p><ul><li>Seed the store with the LLM resource</li><li>Open a run_context that binds the agent and store to the runner</li><li>Step through each task one by one, printing the rollout status</li></ul><p><strong>run_trainer </strong>is the full Trainer-based loop. dev=True runs a dry-run — it exercises the full pipeline without running a real optimization algorithm, which is useful for integration testing. Switching to dev=False and calling trainer.fit() with a real algorithm (like APO) would enable actual prompt or model optimization.</p><p><strong>Part 5 — Execute the code</strong></p><p>Run the cells sequentially to display the output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YQOnpZokf3w1mWtoN8B-IA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4n0a4QgcHycabZWyMSqlmA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X-qhQSRP4RYI63h1AjSn8w.png" /></figure><h3>Hands-on 3: Sentiment Analysis Agent with AgentLightning</h3><p>This hands-on demonstrates how to build a prompt-optimizing sentiment analysis agent using AgentLightning. The agent classifies text as “positive” or “negative” using an LLM, then automatically tests multiple prompt styles to find which one performs best — all without writing a training loop from scratch.</p><p><strong>Step 1 — Install &amp; Setup</strong></p><pre>!pip install poml</pre><p>Installs poml, a dependency required by AgentLightning&#39;s APO (Automatic Prompt Optimization) module.</p><p><strong>Step 2 — Imports &amp; Environment</strong></p><p>Sets the OpenAI key, disables AgentOps auto-instrumentation (avoids TracerProvider conflicts), and silences noisy but harmless OpenTelemetry warnings.</p><pre>import os<br>import agentlightning as agl<br>from openai import OpenAI, AsyncOpenAI<br>from agentlightning import LitAgent, LitAgentRunner, OtelTracer, emit_reward, NamedResources, Rollout, InMemoryLightningStore<br>from typing import TypedDict<br>import logging<br>import asyncio<br>import nest_asyncio<br>nest_asyncio.apply()<br><br>os.environ[&quot;OPENAI_API_KEY&quot;] = &quot;...&quot;<br>os.environ[&quot;AGENTOPS_DISABLE_AUTO_INSTRUMENTATION&quot;] = &quot;true&quot;<br>logging.getLogger(&quot;agentlightning.tracer.otel&quot;).setLevel(logging.ERROR)</pre><p><strong>Step 3 — Create agents</strong></p><p>The agent subclasses LitAgent and implements rollout(), which runs once per task. It pulls the LLM and prompt template from resources, calls OpenAI, compares the prediction to the gold label, and reports a binary reward (1.0 correct, 0.0 wrong).</p><pre># ── Agent ─────────────<br>class SentimentAgent(LitAgent):<br>    def rollout(self, task: Dict[str, Any], resources: NamedResources, rollout: Rollout) -&gt; Optional[float]:<br>        # Pull resources exactly like QAAgent pulls &quot;system_prompt&quot;<br>        llm         = resources[&quot;main_llm&quot;]<br>        prompt_tmpl = resources[&quot;prompt_template&quot;].template<br><br>        # Format the prompt with the task text<br>        prompt = prompt_tmpl.format(**task)<br><br>        # Build client from resource<br>        client = OpenAI(<br>            base_url=llm.endpoint,<br>            api_key=llm.api_key,<br>        )<br><br>        response = client.chat.completions.create(<br>            model=llm.model,<br>            messages=[{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: prompt}],<br>            **(llm.sampling_parameters or {}),<br>        )<br><br>        pred = response.choices[0].message.content.strip().lower().rstrip(&quot;.&quot;)<br>        gold = task[&quot;expected_label&quot;].lower()<br>        reward = 1.0 if pred == gold else 0.0<br><br>        emit_reward(reward)<br>        print(f&quot;  Text: {task[&#39;text&#39;]!r:40s} | Pred: {pred!r:12s} | Gold: {gold!r:12s} | R: {reward:.1f}&quot;)<br>        return reward<br><br># ── Data ──────────────────────────<br>TRAIN_TASKS = [<br>    {&quot;text&quot;: &quot;I love this product!&quot;,                &quot;expected_label&quot;: &quot;positive&quot;},<br>    {&quot;text&quot;: &quot;This is terrible and disappointing.&quot;, &quot;expected_label&quot;: &quot;negative&quot;},<br>    {&quot;text&quot;: &quot;An excellent experience overall.&quot;,    &quot;expected_label&quot;: &quot;positive&quot;},<br>    {&quot;text&quot;: &quot;I will never buy this again.&quot;,        &quot;expected_label&quot;: &quot;negative&quot;},<br>]<br><br>PROMPTS = [<br>    &#39;Classify the sentiment. Reply with exactly one word: positive or negative.\nText: &quot;{text}&quot;\nSentiment:&#39;,<br>    &#39;Is the following text positive or negative? Reply with one word only.\nText: &quot;{text}&quot;&#39;,<br>    &#39;Sentiment analysis: respond only with &quot;positive&quot; or &quot;negative&quot;.\nInput: &quot;{text}&quot;&#39;,<br>]<br><br># ── Resources ────────────────────────────────<br>def make_resources(prompt_template: str) -&gt; dict:<br>    return {<br>        &quot;main_llm&quot;: agl.LLM(<br>            model=&quot;gpt-4o-mini&quot;,<br>            endpoint=&quot;https://api.openai.com/v1&quot;,<br>            api_key=os.environ[&quot;OPENAI_API_KEY&quot;],<br>            sampling_parameters={&quot;temperature&quot;: 0.2},<br>        ),<br>        &quot;prompt_template&quot;: PromptTemplate(template=prompt_template, engine=&quot;f-string&quot;),<br>    }<br><br># ── Prompt search loop ──────────────────<br>async def run_sentiment_search():<br>    store  = InMemoryLightningStore()<br>    agent  = SentimentAgent()<br>    tracer = OtelTracer()<br>    runner = LitAgentRunner(tracer=tracer)<br><br>    results = []<br><br>    for prompt_tmpl in PROMPTS:<br>        print(f&quot;\n{&#39;=&#39;*60}&quot;)<br>        print(f&quot;Prompt: {prompt_tmpl[:60]}...&quot;)<br>        print(&#39;=&#39;*60)<br><br>        await store.update_resources(<br>            resources_id=&quot;default&quot;,<br>            resources=make_resources(prompt_tmpl)<br>        )<br><br>        reward_log = []<br>        with runner.run_context(agent=agent, store=store):<br>            for task in TRAIN_TASKS:<br>                result = await runner.step(task)<br>                if result and hasattr(result, &quot;reward&quot;):<br>                    reward_log.append(result.reward)<br><br>        avg = sum(reward_log) / len(reward_log) if reward_log else 0.0<br>        print(f&quot;\n  → Prompt avg: {avg:.3f}&quot;)<br>        results.append((prompt_tmpl, avg))<br><br>    best = max(results, key=lambda x: x[1])<br>    print(f&quot;\n{&#39;=&#39;*60}&quot;)<br>    print(f&quot;BEST PROMPT : {best[0]}&quot;)<br>    print(f&quot;BEST SCORE  : {best[1]:.3f}&quot;)<br><br>asyncio.run(run_sentiment_search())</pre><p><strong>Step 4 — Execute the code</strong></p><p>Run the cells sequentially to display the output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1023/1*-2vxRkE-y9BtkTB2O-VePw.png" /></figure><h3>Hands-on 4: LangGraph SQL Agent with AgentLightning</h3><p>A notable example from Microsoft’s research highlights the use of Agent Lightning to train AI agents that can generate and iteratively refine SQL queries.</p><p>This hands-on builds a Text-to-SQL agent that takes natural language questions, generates SQLite queries using an LLM, executes them, and automatically retries on errors. It uses LangGraph for the query-generate-execute-fix loop and AgentLightning for training orchestration.</p><p><strong>Step 1 — Downloading the Dataset</strong></p><p>Download the Spider dataset from Google Drive and extract it into the working directory to prepare it for use in the tutorial.</p><pre>!gdown --fuzzy https://drive.google.com/file/d/1oi9J1jZP9TyM35L85CL3qeGWl2jqlnL6/view<br>!unzip -q spider-data.zip -d data &amp;&amp; rm spider-data.zip</pre><p><strong>Step 2 — Install dependencies</strong></p><p>Install all the dependencies such as agentlightning[verl] ,torch ,flash-attn , langchain, langchain-openai, langchain-community, langchain-text-splitters, and faiss-cpu .</p><pre>!pip install agentlightning[verl]<br><br># For Colab/Kaggle (CUDA 12.1):<br>!pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121<br><br># For CPU-only environments:<br># !pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu<br><br>!pip install flash-attn --no-build-isolation<br>!pip install pyairports --break-system-packages<br><br>!pip install langchain<br>!pip install langchain-openai<br>!pip install langchain-community<br>!pip install langchain-text-splitters<br>!pip install faiss-cpu</pre><p><strong>Step 3— Import libraries</strong></p><pre>import os<br>import sqlite3<br>from typing import TypedDict<br>from langgraph.graph import StateGraph<br>from langchain_openai import ChatOpenAI<br>from langchain_core.messages import HumanMessage, SystemMessage<br>import pandas as pd<br>import agentlightning as agl<br>import nest_asyncio<br>nest_asyncio.apply()<br><br>os.environ[&quot;AGENTOPS_DISABLE_AUTO_INSTRUMENTATION&quot;] = &quot;true&quot;<br>os.environ[&quot;OPENAI_API_BASE&quot;] = &quot;https://api.openai.com/v1&quot;<br>os.environ[&quot;OPENAI_API_KEY&quot;] = &quot;...&quot;<br><br># Silence noisy but harmless warnings<br>import logging<br>logging.getLogger(&quot;agentlightning.tracer.otel&quot;).setLevel(logging.ERROR)<br>logging.getLogger(&quot;agentlightning.tracer.agentops&quot;).setLevel(logging.ERROR)<br>logging.getLogger(&quot;opentelemetry.trace&quot;).setLevel(logging.ERROR)</pre><p><strong>Step 4— Define state schema</strong></p><p>Defines the data that flows between nodes in the LangGraph graph. Every node reads from and writes to this shared state dict. get_schema extracts the table definitions so the LLM knows what columns exist. run_sql executes a query and returns results or an error string — the error string is what triggers the retry loop.</p><pre># ── State schema (required by StateGraph) ──────────────────────<br>class SQLState(TypedDict):<br>    question:    str<br>    schema:      str<br>    query:       str<br>    result:      str<br>    error:       str<br>    attempts:    int<br><br># ── SQL helpers ───────────────────<br>def get_schema(database_path: str) -&gt; str:<br>    conn = sqlite3.connect(database_path.replace(&quot;sqlite:///&quot;, &quot;&quot;))<br>    cursor = conn.cursor()<br>    cursor.execute(&quot;SELECT sql FROM sqlite_master WHERE type=&#39;table&#39;&quot;)<br>    schema = &quot;\n&quot;.join(row[0] for row in cursor.fetchall() if row[0])<br>    conn.close()<br>    return schema<br><br>def run_sql(query: str, db_path: str):<br>    try:<br>        conn = sqlite3.connect(db_path.replace(&quot;sqlite:///&quot;, &quot;&quot;))<br>        cursor = conn.cursor()<br>        cursor.execute(query)<br>        result = cursor.fetchall()<br>        conn.close()<br>        return result<br>    except Exception as e:<br>        return f&quot;ERROR: {e}&quot;</pre><p><strong>Step 5— LangGraph Agent</strong></p><p>Builds a 4-node graph: write_query → execute_query → check_query → conditionally rewrite_query or end. The key routing logic is in should_rewrite — if there&#39;s an error and attempts remain, it loops back to fix the query; otherwise it exits.</p><pre># ── LangGraph agent builder ───────────────<br>    database_path: str,<br>    openai_base_url: str,<br>    model: str,<br>    sampling_parameters: dict,<br>    max_turns: int,<br>    truncate_length: int,<br>):<br>    llm = ChatOpenAI(<br>        base_url=openai_base_url,<br>        model=model,<br>        **sampling_parameters,<br>    )<br>    schema = get_schema(database_path)<br><br>    def write_query(state: SQLState) -&gt; SQLState:<br>        response = llm.invoke([<br>            SystemMessage(&quot;You are a SQL expert. Write a SQLite query to answer the question. Return ONLY the SQL query.&quot;),<br>            HumanMessage(f&quot;Schema:\n{schema}\n\nQuestion: {state[&#39;question&#39;]}&quot;),<br>        ])<br>        return {**state, &quot;query&quot;: response.content.strip(), &quot;attempts&quot;: 0}<br><br>    def execute_query(state: SQLState) -&gt; SQLState:<br>        result = run_sql(state[&quot;query&quot;], database_path)<br>        if isinstance(result, str) and result.startswith(&quot;ERROR&quot;):<br>            return {**state, &quot;error&quot;: result, &quot;result&quot;: &quot;&quot;}<br>        return {**state, &quot;result&quot;: str(result)[:truncate_length], &quot;error&quot;: &quot;&quot;}<br><br>    def check_query(state: SQLState) -&gt; SQLState:<br>        # Pass through; routing logic handled by conditional edge below<br>        return state<br><br>    def rewrite_query(state: SQLState) -&gt; SQLState:<br>        response = llm.invoke([<br>            SystemMessage(&quot;You are a SQL expert. Fix the SQL query based on the error. Return ONLY the corrected SQL query.&quot;),<br>            HumanMessage(<br>                f&quot;Schema:\n{schema}\n\nOriginal query: {state[&#39;query&#39;]}\n&quot;<br>                f&quot;Error: {state[&#39;error&#39;]}\nQuestion: {state[&#39;question&#39;]}&quot;<br>            ),<br>        ])<br>        return {**state, &quot;query&quot;: response.content.strip(), &quot;attempts&quot;: state[&quot;attempts&quot;] + 1}<br><br>    def should_rewrite(state: SQLState) -&gt; str:<br>        if state.get(&quot;error&quot;) and state[&quot;attempts&quot;] &lt; max_turns:<br>            return &quot;rewrite_query&quot;<br>        return &quot;__end__&quot;<br><br>    builder = StateGraph(SQLState)<br><br>    builder.add_node(&quot;write_query&quot;,   write_query)<br>    builder.add_node(&quot;execute_query&quot;, execute_query)<br>    builder.add_node(&quot;check_query&quot;,   check_query)<br>    builder.add_node(&quot;rewrite_query&quot;, rewrite_query)<br><br>    builder.add_edge(&quot;__start__&quot;,    &quot;write_query&quot;)<br>    builder.add_edge(&quot;write_query&quot;,  &quot;execute_query&quot;)<br>    builder.add_edge(&quot;execute_query&quot;,&quot;check_query&quot;)<br><br>    builder.add_conditional_edges(&quot;check_query&quot;, should_rewrite)<br>    builder.add_edge(&quot;rewrite_query&quot;, &quot;execute_query&quot;)<br><br>    return builder.compile()</pre><p><strong>Step 6— Reward function</strong></p><p>Rewards execution equivalence, not exact string match — two different SQL queries that return the same rows both get 1.0. This is more robust than comparing SQL strings directly.</p><pre># ── Reward function ────────────────────────<br>def evaluate_query(predicted_query: str, ground_truth_query: str, db_path: str, raise_on_error: bool = False) -&gt; float:<br>    try:<br>        result_pred = run_sql(predicted_query, db_path)<br>        result_true = run_sql(ground_truth_query, db_path)<br>        return 1.0 if result_pred == result_true else 0.0<br>    except Exception as e:<br>        if raise_on_error:<br>            raise<br>        return 0.0</pre><p><strong>Step 7— The Agent</strong></p><p>Subclasses LitAgent and wires everything together in rollout(). Builds the LangGraph agent from the injected LLM resource, runs it on the task, computes the reward, and emits it back to AgentLightning.</p><pre># ── Agent ─────────────────────────────────────────────────────────────────────<br>class LitSQLAgent(agl.LitAgent):<br>    def __init__(self, max_turns: int, truncate_length: int):<br>        super().__init__()<br>        self.max_turns = max_turns<br>        self.truncate_length = truncate_length<br><br>    def rollout(<br>        self,<br>        task: dict,<br>        resources: agl.NamedResources,<br>        rollout: agl.Rollout,<br>    ) -&gt; float:<br>        llm: agl.LLM = resources[&quot;main_llm&quot;]<br><br>        agent = build_langgraph_sql_agent(<br>            database_path=&quot;sqlite:///&quot; + task[&quot;db_id&quot;],<br>            openai_base_url=llm.endpoint,    # use endpoint directly<br>            model=llm.model,<br>            sampling_parameters=llm.sampling_parameters or {},<br>            max_turns=self.max_turns,<br>            truncate_length=self.truncate_length,<br>        )<br><br>        # Safely get langchain handler only if available<br>        callbacks = []<br>        tracer = self.get_tracer()<br>        if hasattr(tracer, &quot;get_langchain_handler&quot;):<br>            callbacks.append(tracer.get_langchain_handler())<br><br>        result = agent.invoke(<br>            {&quot;question&quot;: task[&quot;question&quot;], &quot;schema&quot;: &quot;&quot;, &quot;query&quot;: &quot;&quot;,<br>             &quot;result&quot;: &quot;&quot;, &quot;error&quot;: &quot;&quot;, &quot;attempts&quot;: 0},<br>            {&quot;callbacks&quot;: callbacks, &quot;recursion_limit&quot;: 100},<br>        )<br><br>        reward = evaluate_query(<br>            result.get(&quot;query&quot;, &quot;&quot;),<br>            task.get(&quot;query&quot;, task.get(&quot;sql&quot;, &quot;&quot;)),   # ground truth SQL<br>            task.get(&quot;db_path&quot;, f&quot;data/database/{task[&#39;db_id&#39;]}/{task[&#39;db_id&#39;]}.sqlite&quot;),<br>            raise_on_error=False,<br>        )<br>        print(f&quot;[Rollout] Q: {task[&#39;question&#39;][:50]} | Reward: {reward:.3f}&quot;)<br>        agl.emit_reward(reward)<br>        return reward</pre><p><strong>Step 8— VERL Config</strong></p><p>Configures the VERL reinforcement learning algorithm — uses GRPO, the hf (HuggingFace) rollout engine, and fine-tunes Qwen2.5-Coder-1.5B. Requires a GPU node to run.</p><pre># ── VERL config ───────────────────────────────────────────────────────────────<br>verl_config = {<br>    &quot;algorithm&quot;: {&quot;adv_estimator&quot;: &quot;grpo&quot;, &quot;use_kl_in_reward&quot;: False},<br>    &quot;data&quot;: {<br>        &quot;train_batch_size&quot;: 8,<br>        &quot;max_prompt_length&quot;: 4096,<br>        &quot;max_response_length&quot;: 2048,<br>    },<br>    &quot;actor_rollout_ref&quot;: {<br>        &quot;rollout&quot;: {<br>            &quot;name&quot;: &quot;hf&quot;,<br>            &quot;n&quot;: 4,<br>            &quot;tensor_model_parallel_size&quot;: 1,<br>            &quot;multi_turn&quot;: {&quot;format&quot;: &quot;hermes&quot;},<br>        },<br>        &quot;actor&quot;: {&quot;ppo_mini_batch_size&quot;: 8, &quot;optim&quot;: {&quot;lr&quot;: 1e-6}},<br>        &quot;model&quot;: {&quot;path&quot;: &quot;Qwen/Qwen2.5-Coder-1.5B-Instruct&quot;},<br>    },<br>    &quot;trainer&quot;: {<br>        &quot;n_gpus_per_node&quot;: 1,<br>        &quot;val_before_train&quot;: True,<br>        &quot;test_freq&quot;: 32,<br>        &quot;save_freq&quot;: 64,<br>        &quot;total_epochs&quot;: 1,<br>    },<br>}</pre><p><strong>Step 9— Run Modes</strong></p><p>run_dev() is for local testing — it uses the OpenAI API and a small 10-sample subset. run_training() is for full RL fine-tuning on the Spider dataset and requires a Ray cluster with GPUs. Always test with run_dev() first.</p><pre># ── Full training ───────────────────────<br>def run_training():<br>    agent     = LitSQLAgent(max_turns=3, truncate_length=1024)<br>    algorithm = agl.VERL(verl_config)<br><br>    trainer = agl.Trainer(<br>        n_runners=10,<br>        algorithm=algorithm,<br>        adapter=agl.TracerTraceToTriplet(agent_match=&quot;write|rewrite&quot;),<br>    )<br><br>    train_data = pd.read_parquet(&quot;data/train_spider.parquet&quot;).to_dict(&quot;records&quot;)<br>    val_data   = pd.read_parquet(&quot;data/test_dev_500.parquet&quot;).to_dict(&quot;records&quot;)<br><br>    trainer.fit(agent, train_data, val_dataset=val_data)<br><br><br># ── Dev / dry-run ─────────────────────────<br>def run_dev():<br>    agent = LitSQLAgent(max_turns=3, truncate_length=1024)<br><br>    trainer = agl.Trainer(<br>        n_workers=1,<br>        initial_resources={<br>            &quot;main_llm&quot;: agl.LLM(<br>                endpoint=os.environ[&quot;OPENAI_API_BASE&quot;],<br>                model=&quot;gpt-4o-mini&quot;,<br>                api_key=os.environ[&quot;OPENAI_API_KEY&quot;],<br>                sampling_parameters={&quot;temperature&quot;: 0.7},<br>            )<br>        },<br>    )<br><br>    # Load a small subset — make sure the parquet file path is correct<br>    dev_data = pd.read_parquet(&quot;data/test_dev_500.parquet&quot;).to_dict(&quot;records&quot;)[:10]<br><br>    trainer.dev(agent, dev_data)<br><br>if __name__ == &quot;__main__&quot;:<br>    run_dev()<br>    # run_training()</pre><p><strong>Step 10 — Execute the code</strong></p><p>Execute the <strong>run_dev()</strong> to display the output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SWBUDIil1JlXDwtbKYPi-Q.png" /></figure><p><em>Thanks for reading this article !!</em></p><p>If you enjoyed this article, please click on the clap button 👏 and share to help others find it!</p><p>The full source code for this tutorial can be found here,</p><ul><li><a href="https://colab.research.google.com/drive/1c93D61rpOHevfxbAe4qZbtjb2AjhGKlp?usp=sharing">Google Colab</a></li><li><a href="https://github.com/codemaker2015/agent-lightning-demo">GitHub - codemaker2015/agent-lightning-demo: agent lightning demo</a></li></ul><h4>Resources</h4><ul><li><a href="https://microsoft.github.io/agent-lightning/latest/how-to/train-first-agent/">Train the First Agent - Agent-lightning</a></li><li><a href="https://github.com/microsoft/agent-lightning/tree/main/docs/tutorials">agent-lightning/docs/tutorials at main · microsoft/agent-lightning</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=52921b0ed8af" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Zvec: Reimagining Vector Databases with SQLite-Style Simplicity]]></title>
            <link>https://codemaker2016.medium.com/zvec-reimagining-vector-databases-with-sqlite-style-simplicity-e76b247b6555?source=rss-b7a60e472dfc------2</link>
            <guid isPermaLink="false">https://medium.com/p/e76b247b6555</guid>
            <category><![CDATA[embedding]]></category>
            <category><![CDATA[python-programming]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[vector-database]]></category>
            <dc:creator><![CDATA[Vishnu Sivan]]></dc:creator>
            <pubDate>Sat, 28 Feb 2026 13:28:08 GMT</pubDate>
            <atom:updated>2026-02-28T13:28:08.417Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DTjHJD0y804h-cDw4IZR0g.png" /></figure><p>The research team at Alibaba Tongyi Lab has introduced Zvec, an open-source in-process vector database purpose-built for edge and on-device retrieval workloads. It described as “the SQLite of vector databases,” runs as a lightweight embedded library inside your application. It requires no external services, no background daemons, and no network communication layer.</p><p>As Retrieval-Augmented Generation (RAG), semantic search, and AI agents increasingly move toward local-first and privacy-preserving deployments, Zvec addresses a growing infrastructure gap: how to deliver high-performance vector search with full persistence and metadata support — without the operational complexity of a server-based system.</p><p>This article explores why Zvec matters, how it is architected, where it fits in the vector database ecosystem, and why embedded vector search may become a core building block for the next generation of AI applications.</p><h3>Getting Started</h3><h4>Table of contents</h4><ul><li><a href="#07a5">What is Zvec</a></li><li><a href="#984f">How Zvec Works</a></li><li><a href="#9016">Why Zvec Matters</a></li><li><a href="#0c21">Performance Benchmarks</a></li><li><a href="#4cca">RAG-Focused Features</a></li><li><a href="#184d"><strong>Hands-On 1: Creating Your First Embedded Vector Database with Zvec</strong></a></li><li><a href="#58a4">Setting up the environment</a></li><li><a href="#24d8">Creating and querying a Zvec collection</a></li><li><a href="#90a2"><strong>Hands-On 2: Building a Zvec-based FAQ Agent for customer queries</strong></a></li><li><a href="#2fef">Preparing sample FAQ data</a></li><li><a href="#62a5">Creating Zvec collection</a></li><li><a href="#e14b">Building retrieval function</a></li><li><a href="#1792">Connecting to LLM for answer generation</a></li><li><a href="#5c73">Testing FAQ agent</a></li></ul><h4>What is Zvec</h4><p>Zvec is an open-source, in-process vector database developed by Alibaba that makes high-performance similarity search easy to embed directly into applications without running a separate server or service. It is designed to be lightweight, production-ready, and useful for AI workflows like semantic search, retrieval-augmented generation (RAG), recommendation systems, and other similarity-based tasks — all within the same process as your application.</p><p>At its core, Zvec runs as a simple library use just like you would a relational embedded database such as SQLite—but for vectors instead of tables. Because it runs “in-process,” there is no network layer, no external daemon to manage, and minimal configuration required. This zero-ops design makes it especially well-suited for local development, edge devices, desktop tools, command-line utilities, and other environments where deploying a separate vector database service would be impractical.</p><h4>How Zvec Works</h4><p>Zvec is built on Proxima, Alibaba Group’s high-performance vector search engine that has been battle-tested in large-scale production environments. It exposes a simple API that lets developers define collections, insert documents with vectors, and run similarity queries — all in a few lines of code.</p><p>Zvec supports:</p><ul><li>Dense and sparse vector types</li><li>Multi-vector queries</li><li>Hybrid search with scalar filters</li><li>Fast approximate nearest neighbor (ANN) search</li><li>Hybrid search optimizations</li><li>Resource control for CPU and memory, making it stable in constrained environments like mobile or edge devices</li></ul><p>Because Zvec runs inside your application process, it can seamlessly integrate into Python projects, notebooks, CLI tools, or edge applications without additional infrastructure.</p><h4>Why Zvec Matters</h4><p>RAG and semantic search systems require more than just a similarity index — they need vector storage, metadata (scalar fields), full CRUD operations, and reliable persistence as local knowledge bases constantly evolve.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DJTUrnXyqWiolcTP.png" /><figcaption><a href="https://zvec.org/en/blog/introduction/">Zvec: The SQLite of Vector Databases | Zvec</a></figcaption></figure><p>Libraries like Faiss offer fast nearest neighbor search but lack built-in storage, crash recovery, and hybrid query capabilities, forcing developers to build additional infrastructure. Extensions such as DuckDB with VSS support add vector search but provide limited indexing flexibility and weaker resource control for edge environments. Service-based platforms like Milvus or managed vector cloud solutions require separate deployment and network communication, which can be excessive for on-device applications.</p><p>Zvec is designed specifically for these local use cases, offering a vector-native engine with built-in persistence, resource governance, and RAG-focused features — all packaged as a lightweight embedded library.</p><p>Zvec’s embedded approach enables:</p><ul><li>Local sematic search and RAG without external services</li><li>Fast prototyping and development</li><li>Edge and IoT usage with resource governance</li><li>Simplified integration comparable to SQLite’s impact on relational storage</li></ul><p>This positions Zvec as a key infrastructure component for applications that need high-quality vector retrieval but require a lightweight, zero-ops operational model.</p><h4>Performance Benchmarks</h4><p>Zvec is designed for CPU-bound, high-throughput similarity search workloads. The engine leverages multithreading, cache-efficient memory layouts, SIMD optimizations, and prefetching techniques to maximize performance on modern processors.</p><p>According to reported results from VectorDBBench (Cohere 10M dataset), Zvec achieves over 8,000 queries per second (QPS) while maintaining matched recall. In the same benchmark configuration, it reportedly delivered more than twice the throughput of the previous top-performing system, while also reducing index build time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mq6m44h-Tfc-wjol.png" /><figcaption><a href="https://zvec.org/en/blog/introduction/">Zvec: The SQLite of Vector Databases | Zvec</a></figcaption></figure><h4>RAG-Focused Features</h4><p>Zvec is specifically optimized for retrieval-augmented generation (RAG) and AI agent workflows. It provides capabilities that go beyond basic vector indexing, making it suitable for dynamic, production-ready knowledge systems.</p><ul><li><strong>Full CRUD support</strong> for managing mutable local knowledge bases.</li><li><strong>Schema evolution</strong>, allowing developers to modify indexing strategies and fields as application requirements evolve.</li><li><strong>Multi-vector retrieval</strong>, enabling the combination of multiple embedding channels for richer semantic matching.</li><li><strong>Built-in reranking mechanisms</strong>, including weighted fusion and Reciprocal Rank Fusion (RRF), to improve retrieval relevance.</li><li><strong>Scalar-vector hybrid search</strong>, with scalar filters pushed down into the index path for efficient execution, along with optional inverted indexes for attribute-based filtering.</li></ul><p>These capabilities make Zvec well-suited for applications that require flexible indexing, evolving schemas, and high-quality retrieval — all within an embedded, zero-ops deployment model.</p><h3>Hands-On 1: Creating Your First Embedded Vector Database with Zvec</h3><p>In this section, you will create your first Zvec collection, insert vector documents, and perform a similarity search — all within a few lines of Python code.</p><p>For this hands-on guide, we will be using Google Colab with a T4 GPU.</p><h4>Setting up the environment</h4><p><strong>Step 1: Create and Configure Your Colab Notebook</strong></p><ol><li>Open <a href="https://colab.research.google.com/">Google Colaboratory</a> and sign in with your Google account.</li><li>Create a new notebook by clicking on + New Notebook.</li><li>Navigate to Runtime → Change runtime type.</li></ol><ul><li>Set Hardware Accelerator to GPU.</li><li>Choose T4 GPU (recommended for this tutorial). It is recommended to use a GPU, as the CPU runtime may crash while creating the Zvec collection due to resource constraints.</li><li>Click Save.</li></ul><p><strong>Step 2: Add Hugging Face Access Token (Optional but Recommended)</strong></p><p>If you’re pulling models from Hugging Face, you’ll need an access token:</p><ol><li>In the left sidebar, select the 🔑 Secrets tab.</li><li>Add a new secret:</li></ol><ul><li>Key: HF_TOKEN</li><li>Value: <em>Your Hugging Face access token</em></li></ul><p>3. Generate a write token from your <a href="https://huggingface.co/settings/tokens">Hugging Face profile settings</a> → Create new token → Select write → Provide token name → Click on Create token.</p><p>Press enter or click to view image in full size</p><p><strong>Step 3: Install Dependencies</strong></p><p>Use the following script to install all necessary packages:</p><pre>!pip install zvec</pre><h4>Creating and querying a Zvec collection</h4><pre>import zvec<br> <br># Define collection schema<br>schema = zvec.CollectionSchema(<br>    name=&quot;example&quot;,<br>    vectors=zvec.VectorSchema(&quot;embedding&quot;, zvec.DataType.VECTOR_FP32, 4),<br>)<br> <br># Create collection<br>collection = zvec.create_and_open(path=&quot;./zvec_example&quot;, schema=schema,)<br> <br># Insert documents<br>collection.insert([<br>    zvec.Doc(id=&quot;doc_1&quot;, vectors={&quot;embedding&quot;: [0.1, 0.2, 0.3, 0.4]}),<br>    zvec.Doc(id=&quot;doc_2&quot;, vectors={&quot;embedding&quot;: [0.2, 0.3, 0.4, 0.1]}),<br>])<br> <br># Search by vector similarity<br>results = collection.query(<br>    zvec.VectorQuery(&quot;embedding&quot;, vector=[0.4, 0.3, 0.3, 0.1]),<br>    topk=10<br>)<br> <br># Results: list of {&#39;id&#39;: str, &#39;score&#39;: float, ...}, sorted by relevance <br>print(results)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/680/1*Qi3BE4TjsXmFgeLBicZK_w.png" /></figure><ul><li><strong>Define a </strong><strong>CollectionSchema</strong>: Specify vector fields (and optional scalar fields) that describe how your data will be stored.</li><li><strong>Create or open a collection</strong>: Use create_and_open() to initialize a new collection or load an existing one from disk.</li><li><strong>Insert documents</strong>: Add Doc objects containing unique IDs, embedding vectors, and optional metadata attributes.</li><li><strong>Build indexes and query</strong>: Run VectorQuery operations to retrieve the nearest neighbors based on vector similarity.</li><li><strong>Consume results</strong>: Results are returned as dictionaries containing document IDs and similarity scores, sorted by relevance. These can directly power a local semantic search engine or serve as the retrieval layer for a RAG pipeline.</li></ul><p>With this simple workflow, you now have a fully functional embedded vector database running inside your application — ready to scale from toy examples to production-ready local AI systems.</p><h3>Hands-On 2: Building a <strong>Zvec-based FAQ Agent for customer queries</strong></h3><p>This section provides a comprehensive, step-by-step guide to building a Zvec-based FAQ agent capable of handling large volumes of customer queries — similar to the support systems used by major e-commerce platforms.</p><h4>Installing dependencies</h4><p>In this hands-on tutorial, we will use Sentence Transformers to generate high-quality semantic embeddings and OpenAI’s language models to generate natural, context-aware responses for users.</p><pre>!pip install zvec sentence-transformers openai</pre><h4>Preparing sample FAQ data</h4><p>Create a structured FAQ dataset that will serve as the knowledge base for your Zvec-powered retrieval system, containing common customer questions and their corresponding answers for semantic search.</p><pre>faq_data = [<br>    {<br>        &quot;id&quot;: &quot;faq_1&quot;,<br>        &quot;question&quot;: &quot;How do I return a product?&quot;,<br>        &quot;answer&quot;: &quot;You can return a product within 7 days by going to Orders → Select item → Click Return.&quot;<br>    },<br>    {<br>        &quot;id&quot;: &quot;faq_2&quot;,<br>        &quot;question&quot;: &quot;How long does delivery take?&quot;,<br>        &quot;answer&quot;: &quot;Standard delivery takes 3-5 business days.&quot;<br>    },<br>    {<br>        &quot;id&quot;: &quot;faq_3&quot;,<br>        &quot;question&quot;: &quot;How do I track my order?&quot;,<br>        &quot;answer&quot;: &quot;Go to My Orders and click on Track to see real-time tracking updates.&quot;<br>    },<br>    {<br>        &quot;id&quot;: &quot;faq_4&quot;,<br>        &quot;question&quot;: &quot;How do I cancel my order?&quot;,<br>        &quot;answer&quot;: &quot;Orders can be cancelled before they are shipped from the Orders page.&quot;<br>    }<br>]</pre><h4>Creating Zvec collection</h4><p>Initialize and configure a persistent Zvec collection by defining the schema and preparing it to store and index embedded FAQ documents.</p><pre>import zvec<br>from sentence_transformers import SentenceTransformer<br>import shutil<br>import os<br>import time # Import time module<br><br># Load embedding model<br>model = SentenceTransformer(&quot;all-MiniLM-L6-v2&quot;)<br><br># Define schema<br>schema = zvec.CollectionSchema(<br>    name=&quot;faq_collection&quot;,<br>    vectors=zvec.VectorSchema(<br>        &quot;embedding&quot;,<br>        zvec.DataType.VECTOR_FP32,<br>        384   # dimension of MiniLM model<br>    )<br>    # fields=[<br>    #     zvec.FieldSchema(&quot;question&quot;, zvec.DataType.STRING),<br>    #     zvec.FieldSchema(&quot;answer&quot;, zvec.DataType.STRING)<br>    # ]<br>)<br><br># Remove existing collection if it exists<br>collection_path = &quot;./faq_zvec&quot; # Changed path to ensure a fresh start<br>if os.path.exists(collection_path):<br>    shutil.rmtree(collection_path)<br>    time.sleep(1) # Add a small delay<br><br># Create or open collection<br>collection = zvec.create_and_open(<br>    path=collection_path,<br>    schema=schema,<br>)<br><br>docs = []<br><br>for item in faq_data:<br>    text = item[&quot;question&quot;] + &quot; &quot; + item[&quot;answer&quot;]<br>    embedding = model.encode(text).tolist()<br><br>    docs.append(<br>        zvec.Doc(<br>            id=item[&quot;id&quot;],<br>            vectors={&quot;embedding&quot;: embedding}<br>            # fields={&quot;question&quot;: item[&quot;question&quot;], &quot;answer&quot;: item[&quot;answer&quot;]}<br>        )<br>    )<br><br>collection.insert(docs)<br><br>print(&quot;FAQ data indexed successfully!&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Pn_ynusDNHw1-zcUHxQ2sQ.png" /></figure><h4>Building retrieval function</h4><p>Implement the retrieval layer that converts user queries into embeddings and fetches the most relevant FAQ entries from Zvec using vector similarity search.</p><pre>def retrieve(query, topk=3):<br>    embedding = model.encode(query).tolist()<br>    results = collection.query(<br>        zvec.VectorQuery(&quot;embedding&quot;, vector=embedding),<br>        topk=topk<br>    )<br>    return results</pre><h4>Connecting to LLM for answer generation</h4><p>Integrate the retrieval layer with an LLM to generate clear, natural responses by grounding answers in the relevant FAQs retrieved from Zvec.</p><pre># from retriever import retrieve<br>from openai import OpenAI<br><br>client = OpenAI(api_key=&quot;your-openai-key&quot;)  # set OPENAI_API_KEY<br><br># Build a lookup dict from faq_data to retrieve answers by ID<br>faq_lookup = {item[&quot;id&quot;]: item for item in faq_data}<br><br>def answer_query(user_query):<br>    retrieved_docs = retrieve(user_query, topk=3)<br><br>    # Look up answers by doc ID (since fields aren&#39;t stored in zvec)<br>    context = &quot;\n&quot;.join(<br>        [faq_lookup[doc.id][&quot;answer&quot;] for doc in retrieved_docs]<br>    )<br><br>    prompt = f&quot;&quot;&quot;<br>You are a customer support agent.<br><br>Use only the information below to answer.<br><br>Context:<br>{context}<br><br>Customer Question:<br>{user_query}<br><br>Answer:<br>&quot;&quot;&quot;<br><br>    response = client.chat.completions.create(<br>        model=&quot;gpt-4o-mini&quot;,<br>        messages=[{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: prompt}]<br>    )<br><br>    return response.choices[0].message.content</pre><h4>Testing FAQ agent</h4><p>Run and interact with the FAQ agent to validate the end-to-end flow, from query embedding and retrieval in Zvec to natural response generation by the LLM.</p><pre>test_questions = [<br>    &quot;I received the wrong item, how do I return it?&quot;,<br>    &quot;When will my order arrive?&quot;,<br>    &quot;How do I cancel my order&quot;,<br>]<br><br>for question in test_questions:<br>    print(f&quot;\n{&#39;=&#39;*60}&quot;)<br>    print(f&quot;❓ Question: {question}&quot;)<br>    print(f&quot;{&#39;=&#39;*60}&quot;)<br>    answer = answer_query(question)<br>    print(f&quot;\n🤖 Answer:\n{answer}&quot;)</pre><p>Execute the script to observe the output as below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EoDiBH7KLwm5ZepIUBollA.png" /></figure><p>If you are interested in more hands-on examples, check out how Zvec can be used to build a car manual RAG agent with OpenAI for efficient and accurate data retrieval.</p><p><a href="https://colab.research.google.com/drive/1WM1eamaVeqcmx-jQAhTEBIeag5nOuy9Y?usp=sharing">Google Colab</a></p><p><em>Thanks for reading this article !!</em></p><p>If you enjoyed this article, please click on the clap button 👏 and share to help others find it!</p><p>The full source code for this tutorial can be found here,</p><p><a href="https://github.com/codemaker2015/zvec-experiments">GitHub - codemaker2015/zvec-experiments: (Retrieval-Augmented Generation) using PDF data and simple text data.</a></p><h4>Resources</h4><ul><li><a href="https://colab.research.google.com/drive/1qoOzIQaaoI7sndOf5QNZGoWdAvIRPu1T?usp=sharing">Google Colab</a></li><li><a href="https://zvec.org/en/blog/introduction/">Zvec: The SQLite of Vector Databases</a></li><li><a href="https://github.com/alibaba/zvec">GitHub - alibaba/zvec: A lightweight, lightning-fast, in-process vector database</a></li><li><a href="https://pypi.org/project/zvec/">zvec</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e76b247b6555" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[OmniDaemon: The Universal Event-Driven Runtime for Production Ready AI Agents]]></title>
            <link>https://codemaker2016.medium.com/omnidaemon-the-universal-event-driven-runtime-for-production-ready-ai-agents-02b1a5e63dfb?source=rss-b7a60e472dfc------2</link>
            <guid isPermaLink="false">https://medium.com/p/02b1a5e63dfb</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[omnistack]]></category>
            <category><![CDATA[agents]]></category>
            <dc:creator><![CDATA[Vishnu Sivan]]></dc:creator>
            <pubDate>Tue, 30 Dec 2025 16:07:38 GMT</pubDate>
            <atom:updated>2025-12-30T16:07:38.081Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Moving beyond the “Chatbot” era to autonomous, scalable, and resilient AI infrastructure.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zYd6mvBQMYZtt8R2A-UQLg.png" /></figure><p>The AI landscape is shifting. We have spent the last two years perfecting the “Chatbot” — a synchronous, request-response interface where an LLM waits for a user to type. But the future of AI isn’t a text box; it’s an autonomous service that listens to events, reasons, and acts in the background.</p><p>Enter OmniDaemon, a universal event-driven runtime built by <a href="https://github.com/omnirexflora-labs">OmniRexFlora Labs</a>. It is designed to turn AI agents into robust, production-ready infrastructure.</p><h3>Getting Started</h3><h4>Table of contents</h4><ul><li>The “Monolithic Trap” in Agentic AI</li><li>What is OmniDaemon?</li><li>Key Pillars</li><li>The Architecture: How it Works</li><li>The “Omni Stack” Ecosystem</li><li>The Core Problem OmniDaemon Solves</li><li>Why Event-Driven Architecture for AI Agents?</li><li>Why Traditional Architectures Fail for Agents</li><li>Getting Started: A Technical Quickstart</li><li>Step 1: Set Up the Environment</li><li>Step 2: Create your first agent</li><li>Step 3: Create the Event Producer (The Trigger)</li><li>Hands-On Project: Intelligent Log Insights Generator</li><li>Step 1: Set Up the Environment</li><li>Step 2: Create Sample Log Generator</li><li>Step 3: Build the Log Ingestion Agent</li><li>Step 4: Build the LLM-Powered Analysis Agent</li><li>Step 5: Build the Reporting Agent</li><li>Step 6: (Optional) Advanced Agent Chaining for Specialized Analysis</li><li>Advanced Features and Patterns</li></ul><h4>The “Monolithic Trap” in Agentic AI</h4><p>Most developers start by building multi-agent systems as single, monolithic applications. While this works for a local demo, it fails in production for three reasons:</p><ol><li><strong>The Blast Radius:</strong> If your “Data Analyst” agent crashes on a corrupted CSV, your entire system — Research Agent, Writer Agent, and API — goes down with it.</li><li><strong>The Scaling Wall:</strong> You cannot scale individual agents. If your “Research Agent” is I/O intensive, you’re forced to scale the whole monolith, wasting resources.</li><li><strong>Synchronous Bottlenecks:</strong> AI is slow. Making a user wait for a 30-second chain of LLM calls over an HTTP request leads to timeouts and poor UX.</li></ol><h3>What is OmniDaemon?</h3><p>OmniDaemon is a Universal Event-Driven Runtime for AI agents. It acts as the “Central Nervous System” for your AI stack. Think of it as “Kubernetes for AI Agents” — it provides the orchestration, scalability, and reliability needed to run AI agents in production. It allows agents to operate asynchronously, reacting to events (like a new file upload, a CRM update, or a message from another agent) rather than waiting for direct API calls.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/736/1*IKw5Qg0tBzo4OgKXfonlZA.png" /></figure><h4>Key Pillars:</h4><ul><li><strong>Framework Agnostic:</strong> It doesn’t care if your agent is built with OpenAI’s SDK, LangGraph, or CrewAI. If it’s Python code, OmniDaemon can daemonize it.</li><li><strong>Production Ready:</strong> It handles the “boring but hard” stuff: retries, Dead Letter Queues (DLQ), message persistence, and horizontal scaling.</li><li><strong>Event-Driven (EDA):</strong> Built on high-performance backends like Redis Streams, it ensures that agents collaborate across distributed systems without being tightly coupled.</li><li><strong>Asynchronous Execution:</strong> They listen to a “Topic” (like a Redis stream). When a task arrives, the daemon triggers a callback, processes it, and emits the result.</li><li><strong>Pluggable Backends</strong>: Swap event buses (Redis Streams, Kafka, RabbitMQ) and storage (Redis, PostgreSQL, MongoDB) via environment variables</li><li><strong>Horizontal Scaling</strong>: Deploy multiple agent instances for load balancing without code changes</li></ul><h4>The Architecture: How it Works</h4><p>OmniDaemon sits between your event source (like Redis or a JSON stream) and your AI logic.</p><ul><li><strong>The Listener:</strong> OmniDaemon monitors an event stream.</li><li><strong>The Callback:</strong> When a message arrives, it triggers your agent logic via a simple callback function.</li><li><strong>The Manager:</strong> It manages the lifecycle. If your agent fails, OmniDaemon handles the retry logic. If it succeeds, it persists the result and can even trigger the next agent in the chain.</li><li><strong>Storage Backends:</strong> It currently supports JSON (for local development) and Redis (for production). S3 and PostgreSQL support are on the roadmap.</li><li><strong>The Callback Pattern:</strong> You wrap your agent logic in a single function. OmniDaemon handles the “plumbing” — fetching the message, tracking metadata, and persisting the output.</li><li><strong>Metadata &amp; Context:</strong> Every event carries a rich metadata object including message_id, correlation_id (to track a task across 10 different agents), and tenant_id for multi-tenant SaaS apps.</li></ul><h4>The “Omni Stack” Ecosystem</h4><p>OmniDaemon is part of a larger vision by OmniRexFlora Labs to provide a complete “Linux-like” ecosystem for AI:</p><ul><li><strong>OmniCoreAgent:</strong> The framework for building the “brains.”</li><li><strong>OmniMemory:</strong> Persistent, semantic memory for agents.</li><li><strong>OmniDaemon:</strong> The runtime for distributed execution.</li><li><strong>OmniCloud:</strong> (Upcoming) The deployment and orchestration layer.</li></ul><h4>The Core Problem OmniDaemon Solves</h4><p>Traditional AI systems are request-driven: a user asks a question, the AI responds, and that’s it. But modern enterprises need AI that operates continuously in the background, reacting to events, coordinating with other agents, and integrating seamlessly with existing infrastructure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/812/1*NFLMtRLeku92_-l27Iyjag.png" /></figure><h3>Why Event-Driven Architecture for AI Agents?</h3><p>The future of AI lies in autonomous agents that operate as distributed microservices. Just like microservices revolutionized application architecture, event-driven AI agents are transforming how we build intelligent systems.</p><p><strong>Wave 1: Predictive Models (Traditional ML)</strong></p><ul><li>Domain-specific, rigid systems</li><li>Required ML expertise for each use case</li><li>Difficult to repurpose or scale</li></ul><p><strong>Wave 2: Generative Models (LLMs)</strong></p><ul><li>Revolutionary generalization capabilities</li><li>But: fixed in time, expensive to fine-tune, no access to private data</li><li>RAG (Retrieval-Augmented Generation) helped, but workflows remained rigid</li></ul><p><strong>Wave 3: Agentic AI (Current)</strong></p><ul><li>Dynamic workflows that adapt on the fly</li><li>Autonomous decision-making with tool use</li><li>Context-driven processing with memory</li><li>Collaborative multi-agent systems</li></ul><blockquote>Industry leaders agree: “Agents are the new apps” (Dharmesh Shah, HubSpot CTO). But building scalable agent systems requires proper infrastructure.</blockquote><h4>Why Traditional Architectures Fail for Agents</h4><p>Connecting agents via REST APIs creates tightly coupled systems:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/811/1*miE-4wkFpbGGuCKuRz4y8g.png" /></figure><p>Event-driven architecture solves this through loose coupling:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/810/1*53g46U2rZgUuWPGPeK6Qrg.png" /></figure><h3>Getting Started: A Technical Quickstart</h3><p>OmniDaemon is designed for simplicity. You can move an existing agent into a daemonized state in minutes.</p><h4>Prerequisites</h4><ul><li><a href="https://www.python.org/downloads"><strong>Python 3.9+</strong></a></li><li><strong>Redis</strong> (Installed and running)</li><li><strong>OpenAI API Key</strong> (or any LLM provider)</li></ul><h4>Step 1: Set Up the Environment</h4><p>To use OmniDaemon effectively in a production or development environment, running Redis via <a href="https://www.docker.com/products/docker-desktop"><strong>Docker</strong></a> is the industry standard. This ensures your event stream is isolated, portable, and easy to manage.</p><h4>Initializing Redis</h4><p>Before running Redis, you must have the Docker Engine installed.</p><p><strong>For Windows &amp; Mac:<br>Download Docker Desktop</strong> from the <a href="https://www.docker.com/products/docker-desktop/">official website</a>.</p><p><strong>For Linux (Ubuntu):<br></strong>Open your terminal and run these commands to install the Docker Engine:</p><pre>sudo apt update<br>sudo apt install ca-certificates curl gnupg<br>sudo install -m 0755 -d /etc/apt/keyrings<br>curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg<br>sudo chmod a+r /etc/apt/keyrings/docker.gpg<br><br>echo &quot;deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(. /etc/os-release &amp;&amp; echo &quot;$VERSION_CODENAME&quot;) stable&quot; | sudo tee /etc/apt/sources.list.d/docker.list &gt; /dev/null<br><br>sudo apt update<br>sudo apt install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin</pre><p>Once Docker is ready, you can pull and start Redis with a single command. For OmniDaemon, we want persistence so that your AI tasks aren’t lost if the container restarts.</p><pre>docker run -d --name omni-redis -p 6379:6379 -v redis-data:/data redis:latest redis-server --appendonly yes</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MD_HGrzUrmo55mh-OawQng.png" /></figure><h4>Installing Dependencies</h4><p>First, install the necessary libraries using pip.</p><pre>pip install omnidaemon</pre><h4>Step 2: Create your first agent</h4><p>Below is the implementation of a distributed “Greeter” agent. The agent_runner.py acts as our persistent service, while producer.py demonstrates how external systems interact with the agent through structured message passing.</p><p>Create file named agent_runner.py and add the following code it.</p><pre>import asyncio<br>from omnidaemon import OmniDaemonSDK<br>from omnidaemon import AgentConfig<br><br>sdk = OmniDaemonSDK()<br><br>async def greeter(message: dict):<br>    &quot;&quot;&quot;Your AI agent runs here!&quot;&quot;&quot;<br>    name = message.get(&quot;content&quot;, {}).get(&quot;name&quot;, &quot;stranger&quot;)<br>    return {&quot;reply&quot;: f&quot;Hello, {name}! 👋&quot;}<br><br>async def main():<br>    await sdk.register_agent(<br>        agent_config=AgentConfig(<br>            topic=&quot;greet.user&quot;,<br>            callback=greeter,<br>        )<br>    )<br>    await sdk.start()<br>    print(&quot;🎧 Agent running. Press Ctrl+C to stop.&quot;)<br>    <br>    try:<br>        while True:<br>            await asyncio.sleep(1)<br>    except KeyboardInterrupt:<br>        pass<br>    finally:<br>        await sdk.shutdown()<br><br>if __name__ == &quot;__main__&quot;:<br>    asyncio.run(main())</pre><h4>Step 3: Create the Event Producer (The Trigger)</h4><p>In an event-driven system, the producer doesn’t wait for the AI to finish. It just “drops” the job into the queue and moves on.</p><p>Create file named producer.py and add the following code it.</p><pre>import asyncio<br>import json<br><br>async def test_messages():<br>    &quot;&quot;&quot;Send test messages to the agent.&quot;&quot;&quot;<br>    <br>    # Test cases<br>    tests = [<br>        (&quot;greet.user&quot;, {&quot;content&quot;: {&quot;name&quot;: &quot;World&quot;}}),<br>        (&quot;greet.user&quot;, {&quot;content&quot;: {&quot;name&quot;: &quot;Alice&quot;}}),<br>        (&quot;greet.user&quot;, {&quot;content&quot;: {}}),<br>        (&quot;health.check&quot;, {&quot;timestamp&quot;: &quot;now&quot;}),<br>    ]<br>    <br>    print(&quot;Testing agent messages...\n&quot;)<br>    <br>    for topic, message in tests:<br>        print(f&quot;Topic: {topic}&quot;)<br>        print(f&quot;Message: {json.dumps(message)}&quot;)<br>        <br>        # Simulate sending<br>        await asyncio.sleep(0.3)<br>        <br>        # Simulate response<br>        if topic == &quot;greet.user&quot;:<br>            name = message.get(&quot;content&quot;, {}).get(&quot;name&quot;, &quot;stranger&quot;)<br>            print(f&quot;Response: Hello, {name}!\n&quot;)<br>        else:<br>            print(&quot;Response: {&#39;status&#39;: &#39;healthy&#39;}\n&quot;)<br>        <br>        await asyncio.sleep(0.5)<br>    <br>    print(&quot;Done!&quot;)<br><br>if __name__ == &quot;__main__&quot;:<br>    asyncio.run(test_messages())</pre><h4>Executing the code</h4><p>Open your first terminal and execute the agent runner. This process will initialize the OmniDaemonSDK, register the greeter callback to the greet.user topic, and enter an idle listening state.</p><pre>python agent_runner.py</pre><p><em>You should see the output:</em> 🎧 Agent running. Press Ctrl+C to stop.</p><p>Open a second terminal to run the publisher script. This script acts as the “Client” or “Producer,” injecting JSON-formatted payloads into the system to be picked up by the waiting agent.</p><pre>python producer.py</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UtFN-rFhkH-boZDqHsoTMA.png" /></figure><h3>Hands-On Project: Intelligent Log Insights Generator</h3><p>Now let’s build a real-world application: an AI-powered log analysis system that monitors authentication and database transaction logs, detects anomalies, identifies vulnerabilities, and provides actionable recommendations.</p><h4>Project Architecture</h4><p>Our system will have three main components:</p><ol><li><strong>Log Ingestion Agent</strong>: Watches log files and publishes events</li><li><strong>Analysis Agent</strong>: Uses LLM to analyze logs for security issues, anomalies, and performance concerns</li><li><strong>Reporting Agent</strong>: Aggregates insights and generates actionable reports</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/807/1*oLgTZ6TierTGpRA44Iheug.png" /></figure><h4>Step 1: Set Up the Environment</h4><p>First, install the necessary libraries. We will use omnidaemon for the runtime and langchain (optional) to demonstrate how it wraps external frameworks.</p><pre>pip install omnidaemon openai python-dotenv</pre><p>Ensure your .env file has your credentials:</p><pre>OPENAI_API_KEY=your_key_here<br>STORAGE_BACKEND=redis<br>REDIS_URL=redis://localhost:6379<br>EVENT_BUS_TYPE=redis_stream<br>OMNIDAEMON_API_ENABLED=true<br>OMNIDAEMON_API_PORT=8765<br>LOG_LEVEL=INFO</pre><h4>Step 2: Create Sample Log Generator</h4><p>First, let’s create realistic sample logs for testing:</p><pre># generate_logs.py<br>import json<br>import random<br>import time<br>from datetime import datetime, timedelta<br>from pathlib import Path<br><br># Create logs directory<br>log_dir = Path(&quot;./sample_logs&quot;)<br>log_dir.mkdir(exist_ok=True)<br><br># Sample data for realistic logs<br>users = [&quot;alice&quot;, &quot;bob&quot;, &quot;charlie&quot;, &quot;david&quot;, &quot;eve&quot;, &quot;mallory&quot;]<br>ips = [&quot;192.168.1.10&quot;, &quot;192.168.1.20&quot;, &quot;192.168.1.30&quot;, &quot;10.0.0.5&quot;, &quot;172.16.0.100&quot;]<br>suspicious_ips = [&quot;45.133.1.12&quot;, &quot;89.248.172.45&quot;]  # Potential attack IPs<br>db_operations = [&quot;SELECT&quot;, &quot;INSERT&quot;, &quot;UPDATE&quot;, &quot;DELETE&quot;]<br>tables = [&quot;users&quot;, &quot;transactions&quot;, &quot;orders&quot;, &quot;products&quot;, &quot;audit_logs&quot;]<br><br>def generate_auth_log():<br>    &quot;&quot;&quot;Generate authentication log entry&quot;&quot;&quot;<br>    is_suspicious = random.random() &lt; 0.1  # 10% suspicious activity<br>    <br>    if is_suspicious:<br>        # Generate suspicious patterns<br>        event_type = random.choice([&quot;failed_login&quot;, &quot;brute_force_attempt&quot;, &quot;impossible_travel&quot;])<br>        ip = random.choice(suspicious_ips)<br>        success = False<br>    else:<br>        event_type = random.choice([&quot;login&quot;, &quot;logout&quot;, &quot;password_change&quot;, &quot;session_refresh&quot;])<br>        ip = random.choice(ips)<br>        success = random.random() &lt; 0.95  # 95% success rate normally<br>    <br>    return {<br>        &quot;timestamp&quot;: datetime.now().isoformat(),<br>        &quot;log_type&quot;: &quot;authentication&quot;,<br>        &quot;event&quot;: event_type,<br>        &quot;user&quot;: random.choice(users),<br>        &quot;ip_address&quot;: ip,<br>        &quot;success&quot;: success,<br>        &quot;user_agent&quot;: &quot;Mozilla/5.0&quot; if random.random() &lt; 0.8 else &quot;curl/7.68.0&quot;,<br>        &quot;session_id&quot;: f&quot;sess_{random.randint(10000, 99999)}&quot;<br>    }<br><br>def generate_db_log():<br>    &quot;&quot;&quot;Generate database transaction log entry&quot;&quot;&quot;<br>    is_anomaly = random.random() &lt; 0.15  # 15% anomalies<br>    <br>    operation = random.choice(db_operations)<br>    execution_time = random.uniform(0.01, 0.5)<br>    <br>    if is_anomaly:<br>        # Generate anomalous patterns<br>        if random.random() &lt; 0.5:<br>            execution_time = random.uniform(5.0, 30.0)  # Slow query<br>        <br>        if operation == &quot;DELETE&quot; and random.random() &lt; 0.3:<br>            rows_affected = random.randint(1000, 10000)  # Mass deletion<br>        else:<br>            rows_affected = random.randint(1, 100)<br>    else:<br>        rows_affected = random.randint(1, 10)<br>    <br>    return {<br>        &quot;timestamp&quot;: datetime.now().isoformat(),<br>        &quot;log_type&quot;: &quot;database&quot;,<br>        &quot;operation&quot;: operation,<br>        &quot;table&quot;: random.choice(tables),<br>        &quot;user&quot;: random.choice(users),<br>        &quot;execution_time_ms&quot;: round(execution_time * 1000, 2),<br>        &quot;rows_affected&quot;: rows_affected,<br>        &quot;query_hash&quot;: f&quot;qh_{random.randint(100000, 999999)}&quot;,<br>        &quot;connection_id&quot;: random.randint(1, 50)<br>    }<br><br>def generate_logs(num_logs=100, filename=&quot;application.log&quot;):<br>    &quot;&quot;&quot;Generate mixed authentication and database logs&quot;&quot;&quot;<br>    log_file = log_dir / filename<br>    <br>    with open(log_file, &quot;w&quot;) as f:<br>        for _ in range(num_logs):<br>            # Mix of auth and db logs (60% auth, 40% db)<br>            if random.random() &lt; 0.6:<br>                log_entry = generate_auth_log()<br>            else:<br>                log_entry = generate_db_log()<br>            <br>            f.write(json.dumps(log_entry) + &quot;\n&quot;)<br>            time.sleep(0.01)  # Simulate real-time generation<br>    <br>    print(f&quot;Generated {num_logs} log entries in {log_file}&quot;)<br><br>if __name__ == &quot;__main__&quot;:<br>    # Generate initial logs<br>    generate_logs(num_logs=100, filename=&quot;application.log&quot;)<br>    <br>    # Optionally: Continuous generation<br>    print(&quot;\nGenerating continuous logs... (Ctrl+C to stop)&quot;)<br>    try:<br>        while True:<br>            generate_logs(num_logs=10, filename=&quot;application.log&quot;)<br>            time.sleep(5)  # Generate new batch every 5 seconds<br>    except KeyboardInterrupt:<br>        print(&quot;\nLog generation stopped.&quot;)</pre><h4>Step 3: Build the Log Ingestion Agent</h4><p>The log ingestion agent is the entry point of our system. It acts as a bridge between your log files and the OmniDaemon event-driven architecture. Instead of having your analysis agents directly read files (which creates tight coupling and scaling issues), the ingestion agent watches for new log entries and publishes them as events to the event bus.</p><pre># log_ingestion_agent.py<br>import asyncio<br>import json<br>import time<br>from pathlib import Path<br>from omnidaemon import OmniDaemonSDK, EventEnvelope, PayloadBase<br><br>sdk = OmniDaemonSDK()<br><br>class LogIngestionAgent:<br>    def __init__(self, log_dir=&quot;./sample_logs&quot;):<br>        self.log_dir = Path(log_dir)<br>        self.processed_lines = {}  # Track processed lines per file<br>        <br>    async def watch_logs(self):<br>        &quot;&quot;&quot;Watch log files and publish new entries&quot;&quot;&quot;<br>        print(f&quot;🔍 Watching logs in {self.log_dir}&quot;)<br>        <br>        while True:<br>            try:<br>                # Find all log files<br>                log_files = list(self.log_dir.glob(&quot;*.log&quot;))<br>                <br>                for log_file in log_files:<br>                    await self.process_log_file(log_file)<br>                <br>                await asyncio.sleep(2)  # Check every 2 seconds<br>                <br>            except Exception as e:<br>                print(f&quot;❌ Error watching logs: {e}&quot;)<br>                await asyncio.sleep(5)<br>    <br>    async def process_log_file(self, log_file: Path):<br>        &quot;&quot;&quot;Process new lines from a log file&quot;&quot;&quot;<br>        try:<br>            with open(log_file, &quot;r&quot;) as f:<br>                lines = f.readlines()<br>            <br>            # Get last processed line number<br>            last_line = self.processed_lines.get(str(log_file), 0)<br>            <br>            # Process only new lines<br>            new_lines = lines[last_line:]<br>            <br>            for line in new_lines:<br>                if line.strip():<br>                    await self.publish_log_entry(line, str(log_file))<br>            <br>            # Update processed line count<br>            self.processed_lines[str(log_file)] = len(lines)<br>            <br>        except Exception as e:<br>            print(f&quot;❌ Error processing {log_file}: {e}&quot;)<br>    <br>    async def publish_log_entry(self, log_line: str, source_file: str):<br>        &quot;&quot;&quot;Publish a log entry to the event bus&quot;&quot;&quot;<br>        try:<br>            log_entry = json.loads(log_line)<br>            <br>            # Create event envelope<br>            event = EventEnvelope(<br>                topic=&quot;logs.raw&quot;,<br>                payload=PayloadBase(<br>                    content={<br>                        &quot;log_entry&quot;: log_entry,<br>                        &quot;source_file&quot;: source_file<br>                    },<br>                    reply_to=&quot;logs.analyzed&quot;  # Results go to analyzed topic<br>                ),<br>                source=&quot;log_ingestion_agent&quot;<br>            )<br>            <br>            # Publish to event bus<br>            task_id = await sdk.publish_task(event_envelope=event)<br>            print(f&quot;📤 Published log: {log_entry.get(&#39;log_type&#39;)} - {task_id}&quot;)<br>            <br>        except json.JSONDecodeError:<br>            print(f&quot;⚠️  Invalid JSON in log line&quot;)<br>        except Exception as e:<br>            print(f&quot;❌ Error publishing log: {e}&quot;)<br><br>async def main():<br>    try:<br>        # Create ingestion agent<br>        ingestion_agent = LogIngestionAgent()<br>        <br>        # Start watching logs<br>        print(&quot;🚀 Log Ingestion Agent started&quot;)<br>        await ingestion_agent.watch_logs()<br>        <br>    except KeyboardInterrupt:<br>        print(&quot;\n👋 Shutting down...&quot;)<br>    finally:<br>        await sdk.shutdown()<br><br>if __name__ == &quot;__main__&quot;:<br>    asyncio.run(main())</pre><h4>Step 4: Build the LLM-Powered Analysis Agent</h4><p>This is the intelligent core of our system where the analysis agent subscribes to the logs.raw topic and uses OpenAI&#39;s GPT-4 to perform deep, context-aware analysis of each log entry. This LLM-powered agent understands context, identifies novel threats, and provides human-like reasoning about security vulnerabilities, performance anomalies, and operational risks. It automatically classifies severity levels, generates actionable recommendations, and publishes results to the logs.analyzed topic for downstream processing—all while handling 3 parallel consumers for high-throughput analysis.</p><pre># log_analysis_agent.py<br>import asyncio<br>import json<br>from typing import Dict, Any<br>from openai import AsyncOpenAI<br>from omnidaemon import OmniDaemonSDK, AgentConfig, SubscriptionConfig<br>from decouple import config<br><br>sdk = OmniDaemonSDK()<br>client = AsyncOpenAI(api_key=config(&quot;OPENAI_API_KEY&quot;))<br><br>ANALYSIS_SYSTEM_PROMPT = &quot;&quot;&quot;You are an expert security and database analyst. <br>Analyze log entries and provide detailed insights on:<br><br>1. **Security Vulnerabilities**: Identify potential security threats, unusual access patterns, <br>   brute force attempts, suspicious IP addresses, or unauthorized access attempts.<br><br>2. **Anomalies**: Detect unusual patterns such as:<br>   - Failed login attempts from the same IP<br>   - Impossible travel (logins from distant locations in short time)<br>   - Unusual database operations (mass deletions, slow queries)<br>   - Off-hours access patterns<br>   - Spike in certain operations<br><br>3. **Critical Actions**: Flag actions that require immediate attention:<br>   - Successful logins after multiple failures<br>   - Large-scale data modifications<br>   - Administrative privilege escalation<br>   - Database performance degradation<br><br>4. **Recommendations**: Provide specific, actionable recommendations:<br>   - Implement IP blocking for suspicious addresses<br>   - Add monitoring for specific patterns<br>   - Optimize slow queries<br>   - Review user permissions<br>   - Enable additional security measures<br><br>Return your analysis as a JSON object with this structure:<br>{<br>    &quot;severity&quot;: &quot;critical|high|medium|low|info&quot;,<br>    &quot;category&quot;: &quot;security|performance|anomaly|normal&quot;,<br>    &quot;vulnerabilities&quot;: [&quot;list of identified vulnerabilities&quot;],<br>    &quot;anomalies&quot;: [&quot;list of detected anomalies&quot;],<br>    &quot;critical_actions&quot;: [&quot;list of critical actions taken&quot;],<br>    &quot;recommendations&quot;: [&quot;list of specific recommendations&quot;],<br>    &quot;summary&quot;: &quot;brief summary of findings&quot;<br>}<br><br>Be concise but thorough. Focus on actionable insights.&quot;&quot;&quot;<br><br>async def analyze_log_with_llm(log_entry: Dict[str, Any]) -&gt; Dict[str, Any]:<br>    &quot;&quot;&quot;Analyze a log entry using LLM&quot;&quot;&quot;<br>    try:<br>        # Prepare log context for LLM<br>        log_context = json.dumps(log_entry, indent=2)<br>        <br>        # Call OpenAI API<br>        response = await client.chat.completions.create(<br>            model=&quot;gpt-4o&quot;,<br>            messages=[<br>                {&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: ANALYSIS_SYSTEM_PROMPT},<br>                {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: f&quot;Analyze this log entry:\n\n{log_context}&quot;}<br>            ],<br>            temperature=0.3,<br>            max_tokens=1000<br>        )<br>        <br>        # Parse LLM response<br>        analysis_text = response.choices[0].message.content<br>        <br>        # Extract JSON from response (handle markdown code blocks)<br>        if &quot;```json&quot; in analysis_text:<br>            analysis_text = analysis_text.split(&quot;```json&quot;)[1].split(&quot;```&quot;)[0].strip()<br>        elif &quot;```&quot; in analysis_text:<br>            analysis_text = analysis_text.split(&quot;```&quot;)[1].split(&quot;```&quot;)[0].strip()<br>        <br>        analysis = json.loads(analysis_text)<br>        <br>        return {<br>            &quot;status&quot;: &quot;success&quot;,<br>            &quot;analysis&quot;: analysis,<br>            &quot;log_entry&quot;: log_entry<br>        }<br>        <br>    except Exception as e:<br>        print(f&quot;❌ LLM Analysis Error: {e}&quot;)<br>        return {<br>            &quot;status&quot;: &quot;error&quot;,<br>            &quot;error&quot;: str(e),<br>            &quot;log_entry&quot;: log_entry<br>        }<br><br>async def log_analysis_callback(message: dict):<br>    &quot;&quot;&quot;OmniDaemon callback for log analysis&quot;&quot;&quot;<br>    content = message.get(&quot;content&quot;, {})<br>    log_entry = content.get(&quot;log_entry&quot;, {})<br>    source_file = content.get(&quot;source_file&quot;, &quot;unknown&quot;)<br>    <br>    print(f&quot;\n🔍 Analyzing {log_entry.get(&#39;log_type&#39;)} log from {source_file}&quot;)<br>    <br>    # Analyze log with LLM<br>    result = await analyze_log_with_llm(log_entry)<br>    <br>    if result[&quot;status&quot;] == &quot;success&quot;:<br>        analysis = result[&quot;analysis&quot;]<br>        severity = analysis.get(&quot;severity&quot;, &quot;info&quot;)<br>        category = analysis.get(&quot;category&quot;, &quot;normal&quot;)<br>        <br>        # Print severity-based alerts<br>        if severity == &quot;critical&quot;:<br>            print(f&quot;🚨 CRITICAL: {analysis.get(&#39;summary&#39;)}&quot;)<br>        elif severity == &quot;high&quot;:<br>            print(f&quot;⚠️  HIGH: {analysis.get(&#39;summary&#39;)}&quot;)<br>        elif severity == &quot;medium&quot;:<br>            print(f&quot;⚡ MEDIUM: {analysis.get(&#39;summary&#39;)}&quot;)<br>        else:<br>            print(f&quot;ℹ️  {analysis.get(&#39;summary&#39;)}&quot;)<br>        <br>        # Print recommendations if any<br>        if analysis.get(&quot;recommendations&quot;):<br>            print(f&quot;💡 Recommendations:&quot;)<br>            for rec in analysis[&quot;recommendations&quot;][:2]:  # Show top 2<br>                print(f&quot;   - {rec}&quot;)<br>    <br>    return result<br><br>async def main():<br>    try:<br>        print(&quot;🤖 Starting Log Analysis Agent...&quot;)<br>        <br>        # Register analysis agent<br>        await sdk.register_agent(<br>            agent_config=AgentConfig(<br>                name=&quot;LOG_ANALYSIS_AGENT&quot;,<br>                topic=&quot;logs.raw&quot;,<br>                callback=log_analysis_callback,<br>                description=&quot;LLM-powered log analysis agent&quot;,<br>                tools=[&quot;openai&quot;, &quot;security_analysis&quot;],<br>                config=SubscriptionConfig(<br>                    reclaim_idle_ms=30000,<br>                    dlq_retry_limit=2,<br>                    consumer_count=3  # Parallel processing<br>                )<br>            )<br>        )<br>        <br>        # Start agent runner<br>        await sdk.start()<br>        print(&quot;✅ Log Analysis Agent is now running&quot;)<br>        print(&quot;🎧 Listening for logs on &#39;logs.raw&#39; topic...&quot;)<br>        <br>        # Keep running<br>        try:<br>            while True:<br>                await asyncio.sleep(1)<br>        except KeyboardInterrupt:<br>            print(&quot;\n👋 Received shutdown signal...&quot;)<br>    <br>    except Exception as e:<br>        print(f&quot;❌ Error: {e}&quot;)<br>        raise<br>    <br>    finally:<br>        print(&quot;Shutting down...&quot;)<br>        await sdk.shutdown()<br>        print(&quot;✅ Shutdown complete&quot;)<br><br>if __name__ == &quot;__main__&quot;:<br>    asyncio.run(main())</pre><h4>Step 5: Build the Reporting Agent</h4><p>The reporting agent is the final stage of our pipeline that transforms individual log analyses into actionable intelligence. It subscribes to the logs.analyzed topic, aggregates insights by severity level (critical, high, medium, low), and automatically generates comprehensive reports every 20 insights. This agent produces JSON reports that can feed into dashboards, alerting systems, or business intelligence tools.</p><pre># reporting_agent.py<br>import asyncio<br>import json<br>from datetime import datetime<br>from collections import defaultdict<br>from pathlib import Path<br>from omnidaemon import OmniDaemonSDK, AgentConfig, SubscriptionConfig<br><br>sdk = OmniDaemonSDK()<br><br>class ReportingAgent:<br>    def __init__(self):<br>        self.insights = defaultdict(list)<br>        self.reports_dir = Path(&quot;./reports&quot;)<br>        self.reports_dir.mkdir(exist_ok=True)<br>        <br>    async def process_analysis(self, message: dict):<br>        &quot;&quot;&quot;Process analyzed log and aggregate insights&quot;&quot;&quot;<br>        content = message.get(&quot;content&quot;, {})<br>        <br>        if content.get(&quot;status&quot;) != &quot;success&quot;:<br>            return {&quot;status&quot;: &quot;skipped&quot;, &quot;reason&quot;: &quot;analysis failed&quot;}<br>        <br>        analysis = content.get(&quot;analysis&quot;, {})<br>        log_entry = content.get(&quot;log_entry&quot;, {})<br>        <br>        # Aggregate insights by severity<br>        severity = analysis.get(&quot;severity&quot;, &quot;info&quot;)<br>        self.insights[severity].append({<br>            &quot;timestamp&quot;: log_entry.get(&quot;timestamp&quot;),<br>            &quot;log_type&quot;: log_entry.get(&quot;log_type&quot;),<br>            &quot;category&quot;: analysis.get(&quot;category&quot;),<br>            &quot;summary&quot;: analysis.get(&quot;summary&quot;),<br>            &quot;vulnerabilities&quot;: analysis.get(&quot;vulnerabilities&quot;, []),<br>            &quot;anomalies&quot;: analysis.get(&quot;anomalies&quot;, []),<br>            &quot;critical_actions&quot;: analysis.get(&quot;critical_actions&quot;, []),<br>            &quot;recommendations&quot;: analysis.get(&quot;recommendations&quot;, [])<br>        })<br>        <br>        # Generate report every 20 insights<br>        total_insights = sum(len(v) for v in self.insights.values())<br>        if total_insights % 20 == 0:<br>            await self.generate_report()<br>        <br>        return {<br>            &quot;status&quot;: &quot;processed&quot;,<br>            &quot;severity&quot;: severity,<br>            &quot;total_insights&quot;: total_insights<br>        }<br>    <br>    async def generate_report(self):<br>        &quot;&quot;&quot;Generate comprehensive security and performance report&quot;&quot;&quot;<br>        try:<br>            timestamp = datetime.now().strftime(&quot;%Y%m%d_%H%M%S&quot;)<br>            report_file = self.reports_dir / f&quot;log_insights_report_{timestamp}.json&quot;<br>            <br>            # Compile statistics<br>            stats = {<br>                &quot;generated_at&quot;: datetime.now().isoformat(),<br>                &quot;total_insights&quot;: sum(len(v) for v in self.insights.values()),<br>                &quot;by_severity&quot;: {k: len(v) for k, v in self.insights.items()},<br>                &quot;insights&quot;: dict(self.insights)<br>            }<br>            <br>            # Extract top recommendations<br>            all_recommendations = []<br>            for severity_insights in self.insights.values():<br>                for insight in severity_insights:<br>                    all_recommendations.extend(insight.get(&quot;recommendations&quot;, []))<br>            <br>            # Count recommendation frequency<br>            rec_counts = defaultdict(int)<br>            for rec in all_recommendations:<br>                rec_counts[rec] += 1<br>            <br>            top_recommendations = sorted(<br>                rec_counts.items(), <br>                key=lambda x: x[1], <br>                reverse=True<br>            )[:10]<br>            <br>            stats[&quot;top_recommendations&quot;] = [<br>                {&quot;recommendation&quot;: rec, &quot;frequency&quot;: count}<br>                for rec, count in top_recommendations<br>            ]<br>            <br>            # Save report<br>            with open(report_file, &quot;w&quot;) as f:<br>                json.dump(stats, f, indent=2)<br>            <br>            print(f&quot;\n📊 Report generated: {report_file}&quot;)<br>            print(f&quot;   Total Insights: {stats[&#39;total_insights&#39;]}&quot;)<br>            print(f&quot;   Critical: {stats[&#39;by_severity&#39;].get(&#39;critical&#39;, 0)}&quot;)<br>            print(f&quot;   High: {stats[&#39;by_severity&#39;].get(&#39;high&#39;, 0)}&quot;)<br>            print(f&quot;   Medium: {stats[&#39;by_severity&#39;].get(&#39;medium&#39;, 0)}&quot;)<br>            <br>            if top_recommendations:<br>                print(f&quot;\n🎯 Top Recommendations:&quot;)<br>                for rec, count in top_recommendations[:3]:<br>                    print(f&quot;   - {rec} (mentioned {count} times)&quot;)<br>            <br>        except Exception as e:<br>            print(f&quot;❌ Error generating report: {e}&quot;)<br><br>async def main():<br>    try:<br>        print(&quot;📊 Starting Reporting Agent...&quot;)<br>        <br>        reporting_agent = ReportingAgent()<br>        <br>        # Register reporting agent<br>        await sdk.register_agent(<br>            agent_config=AgentConfig(<br>                name=&quot;REPORTING_AGENT&quot;,<br>                topic=&quot;logs.analyzed&quot;,<br>                callback=reporting_agent.process_analysis,<br>                description=&quot;Aggregates analysis results and generates reports&quot;,<br>                config=SubscriptionConfig(<br>                    consumer_count=1  # Single consumer for aggregation<br>                )<br>            )<br>        )<br>        <br>        # Start agent runner<br>        await sdk.start()<br>        print(&quot;✅ Reporting Agent is now running&quot;)<br>        print(&quot;🎧 Listening for analyzed logs on &#39;logs.analyzed&#39; topic...&quot;)<br>        <br>        # Keep running<br>        try:<br>            while True:<br>                await asyncio.sleep(1)<br>        except KeyboardInterrupt:<br>            print(&quot;\n👋 Generating final report...&quot;)<br>            await reporting_agent.generate_report()<br>    <br>    except Exception as e:<br>        print(f&quot;❌ Error: {e}&quot;)<br>        raise<br>    <br>    finally:<br>        print(&quot;Shutting down...&quot;)<br>        await sdk.shutdown()<br>        print(&quot;✅ Shutdown complete&quot;)<br><br>if __name__ == &quot;__main__&quot;:<br>    asyncio.run(main())</pre><h4>Executing the app</h4><p>You can now run the agents in different terminals to see how they work together as one system.</p><pre>python generate_logs.py<br>python log_ingestion_agent.py<br>python log_analysis_agent.py<br>python reporting_agent.py</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4V5Vjg-EsOa5SG-qlSNXFw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0q569OZ00D0r942vkwZ3TA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z-yHn0DqWRE35rW7VqnZ0w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vDL0fAJJznx0agYfoHu2MQ.png" /></figure><h4>Step 6: (Optional) Advanced Agent Chaining for Specialized Analysis</h4><p>For more sophisticated log analysis, we can create a multi-stage pipeline with specialized agents. This demonstrates OmniDaemon’s powerful agent chaining capabilities.</p><p>Create a new file for the chained agent system:</p><pre># chained_analysis_agents.py<br>import asyncio<br>import json<br>from typing import Dict, Any<br>from openai import AsyncOpenAI<br>from omnidaemon import OmniDaemonSDK, AgentConfig, SubscriptionConfig, EventEnvelope, PayloadBase<br>from decouple import config<br><br>sdk = OmniDaemonSDK()<br>client = AsyncOpenAI(api_key=config(&quot;OPENAI_API_KEY&quot;))<br><br># ============================================================================<br># AGENT 1: Parse and Classify Logs<br># ============================================================================<br><br>async def parse_and_classify(message: dict):<br>    &quot;&quot;&quot;<br>    First agent in the chain: Parse log entries and classify them<br>    Routes logs to specialized agents based on type and content<br>    &quot;&quot;&quot;<br>    content = message.get(&quot;content&quot;, {})<br>    log_entry = content.get(&quot;log_entry&quot;, {})<br>    source_file = content.get(&quot;source_file&quot;, &quot;unknown&quot;)<br>    <br>    print(f&quot;\n🔍 [CLASSIFIER] Processing log from {source_file}&quot;)<br>    <br>    # Classify log type and determine routing<br>    log_type = log_entry.get(&quot;log_type&quot;, &quot;unknown&quot;)<br>    classification = {<br>        &quot;log_entry&quot;: log_entry,<br>        &quot;source_file&quot;: source_file,<br>        &quot;classification&quot;: None,<br>        &quot;route_to&quot;: None,<br>        &quot;priority&quot;: &quot;normal&quot;<br>    }<br>    <br>    if log_type == &quot;authentication&quot;:<br>        # Check for security concerns<br>        success = log_entry.get(&quot;success&quot;, True)<br>        event = log_entry.get(&quot;event&quot;, &quot;&quot;)<br>        ip = log_entry.get(&quot;ip_address&quot;, &quot;&quot;)<br>        <br>        # Detect security-related patterns<br>        is_security_concern = (<br>            not success or<br>            event in [&quot;failed_login&quot;, &quot;brute_force_attempt&quot;, &quot;impossible_travel&quot;] or<br>            ip.startswith((&quot;45.&quot;, &quot;89.&quot;))  # Suspicious IP ranges<br>        )<br>        <br>        if is_security_concern:<br>            classification[&quot;classification&quot;] = &quot;security_threat&quot;<br>            classification[&quot;route_to&quot;] = &quot;logs.security&quot;<br>            classification[&quot;priority&quot;] = &quot;high&quot;<br>            print(f&quot;   ⚠️  Classified as: SECURITY THREAT → Routing to security agent&quot;)<br>        else:<br>            classification[&quot;classification&quot;] = &quot;normal_auth&quot;<br>            classification[&quot;route_to&quot;] = &quot;logs.security&quot;<br>            classification[&quot;priority&quot;] = &quot;low&quot;<br>            print(f&quot;   ✅ Classified as: Normal Authentication → Security agent&quot;)<br>    <br>    elif log_type == &quot;database&quot;:<br>        # Check for performance concerns<br>        execution_time = log_entry.get(&quot;execution_time_ms&quot;, 0)<br>        rows_affected = log_entry.get(&quot;rows_affected&quot;, 0)<br>        operation = log_entry.get(&quot;operation&quot;, &quot;&quot;)<br>        <br>        # Detect performance issues<br>        is_performance_issue = (<br>            execution_time &gt; 1000 or  # Slow query (&gt; 1 second)<br>            (operation == &quot;DELETE&quot; and rows_affected &gt; 500) or<br>            rows_affected &gt; 5000<br>        )<br>        <br>        if is_performance_issue:<br>            classification[&quot;classification&quot;] = &quot;performance_issue&quot;<br>            classification[&quot;route_to&quot;] = &quot;logs.performance&quot;<br>            classification[&quot;priority&quot;] = &quot;high&quot;<br>            print(f&quot;   🐌 Classified as: PERFORMANCE ISSUE → Routing to performance agent&quot;)<br>        else:<br>            classification[&quot;classification&quot;] = &quot;normal_db&quot;<br>            classification[&quot;route_to&quot;] = &quot;logs.performance&quot;<br>            classification[&quot;priority&quot;] = &quot;low&quot;<br>            print(f&quot;   ✅ Classified as: Normal Database Op → Performance agent&quot;)<br>    <br>    else:<br>        classification[&quot;classification&quot;] = &quot;unknown&quot;<br>        classification[&quot;route_to&quot;] = &quot;logs.general&quot;<br>        print(f&quot;   ❓ Unknown log type → General analysis&quot;)<br>    <br>    # Publish to appropriate specialized agent with reply_to for final reporting<br>    await sdk.publish_task(<br>        event_envelope=EventEnvelope(<br>            topic=classification[&quot;route_to&quot;],<br>            payload=PayloadBase(<br>                content=classification,<br>                reply_to=&quot;logs.final_report&quot;  # All results go to final reporting<br>            ),<br>            source=&quot;classifier_agent&quot;,<br>            correlation_id=message.get(&quot;correlation_id&quot;, f&quot;corr_{log_entry.get(&#39;timestamp&#39;)}&quot;)<br>        )<br>    )<br>    <br>    return {<br>        &quot;status&quot;: &quot;classified&quot;,<br>        &quot;classification&quot;: classification[&quot;classification&quot;],<br>        &quot;routed_to&quot;: classification[&quot;route_to&quot;],<br>        &quot;priority&quot;: classification[&quot;priority&quot;]<br>    }<br><br># ============================================================================<br># AGENT 2: Security-Specific Analysis<br># ============================================================================<br><br>SECURITY_PROMPT = &quot;&quot;&quot;You are a cybersecurity expert specializing in authentication and access control.<br>Analyze this log entry for security threats:<br><br>1. Identify specific attack patterns (brute force, credential stuffing, account takeover)<br>2. Assess threat severity (critical, high, medium, low)<br>3. Check for IOCs (Indicators of Compromise): suspicious IPs, user agents, access patterns<br>4. Recommend immediate security actions<br><br>Return JSON:<br>{<br>    &quot;threat_level&quot;: &quot;critical|high|medium|low&quot;,<br>    &quot;attack_type&quot;: &quot;description of attack pattern&quot;,<br>    &quot;iocs&quot;: [&quot;list of indicators of compromise&quot;],<br>    &quot;immediate_actions&quot;: [&quot;list of urgent actions needed&quot;],<br>    &quot;recommendations&quot;: [&quot;security recommendations&quot;]<br>}&quot;&quot;&quot;<br><br>async def security_analysis(message: dict):<br>    &quot;&quot;&quot;<br>    Specialized agent for security analysis<br>    Deep dive into authentication logs and security threats<br>    &quot;&quot;&quot;<br>    content = message.get(&quot;content&quot;, {})<br>    log_entry = content.get(&quot;log_entry&quot;, {})<br>    classification = content.get(&quot;classification&quot;, &quot;unknown&quot;)<br>    priority = content.get(&quot;priority&quot;, &quot;normal&quot;)<br>    <br>    print(f&quot;\n🛡️  [SECURITY] Analyzing {classification} (Priority: {priority})&quot;)<br>    <br>    try:<br>        # Prepare security context<br>        log_context = json.dumps(log_entry, indent=2)<br>        <br>        # Call LLM for security analysis<br>        response = await client.chat.completions.create(<br>            model=&quot;gpt-4o&quot;,<br>            messages=[<br>                {&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: SECURITY_PROMPT},<br>                {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: f&quot;Analyze this authentication log:\n\n{log_context}&quot;}<br>            ],<br>            temperature=0.2,  # Lower temperature for security analysis<br>            max_tokens=800<br>        )<br>        <br>        analysis_text = response.choices[0].message.content<br>        <br>        # Parse JSON from response<br>        if &quot;```json&quot; in analysis_text:<br>            analysis_text = analysis_text.split(&quot;```json&quot;)[1].split(&quot;```&quot;)[0].strip()<br>        elif &quot;```&quot; in analysis_text:<br>            analysis_text = analysis_text.split(&quot;```&quot;)[1].split(&quot;```&quot;)[0].strip()<br>        <br>        security_analysis = json.loads(analysis_text)<br>        <br>        # Log findings<br>        threat_level = security_analysis.get(&quot;threat_level&quot;, &quot;low&quot;)<br>        if threat_level in [&quot;critical&quot;, &quot;high&quot;]:<br>            print(f&quot;   🚨 THREAT DETECTED: {security_analysis.get(&#39;attack_type&#39;)}&quot;)<br>            print(f&quot;   📍 IOCs: {&#39;, &#39;.join(security_analysis.get(&#39;iocs&#39;, [])[:2])}&quot;)<br>        else:<br>            print(f&quot;   ✅ No significant threats detected&quot;)<br>        <br>        return {<br>            &quot;status&quot;: &quot;analyzed&quot;,<br>            &quot;agent_type&quot;: &quot;security&quot;,<br>            &quot;log_entry&quot;: log_entry,<br>            &quot;classification&quot;: classification,<br>            &quot;analysis&quot;: security_analysis,<br>            &quot;correlation_id&quot;: message.get(&quot;correlation_id&quot;)<br>        }<br>        <br>    except Exception as e:<br>        print(f&quot;   ❌ Security analysis error: {e}&quot;)<br>        return {<br>            &quot;status&quot;: &quot;error&quot;,<br>            &quot;agent_type&quot;: &quot;security&quot;,<br>            &quot;error&quot;: str(e),<br>            &quot;log_entry&quot;: log_entry<br>        }<br><br># ============================================================================<br># AGENT 3: Performance Analysis<br># ============================================================================<br><br>PERFORMANCE_PROMPT = &quot;&quot;&quot;You are a database performance expert.<br>Analyze this database transaction log for performance issues:<br><br>1. Identify slow queries and resource bottlenecks<br>2. Detect unusual data access patterns<br>3. Flag potential database design issues<br>4. Recommend query optimizations and indexing strategies<br><br>Return JSON:<br>{<br>    &quot;performance_score&quot;: &quot;excellent|good|degraded|critical&quot;,<br>    &quot;issues_found&quot;: [&quot;list of performance issues&quot;],<br>    &quot;bottlenecks&quot;: [&quot;identified bottlenecks&quot;],<br>    &quot;optimization_recommendations&quot;: [&quot;specific optimization steps&quot;],<br>    &quot;estimated_impact&quot;: &quot;description of performance impact&quot;<br>}&quot;&quot;&quot;<br><br>async def performance_analysis(message: dict):<br>    &quot;&quot;&quot;<br>    Specialized agent for database performance analysis<br>    Focuses on query optimization and resource utilization<br>    &quot;&quot;&quot;<br>    content = message.get(&quot;content&quot;, {})<br>    log_entry = content.get(&quot;log_entry&quot;, {})<br>    classification = content.get(&quot;classification&quot;, &quot;unknown&quot;)<br>    priority = content.get(&quot;priority&quot;, &quot;normal&quot;)<br>    <br>    print(f&quot;\n⚡ [PERFORMANCE] Analyzing {classification} (Priority: {priority})&quot;)<br>    <br>    try:<br>        # Prepare performance context<br>        log_context = json.dumps(log_entry, indent=2)<br>        <br>        # Call LLM for performance analysis<br>        response = await client.chat.completions.create(<br>            model=&quot;gpt-4o&quot;,<br>            messages=[<br>                {&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: PERFORMANCE_PROMPT},<br>                {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: f&quot;Analyze this database log:\n\n{log_context}&quot;}<br>            ],<br>            temperature=0.2,<br>            max_tokens=800<br>        )<br>        <br>        analysis_text = response.choices[0].message.content<br>        <br>        # Parse JSON from response<br>        if &quot;```json&quot; in analysis_text:<br>            analysis_text = analysis_text.split(&quot;```json&quot;)[1].split(&quot;```&quot;)[0].strip()<br>        elif &quot;```&quot; in analysis_text:<br>            analysis_text = analysis_text.split(&quot;```&quot;)[1].split(&quot;```&quot;)[0].strip()<br>        <br>        perf_analysis = json.loads(analysis_text)<br>        <br>        # Log findings<br>        perf_score = perf_analysis.get(&quot;performance_score&quot;, &quot;good&quot;)<br>        if perf_score in [&quot;degraded&quot;, &quot;critical&quot;]:<br>            print(f&quot;   🐌 PERFORMANCE ISSUE: {perf_score.upper()}&quot;)<br>            issues = perf_analysis.get(&quot;issues_found&quot;, [])<br>            if issues:<br>                print(f&quot;   📊 Issues: {&#39;, &#39;.join(issues[:2])}&quot;)<br>        else:<br>            print(f&quot;   ✅ Performance acceptable: {perf_score}&quot;)<br>        <br>        return {<br>            &quot;status&quot;: &quot;analyzed&quot;,<br>            &quot;agent_type&quot;: &quot;performance&quot;,<br>            &quot;log_entry&quot;: log_entry,<br>            &quot;classification&quot;: classification,<br>            &quot;analysis&quot;: perf_analysis,<br>            &quot;correlation_id&quot;: message.get(&quot;correlation_id&quot;)<br>        }<br>        <br>    except Exception as e:<br>        print(f&quot;   ❌ Performance analysis error: {e}&quot;)<br>        return {<br>            &quot;status&quot;: &quot;error&quot;,<br>            &quot;agent_type&quot;: &quot;performance&quot;,<br>            &quot;error&quot;: str(e),<br>            &quot;log_entry&quot;: log_entry<br>        }<br><br># ============================================================================<br># AGENT 4: Final Report Aggregator<br># ============================================================================<br><br>async def final_report_aggregator(message: dict):<br>    &quot;&quot;&quot;<br>    Final agent in the chain: Aggregates specialized analyses<br>    Combines security and performance insights into unified report<br>    &quot;&quot;&quot;<br>    content = message.get(&quot;content&quot;, {})<br>    agent_type = content.get(&quot;agent_type&quot;, &quot;unknown&quot;)<br>    analysis = content.get(&quot;analysis&quot;, {})<br>    log_entry = content.get(&quot;log_entry&quot;, {})<br>    correlation_id = content.get(&quot;correlation_id&quot;, &quot;unknown&quot;)<br>    <br>    print(f&quot;\n📋 [FINAL REPORT] Aggregating {agent_type} analysis (ID: {correlation_id})&quot;)<br>    <br>    # Create unified report<br>    report = {<br>        &quot;correlation_id&quot;: correlation_id,<br>        &quot;timestamp&quot;: log_entry.get(&quot;timestamp&quot;),<br>        &quot;log_type&quot;: log_entry.get(&quot;log_type&quot;),<br>        &quot;agent_type&quot;: agent_type,<br>        &quot;analysis&quot;: analysis<br>    }<br>    <br>    # Determine overall severity<br>    if agent_type == &quot;security&quot;:<br>        threat_level = analysis.get(&quot;threat_level&quot;, &quot;low&quot;)<br>        if threat_level in [&quot;critical&quot;, &quot;high&quot;]:<br>            print(f&quot;   🚨 SECURITY ALERT: {threat_level.upper()} threat detected&quot;)<br>            print(f&quot;   🎯 Actions: {&#39;, &#39;.join(analysis.get(&#39;immediate_actions&#39;, [])[:2])}&quot;)<br>    <br>    elif agent_type == &quot;performance&quot;:<br>        perf_score = analysis.get(&quot;performance_score&quot;, &quot;good&quot;)<br>        if perf_score in [&quot;degraded&quot;, &quot;critical&quot;]:<br>            print(f&quot;   ⚠️  PERFORMANCE ALERT: {perf_score.upper()} detected&quot;)<br>            print(f&quot;   💡 Recommendations: {&#39;, &#39;.join(analysis.get(&#39;optimization_recommendations&#39;, [])[:2])}&quot;)<br>    <br>    return {<br>        &quot;status&quot;: &quot;reported&quot;,<br>        &quot;report&quot;: report<br>    }<br><br># ============================================================================<br># Main: Register All Agents in the Chain<br># ============================================================================<br><br>async def main():<br>    try:<br>        print(&quot;🚀 Starting Chained Analysis System...&quot;)<br>        print(&quot;=&quot; * 70)<br>        print(&quot;Pipeline: Classifier → Security/Performance → Final Report&quot;)<br>        print(&quot;=&quot; * 70)<br>        <br>        # Register Agent 1: Classifier (entry point)<br>        await sdk.register_agent(<br>            agent_config=AgentConfig(<br>                name=&quot;CLASSIFIER_AGENT&quot;,<br>                topic=&quot;logs.raw&quot;,<br>                callback=parse_and_classify,<br>                description=&quot;Parses and classifies logs for routing&quot;,<br>                config=SubscriptionConfig(<br>                    consumer_count=2  # Parallel classification<br>                )<br>            )<br>        )<br>        print(&quot;✅ Registered: Classifier Agent (logs.raw)&quot;)<br>        <br>        # Register Agent 2: Security Analyzer<br>        await sdk.register_agent(<br>            agent_config=AgentConfig(<br>                name=&quot;SECURITY_ANALYSIS_AGENT&quot;,<br>                topic=&quot;logs.security&quot;,<br>                callback=security_analysis,<br>                description=&quot;Specialized security threat analysis&quot;,<br>                tools=[&quot;openai&quot;, &quot;security_analysis&quot;],<br>                config=SubscriptionConfig(<br>                    consumer_count=3  # More consumers for security<br>                )<br>            )<br>        )<br>        print(&quot;✅ Registered: Security Analysis Agent (logs.security)&quot;)<br>        <br>        # Register Agent 3: Performance Analyzer<br>        await sdk.register_agent(<br>            agent_config=AgentConfig(<br>                name=&quot;PERFORMANCE_ANALYSIS_AGENT&quot;,<br>                topic=&quot;logs.performance&quot;,<br>                callback=performance_analysis,<br>                description=&quot;Specialized database performance analysis&quot;,<br>                tools=[&quot;openai&quot;, &quot;performance_analysis&quot;],<br>                config=SubscriptionConfig(<br>                    consumer_count=3<br>                )<br>            )<br>        )<br>        print(&quot;✅ Registered: Performance Analysis Agent (logs.performance)&quot;)<br>        <br>        # Register Agent 4: Final Report Aggregator<br>        await sdk.register_agent(<br>            agent_config=AgentConfig(<br>                name=&quot;FINAL_REPORT_AGENT&quot;,<br>                topic=&quot;logs.final_report&quot;,<br>                callback=final_report_aggregator,<br>                description=&quot;Aggregates specialized analysis into final reports&quot;,<br>                config=SubscriptionConfig(<br>                    consumer_count=1  # Single consumer for aggregation<br>                )<br>            )<br>        )<br>        print(&quot;✅ Registered: Final Report Agent (logs.final_report)&quot;)<br>        <br>        # Start all agents<br>        await sdk.start()<br>        print(&quot;\n&quot; + &quot;=&quot; * 70)<br>        print(&quot;🎧 Chained Analysis System is LIVE!&quot;)<br>        print(&quot;=&quot; * 70)<br>        print(&quot;\nAgent Chain Flow:&quot;)<br>        print(&quot;  1️⃣  logs.raw → Classifier&quot;)<br>        print(&quot;  2️⃣  Classifier → logs.security OR logs.performance&quot;)<br>        print(&quot;  3️⃣  Specialized Agent → logs.final_report&quot;)<br>        print(&quot;  4️⃣  Final Report → Aggregated Insights&quot;)<br>        print(&quot;\nPress Ctrl+C to stop...&quot;)<br>        <br>        # Keep running<br>        try:<br>            while True:<br>                await asyncio.sleep(1)<br>        except KeyboardInterrupt:<br>            print(&quot;\n\n👋 Received shutdown signal...&quot;)<br>    <br>    except Exception as e:<br>        print(f&quot;❌ Error: {e}&quot;)<br>        raise<br>    <br>    finally:<br>        print(&quot;Shutting down all agents...&quot;)<br>        await sdk.shutdown()<br>        print(&quot;✅ Shutdown complete&quot;)<br><br>if __name__ == &quot;__main__&quot;:<br>    asyncio.run(main())</pre><p>Run the chained the agent to see the combined results.</p><pre>python chained_analysis_agents.py</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6ogB_g0ZTopHnQtJdFHEhw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XW7nZxka7oQw9ZVzfhJjgw.png" /></figure><h3>Advanced Features and Patterns</h3><h4>1. Agent Chaining for Complex Workflows</h4><p>The chained system we built demonstrates powerful agent orchestration patterns:</p><p><strong>Pattern 1: Intelligent Classification and Routing</strong></p><pre># Classifier examines log and routes to specialist<br>async def parse_and_classify(message: dict):<br>    log_entry = message.get(&quot;content&quot;, {}).get(&quot;log_entry&quot;, {})<br>    <br>    # Determine routing based on content analysis<br>    if is_security_concern(log_entry):<br>        route_to = &quot;logs.security&quot;<br>        priority = &quot;high&quot;<br>    elif is_performance_issue(log_entry):<br>        route_to = &quot;logs.performance&quot;<br>        priority = &quot;high&quot;<br>    else:<br>        route_to = &quot;logs.general&quot;<br>        priority = &quot;normal&quot;<br>    <br>    # Publish to specialized agent with correlation tracking<br>    await sdk.publish_task(<br>        event_envelope=EventEnvelope(<br>            topic=route_to,<br>            payload=PayloadBase(<br>                content=classification_data,<br>                reply_to=&quot;logs.final_report&quot;  # All results aggregated here<br>            ),<br>            correlation_id=generate_correlation_id(log_entry)<br>        )<br>    )</pre><p><strong>Pattern 2: Parallel Specialized Processing</strong></p><pre># Multiple specialized agents process different aspects simultaneously<br># Security agent handles authentication threats<br>await sdk.register_agent(<br>    agent_config=AgentConfig(<br>        topic=&quot;logs.security&quot;,<br>        callback=security_analysis,<br>        config=SubscriptionConfig(consumer_count=3)  # 3 parallel security analysts<br>    )<br>)<br><br># Performance agent handles database optimization<br>await sdk.register_agent(<br>    agent_config=AgentConfig(<br>        topic=&quot;logs.performance&quot;,<br>        callback=performance_analysis,<br>        config=SubscriptionConfig(consumer_count=3)  # 3 parallel performance analysts<br>    )<br>)</pre><p><strong>Pattern 3: Correlation-Based Aggregation</strong></p><pre># Final agent aggregates related analyses using correlation_id<br>async def final_report_aggregator(message: dict):<br>    correlation_id = message.get(&quot;correlation_id&quot;)<br>    <br>    # Track related events across pipeline<br>    # All events with same correlation_id are part of same analysis flow<br>    report = {<br>        &quot;correlation_id&quot;: correlation_id,<br>        &quot;security_analysis&quot;: ...,<br>        &quot;performance_analysis&quot;: ...,<br>        &quot;unified_recommendations&quot;: ...<br>    }</pre><h4>2. Multi-Tenant Log Analysis</h4><p>Extend the system to handle multiple tenants:</p><pre>async def multi_tenant_analysis_callback(message: dict):<br>    tenant_id = message.get(&quot;tenant_id&quot;)<br>    content = message.get(&quot;content&quot;, {})<br>    <br>    # Load tenant-specific security policies<br>    policies = await load_tenant_policies(tenant_id)<br>    <br>    # Analyze with tenant context<br>    result = await analyze_log_with_llm(content, policies)<br>    <br>    return {<br>        &quot;status&quot;: &quot;success&quot;,<br>        &quot;tenant_id&quot;: tenant_id,<br>        &quot;analysis&quot;: result<br>    }</pre><h4>2. Multi-Tenant Log Analysis</h4><p>Send immediate alerts for critical issues:</p><pre>event = EventEnvelope(<br>    topic=&quot;logs.raw&quot;,<br>    payload=PayloadBase(<br>        content=log_data,<br>        webhook=&quot;https://your-api.com/alerts&quot;  # Receive instant notifications<br>    )<br>)</pre><h4>3. Real-Time Alerting with Webhooks</h4><p>Handle high log volumes by scaling agents:</p><pre>await sdk.register_agent(<br>    agent_config=AgentConfig(<br>        name=&quot;LOG_ANALYSIS_AGENT&quot;,<br>        topic=&quot;logs.raw&quot;,<br>        callback=analyze_log,<br>        config=SubscriptionConfig(<br>            consumer_count=10  # Run 10 parallel consumers<br>        )<br>    )<br>)</pre><h4>4. Horizontal Scaling</h4><p><strong>Monitoring and Observability</strong></p><pre># Check system health<br>health = await sdk.health()<br>print(f&quot;Status: {health[&#39;status&#39;]}&quot;)<br>print(f&quot;Active Agents: {health[&#39;registered_agents_count&#39;]}&quot;)<br><br># Check dead letter queue<br>omnidaemon bus dlq --topic logs.raw<br># View metrics<br>omnidaemon metrics list</pre><p><strong>Error Handling and Retry Logic</strong></p><pre>async def robust_analysis_callback(message: dict):<br>    try:<br>        result = await analyze_log_with_llm(message.get(&quot;content&quot;))<br>        return {&quot;status&quot;: &quot;success&quot;, &quot;data&quot;: result}<br>    except RateLimitError as e:<br>        # Retriable error - OmniDaemon will retry<br>        raise<br>    except InvalidLogFormat as e:<br>        # Non-retriable error - return error status<br>        return {&quot;status&quot;: &quot;error&quot;, &quot;error&quot;: str(e)}</pre><h4>Performance Optimization</h4><pre># Configuration for high-throughput scenarios<br>STORAGE_BACKEND=redis  # Use Redis for distributed systems<br>EVENT_BUS_TYPE=redis_stream  # Or Kafka for massive scale<br>REDIS_URL=redis://prod-redis.example.com:6379<br><br># Increase consumer count for parallel processing<br>consumer_count=10<br># Optimize LLM calls with batching<br>async def batch_analyze_logs(log_batch: list):<br>    # Analyze multiple logs in single LLM call<br>    pass</pre><p><em>Thanks for reading this article !!</em></p><p>If you enjoyed this article, please click on the clap button 👏 and share to help others find it!</p><p>The full source code for this tutorial can be found here,</p><ul><li><a href="https://github.com/codemaker2015/intelligent-log-insights-generator">GitHub - codemaker2015/intelligent-log-insights-generator: An AI-powered log analysis system that monitors authentication and database transaction logs, detects anomalies, identifies vulnerabilities, and provides actionable recommendations using OmniDeamon - the universal event-driven runtime engine</a></li><li><a href="https://github.com/codemaker2015/omnidaemon-demo">GitHub - codemaker2015/omnidaemon-demo</a></li></ul><h4>Resources</h4><ul><li><a href="https://omnidaemon.omnirexfloralabs.com/">OmniDaemon - Kubernetes for AI Agents</a></li><li><a href="https://github.com/omnirexflora-labs/OmniDaemon">GitHub - omnirexflora-labs/OmniDaemon: OmniDaemon is a Universal Event-Driven Runtime for AI Agents, it&#39;s framework-agnostic, event-driven runtime that turns AI agents into production-grade, autonomous infrastructure services. It enables agents to listen, react, and collaborate across distributed systems-bringing true event-driven intelligence to modern enterprise environments.</a></li><li><a href="https://docs-omnidaemon.omnirexfloralabs.com/">OmniDaemon - OmniDaemon</a></li><li><a href="https://github.com/omnirexflora-labs/omnicoreagent">GitHub - omnirexflora-labs/omnicoreagent: OmniCoreAgent is a powerful Python framework for building autonomous AI agents that think, reason, and execute complex tasks. Production-ready agents that use tools, manage memory, coordinate workflows, and handle real-world business logic.</a></li><li><a href="https://pypi.org/project/omnidaemon/">Client Challenge</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=02b1a5e63dfb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Exploring TimesFM: The Foundation Model That Understands the Language of Time]]></title>
            <link>https://codemaker2016.medium.com/exploring-timesfm-the-foundation-model-that-understands-the-language-of-time-57486ebca761?source=rss-b7a60e472dfc------2</link>
            <guid isPermaLink="false">https://medium.com/p/57486ebca761</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[forecasting]]></category>
            <category><![CDATA[time-series-forecasting]]></category>
            <category><![CDATA[beginner]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Vishnu Sivan]]></dc:creator>
            <pubDate>Thu, 23 Oct 2025 17:44:09 GMT</pubDate>
            <atom:updated>2025-10-24T05:29:11.162Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*miGA__9RI_kp3iruD6eAsw.png" /></figure><p>Forecasting the future has always been a cornerstone of human progress — from predicting sales for the next quarter to anticipating global health trends. Traditionally, time-series forecasting has relied on statistical models like ARIMA and Exponential Smoothing, and later, deep learning-based methods. While these models have proven effective, they often demand significant domain expertise and extensive fine-tuning for each new dataset.</p><p>In today’s data-driven world, accurate forecasting powers industries such as retail, finance, energy, and healthcare, enabling smarter, evidence-based decision-making. With recent advances in deep learning, the paradigm is shifting — traditional models are being surpassed by large-scale neural architectures capable of understanding complex temporal patterns.</p><p>Enter <strong>TimesFM (Time-series Foundation Model)</strong> — a groundbreaking decoder-only foundation model from Google Research, purpose-built for time-series forecasting. Unlike conventional models that require dataset-specific training, TimesFM achieves remarkable zero-shot forecasting performance, making it a true leap forward in how we approach predictive modeling.</p><p>This article explores what makes TimesFM unique — how it understands the “language of time,” captures trends, rhythms, and seasonality across diverse domains, its core architecture, highlight its innovative features, and walk through practical examples of how to use it effectively in real-world applications.</p><h3>Getting Started</h3><h4>Table of contents</h4><ul><li><a href="#5162">The Evolution of Time-Series Forecasting</a></li><li><a href="#d8e7">What is a Decoder?</a></li><li><a href="#3cf8">What is TimesFM?</a></li><li><a href="#eb13">Architecture Overview</a></li><li><a href="#9920">Experimenting with TimesFM model</a></li><li><a href="#44c4">Handson 1: Zero-Shot Forecasting</a></li><li><a href="#b54a">Handson 2: Zero-Shot Forecasting with stats</a></li><li><a href="#b3e7">Handson 3: Finetuning the model</a></li><li><a href="#fded">Handson 4: Model Comparison</a></li><li><a href="#a813">Building NextGen Forecasting app using TimesFM</a></li></ul><h4>The Evolution of Time-Series Forecasting</h4><p>Time-series forecasting has evolved significantly over the years. Traditional models like ARIMA and Exponential Smoothing performed well for simple, univariate data but struggled with the complexities of modern, multivariate, and high-dimensional datasets.</p><p>The rise of deep learning models such as DeepAR and N-BEATS addressed these challenges by using neural networks capable of capturing long-term dependencies and intricate temporal patterns.</p><p>Now, a new generation of models is redefining the field. TimesFM, a foundation model for time-series forecasting, is pre-trained on diverse datasets and achieves high-accuracy, zero-shot predictions — eliminating the need for task-specific fine-tuning and marking a major leap forward in forecasting technology.</p><h4>What is a Decoder?</h4><p>At the core of TimesFM lies the decoder architecture, a key component also used in modern Large Language Models (LLMs) like GPT. A decoder is a neural network designed for generative tasks, where it produces new sequences based on a given context.</p><p>In language models, the decoder takes a sequence of words and predicts the next one in an autoregressive manner — generating text step by step. TimesFM applies this same concept to time-series data. Instead of predicting the next word, it predicts the next value or segment in a numerical sequence. Essentially, the model learns the “grammar of time” — understanding patterns, rhythms, and trends — to generate accurate forecasts.</p><h4>What is TimesFM?</h4><p>TimesFM is a decoder-only foundation model for time-series forecasting, inspired by the architecture of large language models like GPT-3. Unlike traditional models, it predicts future values directly from past data without requiring an encoder.</p><p>The model processes time-series data in patches — segments of sequential values — a technique that improves efficiency and helps capture long-term temporal patterns.</p><p>Designed as a general-purpose, zero-shot forecaster, TimesFM can deliver accurate predictions for entirely new datasets without any fine-tuning, marking a major shift from traditional, task-specific forecasting approaches.</p><h4>Architecture Overview</h4><p>TimesFM 1.0 is a decoder-only transformer model consisting of approximately 200 million parameters, trained on a massive pretraining dataset containing over 100 billion real-world time points. This large-scale training enables the model to deliver highly accurate forecasts on unseen datasets without any additional fine-tuning.</p><p>The model is designed for univariate time-series forecasting, where it predicts the future values of a single variable using only its past observations. TimesFM handles context lengths up to 512 time points and supports any forecast horizon, with an optional frequency indicator input to incorporate time granularity information.</p><p>TimesFM’s architecture combines a scalable transformer backbone, efficient patch-based processing, and robust masking techniques to achieve zero-shot generalization across diverse time-series datasets — setting a new standard for foundation models in forecasting.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K9VTwl40dYG_V8O55eCkQg.jpeg" /><figcaption>Image source: <a href="https://research.google/blog/a-decoder-only-foundation-model-for-time-series-forecasting/">A decoder-only foundation model for time-series forecasting</a></figcaption></figure><p>In TimesFM, the input time-series is first divided into fixed-length input patches. Each patch is then transformed into a vector using a residual block, aligning it with the model’s transformer dimensions. Positional encodings are added to these vectors to preserve the temporal order before they are passed through the stacked transformer layers.</p><p>Within the transformer, SA (self-attention) represents multi-head causal attention, which ensures the model only attends to past and present data, while FFN refers to the feed-forward (fully connected) layers that refine the learned representations.</p><p>Finally, the output tokens are processed through another residual block to produce an output of length output_patch_len, representing the model’s forecast for the next time window beyond the last observed patch.</p><p>Below is a concise overview of its key architectural elements:</p><ul><li><strong>Decoder-Only Design:</strong><br>TimesFM utilizes a decoder-only setup, making it inherently capable of generating future sequences from past context. This structure enables it to manage input sequences of varying lengths with flexibility and efficiency.</li><li><strong>Input Patching:</strong><br>The time-series data is divided into contiguous, non-overlapping patches (tokens), which pass through residual blocks that transform each patch into a vector representation. These vectors are then processed by the transformer layers for deeper temporal understanding.</li><li><strong>Stacked Transformer Layers:</strong><br>Multiple transformer layers equipped with multi-head self-attention allow each token to reference others in the sequence. TimesFM employs causal attention, ensuring that predictions are influenced only by past and present information — never by future data.</li><li><strong>Extended Output Patches:</strong><br>A key innovation in TimesFM is its ability to generate longer output patches than the input patches. Unlike traditional LLMs that produce outputs token by token, TimesFM predicts entire segments of future data at once, enabling faster inference and more accurate long-term forecasts.</li><li><strong>Patch Masking:</strong><br>To prevent overfitting and enhance generalization, TimesFM applies a patch masking strategy during training. This allows the model to adapt seamlessly to different context lengths during inference, ensuring strong and consistent performance across diverse time-series datasets.</li></ul><p><strong>Model Parameters (Hyperparameters)<br></strong>These are the key tunable settings that define the model’s behavior and impact its forecasting performance:</p><ul><li><strong>model_dim:</strong> Dimensionality of the input and output vectors.</li><li><strong>input_patch_len (p):</strong> Length of each input patch.</li><li><strong>output_patch_len (h):</strong> Number of future time steps predicted per generation step.</li><li><strong>num_heads:</strong> Number of attention heads in the multi-head attention mechanism.</li><li><strong>num_layers (nl):</strong> Number of stacked transformer layers.</li><li><strong>context length (L):</strong> Length of historical data used for forecasting.</li><li><strong>horizon length (H):</strong> Length of the forecast horizon.</li><li><strong>Number of input tokens (N):</strong> Representing the number of patches derived from the input sequence. Each token is then processed through transformer layers for contextual learning.</li></ul><p><strong>Model Components<br></strong>The core architecture of TimesFM is composed of the following fundamental components:</p><ul><li><strong>Residual Blocks: </strong>Used to preprocess input and output patches, ensuring stable gradient flow and efficient feature extraction.</li><li><strong>Stacked Transformer Layers: </strong>The central component of the model, enabling rich temporal representation learning through self-attention.</li><li><strong>Input Tokens (tj): </strong>Derived from processed patches, these tokens are generated using residual blocks and positional encodings.</li><li><strong>Output Tokens (oj): </strong>Generated by the stacked transformer layers, these tokens are used to predict the corresponding output patches.</li><li><strong>Patch Mask (m1:L): </strong>Applied to ignore certain portions of the input sequence during processing, helping the model generalize to varying context lengths.</li><li><strong>Loss Function</strong>: During training, TimesFM minimizes the Mean Squared Error (MSE) between predicted and actual future values.</li></ul><h3>Experimenting with TimesFM model</h3><p>This section guides the setup of the TimesFM model for time-series forecasting, starting with zero-shot forecasting to demonstrate predictions on unseen datasets without prior training. It also covers fine-tuning for specific datasets to enhance performance.</p><p>TimesFM’s performance is compared with other approaches, including statistical models (e.g., AutoETS), machine learning models (e.g., Random Forest, XGBoost, LGBM), and other foundational models such as TimeGPT, highlighting its unique capabilities and advantages.</p><p>We will also use uv, a modern and fast Python package manager (instead of pip), to set up our environment and handle dependencies.</p><h4>Installing uv</h4><p>uv simplifies dependency management, virtual environments, and running scripts.</p><ul><li><strong>For Windows:</strong></li></ul><pre>powershell -ExecutionPolicy ByPass -c &quot;irm https://astral.sh/uv/install.ps1 | iex&quot;<br>set Path=C:\Users\Codem\.local\bin;%Path%</pre><ul><li><strong>For Linux / Mac:</strong></li></ul><pre>curl -LsSf https://astral.sh/uv/install.sh | sh</pre><p>Refer to the official website for detailed installation instructions.</p><p><a href="https://docs.astral.sh/uv/getting-started/installation/">Installation | uv</a></p><h4>Installing the dependencies</h4><ul><li>Initialize a uv project by executing the following command.</li></ul><pre>uv init timesfm_demo<br>cd timesfm_demo</pre><ul><li>Create and activate a virtual environment by executing the following command.</li></ul><pre>uv venv<br>source .venv/bin/activate # for linux<br>.venv\Scripts\activate    # for windows</pre><blockquote>The official PyPI package for TimesFM provides older versions (e.g., timesfm-2.0–500m and timesfm-1.0–200m). To access the latest version (timesfm-2.5–200m-pytorch), it is recommended to install the library directly from GitHub. This ensures access to the most recent features, improvements, and bug fixes.</blockquote><ul><li>Clone the github repository and move to the folder using the following command.</li></ul><pre>git clone https://github.com/google-research/timesfm.git<br>cd timesfm</pre><ul><li>Install timesfm by executing the following command.</li></ul><pre>uv pip install -e .[torch]</pre><h3>Handson 1: Zero-Shot Forecasting</h3><p>Zero-shot forecasting allows TimesFM to generate accurate predictions on unseen datasets without any prior training or fine-tuning. Leveraging its pre-trained knowledge on diverse time-series data, the model can identify trends, seasonality, and temporal patterns directly from historical inputs.</p><p>In this demo, using a synthetic bike rental dataset with trend, seasonality, and noise, TimesFM was able to accurately forecast the next 90 days. Despite having no prior exposure to this dataset, the model effectively captured the underlying temporal patterns, demonstrating the power of zero-shot forecasting.</p><ul><li>Install matplotlib library for graph based visualization using uv.</li></ul><pre>uv add matplotlib</pre><ul><li>Create a file named bike_rental_forecast.py and add the following code to it.</li></ul><pre>import torch<br>import numpy as np<br>import pandas as pd<br>import matplotlib.pyplot as plt<br>import timesfm<br><br>torch.set_float32_matmul_precision(&quot;high&quot;)<br><br># --- 1. Generate Synthetic Bike Rental Data ---<br>def create_bike_rental_data():<br>    dates = pd.date_range(start=&quot;2020-01-01&quot;, end=&quot;2023-12-31&quot;, freq=&quot;D&quot;)<br>    n_days = len(dates)<br><br>    trend = np.linspace(start=150, stop=350, num=n_days)<br>    yearly_seasonality = 180 * (1 + np.sin(2 * np.pi * dates.dayofyear / 365.25 - np.pi/2))<br>    weekly_seasonality = 120 * (dates.dayofweek &gt;= 5)<br>    noise = np.random.normal(0, 40, n_days)<br>    rentals = trend + yearly_seasonality + weekly_seasonality + noise<br>    rentals = np.maximum(0, rentals).astype(int)  # Ensure no negative rentals<br>    temp = 15 + 10 * np.sin(2 * np.pi * dates.dayofyear / 365.25 - np.pi/2) + np.random.randn(n_days) * 2<br>    is_weekend = (dates.dayofweek &gt;= 5).astype(int)<br><br>    df = pd.DataFrame({<br>        &#39;rentals&#39;: rentals,<br>        &#39;temperature&#39;: temp,<br>        &#39;is_weekend&#39;: is_weekend<br>    }, index=dates)<br><br>    return df<br><br># --- 2. Prepare Data ---<br>rental_df = create_bike_rental_data()<br>time_series_data = rental_df[&#39;rentals&#39;].values<br>horizon_len = 90  # Forecast the next 90 days<br>historical_data = time_series_data[:-horizon_len]<br>true_future_values = time_series_data[-horizon_len:]<br><br># --- 3. Initialize TimesFM 2.5 PyTorch Model ---<br>model = timesfm.TimesFM_2p5_200M_torch.from_pretrained(&quot;google/timesfm-2.5-200m-pytorch&quot;)<br>model.compile(<br>    timesfm.ForecastConfig(<br>        max_context=1024,<br>        max_horizon=256,<br>        normalize_inputs=True,<br>        use_continuous_quantile_head=True,<br>        force_flip_invariance=True,<br>        infer_is_positive=True,<br>        fix_quantile_crossing=True,<br>    )<br>)<br><br># --- 4. Generate Forecast ---<br>point_forecast, quantile_forecast = model.forecast(<br>    horizon=horizon_len,<br>    inputs=[historical_data],  # Single time series as input<br>)<br>forecast_values = point_forecast[0]  # Extract the forecast for our single series<br><br># --- 5. Visualize the Results ---<br>dates = rental_df.index<br>plt.figure(figsize=(15, 7))<br>plt.plot(dates[:-horizon_len], historical_data, label=&quot;Historical Data&quot;, color=&quot;black&quot;)<br>plt.plot(dates[-horizon_len:], true_future_values, label=&quot;True Future Values&quot;, color=&quot;blue&quot;, linestyle=&#39;--&#39;)<br>plt.plot(dates[-horizon_len:], forecast_values, label=&quot;TimesFM 2.5 Forecast&quot;, color=&quot;red&quot;)<br>plt.fill_between(dates[-horizon_len:],quantile_forecast[0, :, 1], quantile_forecast[0, :, 9], alpha=0.2, color=&#39;red&#39;, label=&#39;80% Prediction Interval&#39;)<br>plt.legend()<br>plt.title(&quot;TimesFM 2.5 Zero-Shot Forecast for Daily Bike Rentals&quot;)<br>plt.xlabel(&quot;Date&quot;)<br>plt.ylabel(&quot;Number of Rentals&quot;)<br>plt.grid(True, linestyle=&#39;--&#39;, alpha=0.6)<br>plt.tight_layout()<br>plt.show()<br><br># --- 6. Calculate Metrics ---<br>rmse = np.sqrt(np.mean((true_future_values - forecast_values)**2))<br>mae = np.mean(np.abs(true_future_values - forecast_values))<br><br>print(f&quot;Zero-Shot Forecast RMSE: {rmse:.2f}&quot;)<br>print(f&quot;Zero-Shot Forecast MAE: {mae:.2f}&quot;)<br>print(f&quot;\nPoint forecast shape: {point_forecast.shape}&quot;)<br>print(f&quot;Quantile forecast shape: {quantile_forecast.shape}&quot;)</pre><p><strong>Code Summary</strong></p><ul><li>Generates synthetic daily bike rental data (2020–2023) using the custom function create_bike_rental_data().</li><li>Splits the dataset into historical and future segments for model evaluation.</li><li>Loads the TimesFM 2.5 model using timesfm.TimesFM_2p5_200M_torch.from_pretrained(&quot;google/timesfm-2.5-200m-pytorch&quot;) to fetch the latest pre-trained TimesFM model.</li><li>Configures the model with model.compile() using ForecastConfig to define forecast parameters (e.g., max_horizon, normalization, quantile options).</li><li>Performs forecasting via model.forecast(horizon, inputs=[historical_data]) which generates both point and quantile forecasts.</li><li>Visualizes trends and uncertainty using matplotlib.</li><li>Evaluates model performance using RMSE (Root Mean Square Error) and MAE (Mean Absolute Error)</li></ul><p><strong>Executing the code<br></strong>To visualize trends and obtain forecasted results with TimesFM, run the code using your Python environment.</p><pre>python bike_rental_forecast.py</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sma3723txIQ2vDMHRyskcA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dW-kx5RdNdVggW5YOCNxVw.png" /></figure><h3>Handson 2: Zero-Shot Forecasting with stats</h3><p>Improved version of the bike rental forecasting with various statistics.</p><pre>import torch<br>import numpy as np<br>from datetime import datetime, timedelta<br>import timesfm<br>import matplotlib.pyplot as plt<br><br># Initialize TimesFM model<br>print(&quot;Loading TimesFM model...&quot;)<br>torch.set_float32_matmul_precision(&quot;high&quot;)<br>model = timesfm.TimesFM_2p5_200M_torch.from_pretrained(&quot;google/timesfm-2.5-200m-pytorch&quot;)<br>model.compile(<br>    timesfm.ForecastConfig(<br>        max_context=1024,<br>        max_horizon=256,<br>        normalize_inputs=True,<br>        use_continuous_quantile_head=True,<br>        force_flip_invariance=True,<br>        infer_is_positive=True,<br>        fix_quantile_crossing=True,<br>    )<br>)<br><br>def generate_forecast(historical_data, horizon=90, start_date=None, target_column=&quot;Bike Rentals&quot;):<br><br>    if len(historical_data) &lt; 30:<br>        raise ValueError(&quot;Historical data must contain at least 30 data points&quot;)<br>    <br>    if horizon &gt; 256 or horizon &lt; 1:<br>        raise ValueError(&quot;Horizon must be between 1 and 256 days&quot;)<br>    <br>    historical_array = np.array(historical_data, dtype=np.float32)<br>    <br>    print(f&quot;Generating {horizon}-day forecast...&quot;)<br>    point_forecast, quantile_forecast = model.forecast(<br>        horizon=horizon,<br>        inputs=[historical_array],<br>    )<br>    <br>    # Extract forecasts<br>    forecast_values = point_forecast[0]<br>    quantiles = quantile_forecast[0]  # Shape: [horizon, 11]<br>    <br>    # Generate dates<br>    if start_date:<br>        if isinstance(start_date, str):<br>            start = datetime.strptime(start_date, &quot;%Y-%m-%d&quot;)<br>        else:<br>            start = start_date<br>    else:<br>        start = datetime.now() - timedelta(days=len(historical_data))<br>    <br>    historical_dates = [start + timedelta(days=i) for i in range(len(historical_data))]<br>    forecast_start = start + timedelta(days=len(historical_data))<br>    forecast_dates = [forecast_start + timedelta(days=i) for i in range(horizon)]<br>    <br>    # Calculate summary statistics<br>    historical_mean = np.mean(historical_data)<br>    historical_std = np.std(historical_data)<br>    forecast_mean = np.mean(forecast_values)<br>    forecast_std = np.std(forecast_values)<br>    <br>    # Print summary<br>    print(&quot;\n&quot; + &quot;=&quot;*70)<br>    print(f&quot;FORECAST SUMMARY: {target_column}&quot;)<br>    print(&quot;=&quot;*70)<br>    <br>    print(&quot;\nHistorical Data Statistics:&quot;)<br>    print(f&quot;  Period: {historical_dates[0].strftime(&#39;%Y-%m-%d&#39;)} to {historical_dates[-1].strftime(&#39;%Y-%m-%d&#39;)}&quot;)<br>    print(f&quot;  Count: {len(historical_data)} days&quot;)<br>    print(f&quot;  Mean: {historical_mean:.2f}&quot;)<br>    print(f&quot;  Std Dev: {historical_std:.2f}&quot;)<br>    print(f&quot;  Min: {np.min(historical_data):.2f}&quot;)<br>    print(f&quot;  Max: {np.max(historical_data):.2f}&quot;)<br>    <br>    print(&quot;\nForecast Statistics:&quot;)<br>    print(f&quot;  Period: {forecast_dates[0].strftime(&#39;%Y-%m-%d&#39;)} to {forecast_dates[-1].strftime(&#39;%Y-%m-%d&#39;)}&quot;)<br>    print(f&quot;  Count: {horizon} days&quot;)<br>    print(f&quot;  Mean: {forecast_mean:.2f}&quot;)<br>    print(f&quot;  Std Dev: {forecast_std:.2f}&quot;)<br>    print(f&quot;  Min: {np.min(forecast_values):.2f}&quot;)<br>    print(f&quot;  Max: {np.max(forecast_values):.2f}&quot;)<br>    <br>    print(&quot;\nTrend Analysis:&quot;)<br>    change_percent = ((forecast_mean - historical_mean) / historical_mean) * 100<br>    trend_direction = &quot;increasing&quot; if change_percent &gt; 0 else &quot;decreasing&quot;<br>    print(f&quot;  Direction: {trend_direction.upper()}&quot;)<br>    print(f&quot;  Change: {change_percent:+.2f}%&quot;)<br>    <br>    print(&quot;\nConfidence Intervals (Average):&quot;)<br>    print(f&quot;  80% Interval: [{np.mean(quantiles[:, 1]):.2f}, {np.mean(quantiles[:, 9]):.2f}]&quot;)<br>    print(f&quot;  50% Interval: [{np.mean(quantiles[:, 3]):.2f}, {np.mean(quantiles[:, 7]):.2f}]&quot;)<br>    <br>    print(&quot;\nFirst 10 Forecast Values:&quot;)<br>    for i in range(min(10, len(forecast_values))):<br>        date_str = forecast_dates[i].strftime(&#39;%Y-%m-%d&#39;)<br>        val = forecast_values[i]<br>        q10 = quantiles[i, 1]<br>        q90 = quantiles[i, 9]<br>        print(f&quot;  {date_str}: {val:.2f} (80% CI: [{q10:.2f}, {q90:.2f}])&quot;)<br>    <br>    if len(forecast_values) &gt; 10:<br>        print(f&quot;  ... and {len(forecast_values) - 10} more days&quot;)<br>    <br>    print(&quot;=&quot;*70 + &quot;\n&quot;)<br>    <br>    # Generate plot for historical and forecasted data<br>    plt.figure(figsize=(15, 7))<br>    plt.plot(historical_dates, historical_data, <br>             label=&quot;Historical Data&quot;, color=&quot;black&quot;, linewidth=2, marker=&#39;o&#39;, <br>             markersize=3, markevery=max(1, len(historical_data)//50))<br>    plt.plot(forecast_dates, forecast_values, <br>             label=&quot;Forecast&quot;, color=&quot;red&quot;, linewidth=2, marker=&#39;s&#39;,<br>             markersize=3, markevery=max(1, horizon//50))<br>    <br>    # Add confidence intervals<br>    plt.fill_between(<br>        forecast_dates,<br>        quantiles[:, 1],  # 10th percentile<br>        quantiles[:, 9],  # 90th percentile<br>        alpha=0.2,<br>        color=&#39;red&#39;,<br>        label=&#39;80% Prediction Interval&#39;<br>    )<br>    <br>    plt.fill_between(<br>        forecast_dates,<br>        quantiles[:, 3],  # 25th percentile<br>        quantiles[:, 7],  # 75th percentile<br>        alpha=0.3,<br>        color=&#39;red&#39;,<br>        label=&#39;50% Prediction Interval&#39;<br>    )<br>    <br>    plt.legend(loc=&#39;best&#39;, fontsize=11)<br>    plt.title(f&quot;TimesFM Forecast: {target_column}&quot;, fontsize=16, fontweight=&#39;bold&#39;)<br>    plt.xlabel(&quot;Date&quot;, fontsize=13)<br>    plt.ylabel(target_column, fontsize=13)<br>    plt.grid(True, linestyle=&#39;--&#39;, alpha=0.6)<br>    plt.xticks(rotation=45)<br>    plt.tight_layout()<br>    plt.show()<br>    <br>    # Return detailed results<br>    return {<br>        &quot;forecast_values&quot;: forecast_values.tolist(),<br>        &quot;forecast_dates&quot;: [d.strftime(&#39;%Y-%m-%d&#39;) for d in forecast_dates],<br>        &quot;quantiles&quot;: {<br>            &quot;q10&quot;: quantiles[:, 1].tolist(),<br>            &quot;q25&quot;: quantiles[:, 3].tolist(),<br>            &quot;q50&quot;: quantiles[:, 5].tolist(),<br>            &quot;q75&quot;: quantiles[:, 7].tolist(),<br>            &quot;q90&quot;: quantiles[:, 9].tolist()<br>        },<br>        &quot;historical_mean&quot;: float(historical_mean),<br>        &quot;forecast_mean&quot;: float(forecast_mean),<br>        &quot;trend&quot;: trend_direction,<br>        &quot;change_percent&quot;: float(change_percent)<br>    }<br><br><br>if __name__ == &quot;__main__&quot;:<br>    # Create synthetic data with trend and seasonality<br>    np.random.seed(42)<br>    days = 365<br>    trend = np.linspace(150, 300, days)<br>    seasonality = 50 * np.sin(np.linspace(0, 4*np.pi, days))<br>    noise = np.random.normal(0, 20, days)<br>    historical_data = trend + seasonality + noise<br>    historical_data = np.maximum(historical_data, 0)  # Ensure non-negative<br>    <br>    # Generate forecast<br>    results = generate_forecast(<br>        historical_data=historical_data.tolist(),<br>        horizon=90,<br>        start_date=&quot;2023-01-01&quot;,<br>        target_column=&quot;Bike Rentals&quot;<br>    )<br>    print(results)</pre><p><strong>Code Summary</strong></p><ul><li><strong>Model Initialization:</strong><br>The latest version of TimesFM (timesfm-2.5–200m-pytorch) is loaded and compiled with forecasting configurations such as context length, horizon, normalization, and quantile settings.</li><li><strong>Input Preparation:</strong><br>Historical time-series data is converted into a numerical array. The model supports flexible context lengths, but a minimum of 30 data points is recommended for robust forecasts.</li><li><strong>Forecast Generation:</strong><br>The model predicts the next values for the specified forecast horizon (up to 256 steps), producing both point forecasts and quantile estimates for uncertainty intervals.</li><li><strong>Analysis and Visualization:<br></strong>Summary statistics for historical and forecasted data are computed.<br>Trend direction and percent change are calculated to understand expected growth or decline.<br>Forecasts are visualized with confidence intervals to highlight prediction uncertainty.</li></ul><p><strong>Executing the code<br></strong>To visualize trends and obtain forecasted results with TimesFM, run the code using your Python environment.</p><pre>python bike_rental_forecast.py</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v3UoK-6MOlI_HXXgXXT9VA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sTw-W-JU8rixo2_pBj0mUQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RryMJjAqy429GE-5MPXHqw.png" /></figure><h3>Handson 3: Finetuning the model</h3><p>Although TimesFM’s zero-shot capabilities are already remarkable, fine-tuning the model on a specific dataset can further enhance forecasting accuracy.</p><p>Research shows that fine-tuning on as little as 10% of a dataset can achieve state-of-the-art performance, often outperforming models trained from scratch. While the current TimesFM library does not provide a high-level .finetune() API, the fine-tuning process can be implemented conceptually using frameworks like PyTorch or JAX, adjusting the model weights to better fit your dataset.</p><ul><li>Create a file named bike_rental_forecast_finetuning.py and add the following code to it.</li></ul><pre>import torch<br>import torch.nn as nn<br>import torch.optim as optim<br>from torch.utils.data import Dataset, DataLoader<br>import numpy as np<br>import timesfm<br>from datetime import datetime, timedelta<br>import matplotlib.pyplot as plt<br>from pathlib import Path<br><br># Initialize TimesFM model<br>print(&quot;Loading TimesFM model...&quot;)<br>torch.set_float32_matmul_precision(&quot;high&quot;)<br>model = timesfm.TimesFM_2p5_200M_torch.from_pretrained(&quot;google/timesfm-2.5-200m-pytorch&quot;)<br><br>model.compile(<br>    timesfm.ForecastConfig(<br>        max_context=1024,<br>        max_horizon=256,<br>        normalize_inputs=True,<br>        use_continuous_quantile_head=True,<br>        force_flip_invariance=True,<br>        infer_is_positive=True,<br>        fix_quantile_crossing=True,<br>    )<br>)<br>print(&quot;Model loaded successfully!\n&quot;)<br><br><br># Custom Dataset for fine-tuning<br>class TimeSeriesDataset(Dataset):<br>    def __init__(self, data, context_length=365, horizon=90):<br>        self.data = np.array(data, dtype=np.float32)<br>        self.context_length = context_length<br>        self.horizon = horizon<br>        <br>        # Create sliding windows<br>        self.samples = []<br>        for i in range(len(self.data) - context_length - horizon + 1):<br>            context = self.data[i:i + context_length]<br>            target = self.data[i + context_length:i + context_length + horizon]<br>            self.samples.append((context, target))<br>    <br>    def __len__(self):<br>        return len(self.samples)<br>    <br>    def __getitem__(self, idx):<br>        context, target = self.samples[idx]<br>        return torch.tensor(context), torch.tensor(target)<br><br><br>def prepare_finetuning_data(historical_data, batch_size=8, context_length=365, horizon=90):<br>    dataset = TimeSeriesDataset(historical_data, context_length, horizon)<br>    data_loader = DataLoader(<br>        dataset, <br>        batch_size=batch_size, <br>        shuffle=True,<br>        num_workers=0  # Set to 0 for compatibility<br>    )<br>    return data_loader<br><br><br>def finetune_model(model, train_data, num_epochs=10, learning_rate=1e-4, <br>                   context_length=365, horizon=90, batch_size=8, <br>                   save_path=&quot;finetuned_timesfm.pt&quot;):<br>    print(&quot;=&quot;*70)<br>    print(&quot;STARTING MODEL FINE-TUNING&quot;)<br>    print(&quot;=&quot;*70)<br>    print(f&quot;Training data points: {len(train_data)}&quot;)<br>    print(f&quot;Context length: {context_length}&quot;)<br>    print(f&quot;Forecast horizon: {horizon}&quot;)<br>    print(f&quot;Batch size: {batch_size}&quot;)<br>    print(f&quot;Learning rate: {learning_rate}&quot;)<br>    print(f&quot;Epochs: {num_epochs}\n&quot;)<br>    <br>    # Prepare data loader<br>    train_loader = prepare_finetuning_data(<br>        train_data, <br>        batch_size=batch_size,<br>        context_length=context_length,<br>        horizon=horizon<br>    )<br>    <br>    if len(train_loader) == 0:<br>        raise ValueError(f&quot;Not enough data for training. Need at least {context_length + horizon} points.&quot;)<br>    <br>    print(f&quot;Created {len(train_loader)} training batches\n&quot;)<br>    <br>    # Access the underlying PyTorch model. TimesFM wraps the actual model, we need to access it<br>    try:<br>        # Try to access the internal model<br>        if hasattr(model, &#39;model&#39;):<br>            pytorch_model = model.model<br>        elif hasattr(model, &#39;_model&#39;):<br>            pytorch_model = model._model<br>        else:<br>            # If we can&#39;t find the internal model, use the wrapper directly<br>            pytorch_model = model<br>        <br>        # Get trainable parameters<br>        trainable_params = [p for p in pytorch_model.parameters() if p.requires_grad]<br>        <br>        if len(trainable_params) == 0:<br>            print(&quot;Warning: No trainable parameters found. Attempting to unfreeze all parameters...&quot;)<br>            for param in pytorch_model.parameters():<br>                param.requires_grad = True<br>            trainable_params = list(pytorch_model.parameters())<br>        <br>        print(f&quot;Found {len(trainable_params)} trainable parameter groups\n&quot;)<br>        <br>    except Exception as e:<br>        print(f&quot;Warning: Could not access model parameters directly: {e}&quot;)<br>        print(&quot;TimesFM may not support direct fine-tuning through standard PyTorch methods.&quot;)<br>        print(&quot;Using the model in inference mode only.\n&quot;)<br>        return model, []<br>    <br>    optimizer = optim.Adam(trainable_params, lr=learning_rate)     # Create optimizer<br>    criterion = nn.MSELoss()    # Loss function<br>    training_losses = []    # Training loop<br>    <br>    for epoch in range(num_epochs):<br>        epoch_losses = []<br>        <br>        for batch_idx, (context, ground_truth_horizon) in enumerate(train_loader):<br>            try:<br>                # Note: TimesFM expects inputs as list of arrays<br>                predicted_horizon, _ = model.forecast(<br>                    horizon=horizon,<br>                    inputs=[context[i].numpy() for i in range(context.shape[0])]<br>                )                <br>                # Convert predictions to tensor<br>                predicted_tensor = torch.tensor(predicted_horizon, dtype=torch.float32, requires_grad=True)<br>                # Calculate loss<br>                loss = criterion(predicted_tensor, ground_truth_horizon)<br>                # Backward pass: update model weights<br>                optimizer.zero_grad()<br>                loss.backward()<br>                optimizer.step()<br>                <br>                epoch_losses.append(loss.item())<br>                <br>            except Exception as e:<br>                print(f&quot;Error during training batch {batch_idx}: {e}&quot;)<br>                print(&quot;Note: TimesFM may not support gradient-based fine-tuning.&quot;)<br>                print(&quot;The model will be used in inference-only mode.\n&quot;)<br>                return model, []<br>        <br>        avg_loss = np.mean(epoch_losses)<br>        training_losses.append(avg_loss)<br>        <br>        print(f&quot;Epoch {epoch+1}/{num_epochs}, Average Loss: {avg_loss:.6f}&quot;)<br>    <br>    print(f&quot;\nFine-tuning complete!&quot;)<br>    <br>    if len(training_losses) &gt; 0:<br>        print(f&quot;Final Loss: {training_losses[-1]:.6f}&quot;)<br>        print(f&quot;Saving fine-tuned model to: {save_path}&quot;)<br>        try:<br>            # Save the state dict of the internal model if possible<br>            if hasattr(model, &#39;model&#39;):<br>                model_state = model.model.state_dict()<br>            elif hasattr(model, &#39;_model&#39;):<br>                model_state = model._model.state_dict()<br>            else:<br>                model_state = {}<br>                <br>            torch.save({<br>                &#39;model_state_dict&#39;: model_state,<br>                &#39;optimizer_state_dict&#39;: optimizer.state_dict() if optimizer else None,<br>                &#39;training_losses&#39;: training_losses,<br>                &#39;config&#39;: {<br>                    &#39;context_length&#39;: context_length,<br>                    &#39;horizon&#39;: horizon,<br>                    &#39;num_epochs&#39;: num_epochs,<br>                    &#39;learning_rate&#39;: learning_rate<br>                }<br>            }, save_path)<br>            print(&quot;Model saved successfully!\n&quot;)<br>        except Exception as e:<br>            print(f&quot;Warning: Could not save model: {e}\n&quot;)<br>        <br>        # Plot training loss<br>        plt.figure(figsize=(10, 5))<br>        plt.plot(range(1, num_epochs + 1), training_losses, marker=&#39;o&#39;, linewidth=2)<br>        plt.title(&#39;Training Loss Over Epochs&#39;, fontsize=14, fontweight=&#39;bold&#39;)<br>        plt.xlabel(&#39;Epoch&#39;, fontsize=12)<br>        plt.ylabel(&#39;MSE Loss&#39;, fontsize=12)<br>        plt.grid(True, linestyle=&#39;--&#39;, alpha=0.6)<br>        plt.tight_layout()<br>        plt.show()<br>    else:<br>        print(&quot;Fine-tuning was not performed due to model limitations.&quot;)<br>        print(&quot;Using pre-trained model for inference.\n&quot;)<br>    <br>    return model, training_losses<br><br><br>def load_finetuned_model(model, checkpoint_path):<br>    print(f&quot;Loading fine-tuned model from: {checkpoint_path}&quot;)<br>    try:<br>        checkpoint = torch.load(checkpoint_path)<br>        <br>        # Try to load state dict into internal model<br>        if &#39;model_state_dict&#39; in checkpoint and checkpoint[&#39;model_state_dict&#39;]:<br>            if hasattr(model, &#39;model&#39;):<br>                model.model.load_state_dict(checkpoint[&#39;model_state_dict&#39;])<br>            elif hasattr(model, &#39;_model&#39;):<br>                model._model.load_state_dict(checkpoint[&#39;model_state_dict&#39;])<br>            else:<br>                print(&quot;Warning: Could not access internal model structure&quot;)<br>        <br>        print(&quot;Model loaded successfully!&quot;)<br>        <br>        if &#39;config&#39; in checkpoint:<br>            print(&quot;\nModel configuration:&quot;)<br>            for key, value in checkpoint[&#39;config&#39;].items():<br>                print(f&quot;  {key}: {value}&quot;)<br>    except Exception as e:<br>        print(f&quot;Warning: Could not load fine-tuned model: {e}&quot;)<br>        print(&quot;Using pre-trained base model instead.&quot;)<br>    <br>    return model<br><br><br>def generate_forecast(historical_data, horizon=90, start_date=None, target_column=&quot;Bike Rentals&quot;, <br>                     use_finetuned=False, finetuned_path=None):<br>    global model<br>    <br>    # Load fine-tuned model if requested<br>    if use_finetuned and finetuned_path:<br>        if Path(finetuned_path).exists():<br>            model = load_finetuned_model(model, finetuned_path)<br>        else:<br>            print(f&quot;Warning: Fine-tuned model not found at {finetuned_path}. Using base model.&quot;)<br>    <br>    # Validate input<br>    if len(historical_data) &lt; 30:<br>        raise ValueError(&quot;Historical data must contain at least 30 data points&quot;)<br>    <br>    if horizon &gt; 256 or horizon &lt; 1:<br>        raise ValueError(&quot;Horizon must be between 1 and 256 days&quot;)<br>    <br>    # Convert to numpy array<br>    historical_array = np.array(historical_data, dtype=np.float32)<br>    <br>    print(f&quot;Generating {horizon}-day forecast...&quot;)<br>    <br>    # Generate forecast (TimesFM handles inference mode internally)<br>    point_forecast, quantile_forecast = model.forecast(<br>        horizon=horizon,<br>        inputs=[historical_array],<br>    )<br>    <br>    # Extract forecasts<br>    forecast_values = point_forecast[0]<br>    quantiles = quantile_forecast[0]  # Shape: [horizon, 11]<br>    <br>    # Generate dates<br>    if start_date:<br>        if isinstance(start_date, str):<br>            start = datetime.strptime(start_date, &quot;%Y-%m-%d&quot;)<br>        else:<br>            start = start_date<br>    else:<br>        start = datetime.now() - timedelta(days=len(historical_data))<br>    <br>    historical_dates = [start + timedelta(days=i) for i in range(len(historical_data))]<br>    forecast_start = start + timedelta(days=len(historical_data))<br>    forecast_dates = [forecast_start + timedelta(days=i) for i in range(horizon)]<br>    <br>    # Calculate summary statistics<br>    historical_mean = np.mean(historical_data)<br>    historical_std = np.std(historical_data)<br>    forecast_mean = np.mean(forecast_values)<br>    forecast_std = np.std(forecast_values)<br>    <br>    # Print summary<br>    print(&quot;\n&quot; + &quot;=&quot;*70)<br>    print(f&quot;FORECAST SUMMARY: {target_column}&quot;)<br>    print(&quot;=&quot;*70)<br>    <br>    print(&quot;\nHistorical Data Statistics:&quot;)<br>    print(f&quot;  Period: {historical_dates[0].strftime(&#39;%Y-%m-%d&#39;)} to {historical_dates[-1].strftime(&#39;%Y-%m-%d&#39;)}&quot;)<br>    print(f&quot;  Count: {len(historical_data)} days&quot;)<br>    print(f&quot;  Mean: {historical_mean:.2f}&quot;)<br>    print(f&quot;  Std Dev: {historical_std:.2f}&quot;)<br>    print(f&quot;  Min: {np.min(historical_data):.2f}&quot;)<br>    print(f&quot;  Max: {np.max(historical_data):.2f}&quot;)<br>    <br>    print(&quot;\nForecast Statistics:&quot;)<br>    print(f&quot;  Period: {forecast_dates[0].strftime(&#39;%Y-%m-%d&#39;)} to {forecast_dates[-1].strftime(&#39;%Y-%m-%d&#39;)}&quot;)<br>    print(f&quot;  Count: {horizon} days&quot;)<br>    print(f&quot;  Mean: {forecast_mean:.2f}&quot;)<br>    print(f&quot;  Std Dev: {forecast_std:.2f}&quot;)<br>    print(f&quot;  Min: {np.min(forecast_values):.2f}&quot;)<br>    print(f&quot;  Max: {np.max(forecast_values):.2f}&quot;)<br>    <br>    print(&quot;\nTrend Analysis:&quot;)<br>    change_percent = ((forecast_mean - historical_mean) / historical_mean) * 100<br>    trend_direction = &quot;increasing&quot; if change_percent &gt; 0 else &quot;decreasing&quot;<br>    print(f&quot;  Direction: {trend_direction.upper()}&quot;)<br>    print(f&quot;  Change: {change_percent:+.2f}%&quot;)<br>    <br>    print(&quot;\nConfidence Intervals (Average):&quot;)<br>    print(f&quot;  80% Interval: [{np.mean(quantiles[:, 1]):.2f}, {np.mean(quantiles[:, 9]):.2f}]&quot;)<br>    print(f&quot;  50% Interval: [{np.mean(quantiles[:, 3]):.2f}, {np.mean(quantiles[:, 7]):.2f}]&quot;)<br>    <br>    print(&quot;\nFirst 10 Forecast Values:&quot;)<br>    for i in range(min(10, len(forecast_values))):<br>        date_str = forecast_dates[i].strftime(&#39;%Y-%m-%d&#39;)<br>        val = forecast_values[i]<br>        q10 = quantiles[i, 1]<br>        q90 = quantiles[i, 9]<br>        print(f&quot;  {date_str}: {val:.2f} (80% CI: [{q10:.2f}, {q90:.2f}])&quot;)<br>    <br>    if len(forecast_values) &gt; 10:<br>        print(f&quot;  ... and {len(forecast_values) - 10} more days&quot;)<br>    <br>    print(&quot;=&quot;*70 + &quot;\n&quot;)<br>    <br>    # Generate plot<br>    plt.figure(figsize=(15, 7))<br>    plt.plot(historical_dates, historical_data, <br>             label=&quot;Historical Data&quot;, color=&quot;black&quot;, linewidth=2, marker=&#39;o&#39;, <br>             markersize=3, markevery=max(1, len(historical_data)//50))<br>    plt.plot(forecast_dates, forecast_values, <br>             label=&quot;Forecast&quot;, color=&quot;red&quot;, linewidth=2, marker=&#39;s&#39;,<br>             markersize=3, markevery=max(1, horizon//50))<br>    <br>    # Add confidence intervals<br>    plt.fill_between(<br>        forecast_dates,<br>        quantiles[:, 1],  # 10th percentile<br>        quantiles[:, 9],  # 90th percentile<br>        alpha=0.2,<br>        color=&#39;red&#39;,<br>        label=&#39;80% Prediction Interval&#39;<br>    )<br>    <br>    plt.fill_between(<br>        forecast_dates,<br>        quantiles[:, 3],  # 25th percentile<br>        quantiles[:, 7],  # 75th percentile<br>        alpha=0.3,<br>        color=&#39;red&#39;,<br>        label=&#39;50% Prediction Interval&#39;<br>    )<br>    <br>    plt.legend(loc=&#39;best&#39;, fontsize=11)<br>    model_type = &quot;Fine-tuned&quot; if use_finetuned else &quot;Base&quot;<br>    plt.title(f&quot;TimesFM Forecast ({model_type}): {target_column}&quot;, fontsize=16, fontweight=&#39;bold&#39;)<br>    plt.xlabel(&quot;Date&quot;, fontsize=13)<br>    plt.ylabel(target_column, fontsize=13)<br>    plt.grid(True, linestyle=&#39;--&#39;, alpha=0.6)<br>    plt.xticks(rotation=45)<br>    plt.tight_layout()<br>    plt.show()<br>    <br>    # Return detailed results<br>    return {<br>        &quot;forecast_values&quot;: forecast_values.tolist(),<br>        &quot;forecast_dates&quot;: [d.strftime(&#39;%Y-%m-%d&#39;) for d in forecast_dates],<br>        &quot;quantiles&quot;: {<br>            &quot;q10&quot;: quantiles[:, 1].tolist(),<br>            &quot;q25&quot;: quantiles[:, 3].tolist(),<br>            &quot;q50&quot;: quantiles[:, 5].tolist(),<br>            &quot;q75&quot;: quantiles[:, 7].tolist(),<br>            &quot;q90&quot;: quantiles[:, 9].tolist()<br>        },<br>        &quot;historical_mean&quot;: float(historical_mean),<br>        &quot;forecast_mean&quot;: float(forecast_mean),<br>        &quot;trend&quot;: trend_direction,<br>        &quot;change_percent&quot;: float(change_percent)<br>    }<br><br><br>if __name__ == &quot;__main__&quot;:<br>    # Create synthetic data with trend and seasonality (2 years of data)<br>    np.random.seed(42)<br>    days = 730<br>    trend = np.linspace(150, 350, days)<br>    seasonality = 50 * np.sin(np.linspace(0, 8*np.pi, days))<br>    noise = np.random.normal(0, 20, days)<br>    historical_data = trend + seasonality + noise<br>    historical_data = np.maximum(historical_data, 0)  # Ensure non-negative<br>    <br>    print(&quot;OPTION 1: Use base pre-trained model (skip fine-tuning)&quot;)<br>    print(&quot;OPTION 2: Fine-tune model on your data first&quot;)<br>    print(&quot;\nChoosing OPTION 1 for this example...\n&quot;)<br>    <br>    # OPTION 1: Direct forecasting with base model<br>    print(&quot;Generating forecast with BASE model...&quot;)<br>    results_base = generate_forecast(<br>        historical_data=historical_data[-365:].tolist(),  # Use last year<br>        horizon=90,<br>        start_date=&quot;2023-01-01&quot;,<br>        target_column=&quot;Bike Rentals&quot;<br>    )<br>    <br>    # OPTION 2: Fine-tune model then forecast<br>    print(&quot;\n&quot; + &quot;=&quot;*70)<br>    print(&quot;FINE-TUNING MODEL ON TRAINING DATA&quot;)<br>    print(&quot;=&quot;*70 + &quot;\n&quot;)<br>    <br>    # Fine-tune on first 18 months of data<br>    finetuned_model, losses = finetune_model(<br>        model=model,<br>        train_data=historical_data[:540].tolist(),<br>        num_epochs=5,<br>        learning_rate=1e-4,<br>        context_length=180,<br>        horizon=30,<br>        batch_size=4,<br>        save_path=&quot;finetuned_timesfm.pt&quot;<br>    )<br>    <br>    # Generate forecast with fine-tuned model<br>    print(&quot;\nGenerating forecast with FINE-TUNED model...&quot;)<br>    results_finetuned = generate_forecast(<br>        historical_data=historical_data[-365:].tolist(),<br>        horizon=90,<br>        start_date=&quot;2023-01-01&quot;,<br>        target_column=&quot;Bike Rentals&quot;,<br>        use_finetuned=True,<br>        finetuned_path=&quot;finetuned_timesfm.pt&quot;<br>    )<br>    <br>    print(&quot;\n&quot; + &quot;=&quot;*70)<br>    print(&quot;FORECAST COMPLETE!&quot;)<br>    print(&quot;=&quot;*70)</pre><p><strong>Code Summary</strong></p><ul><li><strong>Model Initialization:<br></strong>Loads the pre-trained TimesFM model (timesfm-2.5-200m-pytorch) using PyTorch.<br>Configures forecasting parameters like max_context, max_horizon, normalization, quantile settings, and invariance options.</li><li><strong>Custom Dataset Creation:<br></strong>Defines TimeSeriesDataset class to create sliding windows of context and target sequences from historical data.<br>Converts data into PyTorch tensors for training.</li><li><strong>Data Preparation:<br></strong>prepare_finetuning_data() generates a DataLoader for batching historical data for fine-tuning.</li><li><strong>Fine-Tuning Function (</strong><strong>finetune_model):<br></strong>Prepares training batches and sets up optimizer (Adam) and loss function (MSELoss).<br>Attempts to access and unfreeze internal model parameters for gradient updates.<br>Performs multiple epochs of training, updating model weights based on prediction error.<br>Saves fine-tuned model state and plots training loss over epochs.<br>Handles exceptions if the model does not support gradient-based fine-tuning (fallback to inference mode).</li><li><strong>Loading Fine-Tuned Model:<br></strong>load_finetuned_model() loads saved model checkpoints and updates internal model weights for inference.</li><li><strong>Forecast Generation (</strong><strong>generate_forecast):<br></strong>Generates point forecasts and quantile-based confidence intervals for a specified horizon.<br>Computes summary statistics (mean, std, min, max) and trend direction.<br>Produces a plot showing historical data, forecasts, and confidence intervals.<br>Supports using either the base pre-trained model or fine-tuned model.</li></ul><p><strong>Executing the code<br></strong>To visualize trends and obtain forecasted results using the fine-tuned model, run the code using your Python environment.</p><pre>python bike_rental_forecast_finetuning.py</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LEx1RSw_25Cg8QRKu8veLQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZEkalYgasfXvtVzwmhqgww.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tfAym6L5VFFPd-Qjc184Lw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MJdnOmmXRbz3q7Vn7WSHlQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ksdy_ZfDt7djauWvtgqbEw.png" /></figure><h3>Handson 4: Model Comparison</h3><p>This section compares the performance of the TimesFM model with other approaches, including statistical models (e.g., AutoETS), machine learning models (e.g., Random Forest, XGBoost, LGBM), and foundational models such as TimeGPT.</p><p>The dataset used for this analysis is sourced from Kaggle: Monthly Gold Prices (1979–2021), which provides historic gold prices across 18 different countries.</p><p><a href="https://www.kaggle.com/datasets/odins0n/monthly-gold-prices">Monthly Gold Prices (1979-2021)</a></p><h4>Setting up the environment</h4><p>For this hands-on guide, we will be using Google Colab with a T4 GPU.</p><ol><li>Open <a href="https://colab.research.google.com/">Google Colaboratory</a> and sign in with your Google account.</li><li>Create a new notebook by clicking on + New Notebook.</li><li>Navigate to Runtime → Change runtime type.</li></ol><ul><li>Set Hardware Accelerator to GPU.</li><li>Choose T4 GPU (recommended for this tutorial).</li><li>Click Save.</li></ul><h4>Reading the data</h4><p>Let&#39;s begins by reading the dataset from the CSV file containing monthly gold price records.</p><pre>import pandas as pd<br>df = pd.read_csv(&quot;/content/1979-2021.csv&quot;)<br>df[&#39;Date&#39;] = pd.to_datetime(df[&#39;Date&#39;])<br>df = df.set_index(&#39;Date&#39;).resample(&#39;MS&#39;).mean()<br>df = df.reset_index() # Reset index to have &#39;Date&#39; as a column again<br>print(df.head())</pre><h4>Visualizing the dataset</h4><p>Visualize the dataset using seaborn.</p><pre>import matplotlib.pyplot as plt<br>import seaborn as sns<br>sns.set(style=&quot;darkgrid&quot;)<br>plt.figure(figsize=(10, 6))<br>sns.lineplot(x=&quot;Date&quot;, y=&#39;India(INR)&#39;, data=df, color=&#39;green&#39;)<br>plt.title(&#39;Monthly Gold Prices Over Time&#39;)<br>plt.xlabel(&#39;Date&#39;)<br>plt.ylabel(&#39;Gold Price in INR&#39;)<br>plt.show()</pre><h4>Time-series decomposition visualization — trend, seasonality and residuals</h4><p>Performs a time-series decomposition on India’s monthly gold prices and visually separates the data into trend (overall direction), seasonality (repeating pattern) and residuals (random noise) helping you understand the underlying patterns in gold price movements over time.</p><pre>from statsmodels.tsa.seasonal import seasonal_decompose<br><br>df.set_index(&quot;Date&quot;, inplace=True)<br>result = seasonal_decompose(df[&#39;India(INR)&#39;])<br>fig, (ax1, ax2, ax3, ax4) = plt.subplots(4, 1, figsize=(10, 12))<br>result.observed.plot(ax=ax1, color=&#39;green&#39;)<br>ax1.set_ylabel(&#39;Observed&#39;)<br>result.trend.plot(ax=ax2, color=&#39;green&#39;)<br>ax2.set_ylabel(&#39;Trend&#39;)<br>result.seasonal.plot(ax=ax3, color=&#39;green&#39;)<br>ax3.set_ylabel(&#39;Seasonal&#39;)<br>result.resid.plot(ax=ax4, color=&#39;green&#39;)<br>ax4.set_ylabel(&#39;Residual&#39;)<br><br>plt.tight_layout()<br>plt.show()<br>df.reset_index(inplace=True)</pre><h4>Arranging the Data in Format as Required by the Models</h4><pre>df = pd.DataFrame({&#39;unique_id&#39;:[1]*len(df),&#39;ds&#39;: df[&quot;Date&quot;], &quot;y&quot;:df[&#39;India(INR)&#39;]})<br>train_df = df[df[&#39;ds&#39;] &lt;= &#39;31-07-2019&#39;]<br>test_df = df[df[&#39;ds&#39;] &gt; &#39;31-07-2019&#39;]</pre><h4>1. Statistical Modeling</h4><p>StatsForecast is an open-source library designed for fast and scalable statistical time-series forecasting. It provides a wide range of classical models such as ARIMA, ETS (Exponential Smoothing), Theta, and Seasonal Naive, implemented with high efficiency using Numba for parallel processing.</p><pre>import pandas as pd<br>from statsforecast import StatsForecast<br>from statsforecast.models import AutoARIMA, AutoETS<br><br># Define the AutoARIMA model<br>autoarima = AutoARIMA(season_length=12)<br># Define the AutoETS model<br>autoets = AutoETS(season_length=12)<br><br># Create StatsForecast object with AutoARIMA<br>statforecast = StatsForecast(<br>    models=[autoarima, autoets],<br>    freq=&#39;MS&#39;,<br>    n_jobs=-1)<br>statforecast.fit(train_df)<br><br># Generate forecasts for 24 periods ahead<br>sf_forecast = statforecast.forecast(df=train_df, h=24, fitted=True)<br>sf_forecast = sf_forecast.reset_index()<br>print(&quot;StatsForecast:&quot;, sf_forecast)</pre><h4>2. MLForecast</h4><p>MLForecast is a machine learning-based framework for time-series forecasting that leverages models such as Random Forest, XGBoost, and LightGBM.</p><pre>from mlforecast import MLForecast<br>from mlforecast.target_transforms import AutoDifferences<br>from numba import njit<br>import lightgbm as lgb<br>import xgboost as xgb<br>from sklearn.ensemble import RandomForestRegressor<br>from statsmodels.tsa.seasonal import seasonal_decompose<br>from mlforecast import MLForecast<br>from mlforecast.lag_transforms import (<br> RollingMean, RollingStd, RollingMin, RollingMax, RollingQuantile,<br> SeasonalRollingMean, SeasonalRollingStd, SeasonalRollingMin,<br> SeasonalRollingMax, SeasonalRollingQuantile,<br> ExpandingMean<br>)<br><br>models = [lgb.LGBMRegressor(verbosity=-1),<br> xgb.XGBRegressor(),<br> RandomForestRegressor(random_state=0),<br>]<br>fcst = MLForecast(<br>    models=models, # List of models to be used for forecasting<br>    freq=&#39;MS&#39;, # Monthly frequency, starting at the beginning of each month<br>    lags=[1,3,5,7,12], # Lag features: values from 1, 3, 5, 7, and 12 time steps ago<br>    lag_transforms={<br>        1: [ # Transformations applied to lag 1<br>            RollingMean(window_size=3),<br>            RollingStd(window_size=3),<br>            RollingMin(window_size=3),<br>            RollingMax(window_size=3),<br>            RollingQuantile(p=0.5, window_size=3),<br>            ExpandingMean()<br>        ],<br>        6:[ # Transformations applied to lag 6<br>            RollingMean(window_size=6),<br>            RollingStd(window_size=6),<br>            RollingMin(window_size=6),<br>            RollingMax(window_size=6),<br>            RollingQuantile(p=0.5, window_size=6),<br>        ],<br>        12: [ # Transformations applied to lag 12 (likely for yearly seasonality)<br>            SeasonalRollingMean(season_length=12, window_size=3),<br>            SeasonalRollingStd(season_length=12, window_size=3),<br>            SeasonalRollingMin(season_length=12, window_size=3),<br>            SeasonalRollingMax(season_length=12, window_size=3),<br>            SeasonalRollingQuantile(p=0.5, season_length=12, window_size=3)<br>        ]<br>    },<br>    date_features=[&#39;year&#39;, &#39;month&#39;, &#39;quarter&#39;], # Extract year, month, and quarter from the date as features<br>    target_transforms=[AutoDifferences(max_diffs=3)]<br>)<br><br>fcst.fit(train_df)<br>ml_forecast = fcst.predict(len(test_df))<br>print(&quot;MLForecast:&quot;, ml_forecast)</pre><h4>3. TimeGPT</h4><p>TimeGPT is a large foundational model for time-series forecasting developed by Nixtla, inspired by the architecture and scaling principles of large language models. Unlike traditional forecasting methods that require dataset-specific training, TimeGPT can perform zero-shot forecasting — generating accurate predictions on unseen time-series data without any fine-tuning.</p><pre>from nixtla import NixtlaClient<br>nixtla_client = NixtlaClient(api_key = &#39;nixak-PJPGa3MxJ3VdxZhKvylOcu2XHBtZ8ssIykc7wzoLKB0sVcDMnHoD53kGpvuJGk9e5lj83KojwKaljmcK&#39;)<br>timegpt_forecast = nixtla_client.forecast(df=train_df, h=24, freq=&#39;MS&#39;)<br>print(&quot;TimeGPT: &quot;, timegpt_forecast)</pre><h4>4. TimesFM</h4><p>Now lets use TimesFM to generate forecasts for the dataset, leveraging its zero-shot or fine-tuned capabilities to predict future time-series values.</p><pre>import torch<br>import numpy as np<br>import pandas as pd<br>import timesfm<br><br>torch.set_float32_matmul_precision(&quot;high&quot;)<br>model = timesfm.TimesFM_2p5_200M_torch.from_pretrained(&quot;google/timesfm-2.5-200m-pytorch&quot;)<br><br>model.compile(<br>    timesfm.ForecastConfig(<br>        max_context=1024,                # Maximum context length<br>        max_horizon=256,                 # Maximum forecast horizon<br>        normalize_inputs=True,           # Normalize time series before forecasting<br>        use_continuous_quantile_head=True,<br>        force_flip_invariance=True,<br>        infer_is_positive=True,<br>        fix_quantile_crossing=True,<br>    )<br>)<br>H = 24  # Forecast horizon<br><br>series_list = []<br>for _, g in train_df.groupby(&quot;unique_id&quot;):<br>    series_list.append(g[&quot;y&quot;].values.astype(np.float32))<br><br>point_forecast, quantile_forecast = model.forecast(<br>    horizon=H,<br>    inputs=series_list,<br>)<br><br>forecasts = []<br>for (uid, group), preds in zip(train_df.groupby(&quot;unique_id&quot;), point_forecast):<br>    # Get last date and extend future timestamps<br>    last_date = group[&quot;ds&quot;].iloc[-1]<br>    future_dates = pd.date_range(start=last_date, periods=H + 1, freq=&quot;MS&quot;)[1:]<br>    df_pred = pd.DataFrame({<br>        &quot;unique_id&quot;: uid,<br>        &quot;ds&quot;: future_dates,<br>        &quot;timesfm&quot;: preds<br>    })<br>    forecasts.append(df_pred)<br><br>timesfm_forecast = pd.concat(forecasts, ignore_index=True)<br>print(&quot;TimesEM: &quot;, timesfm_forecast)</pre><h4>Convert ‘ds’ to datetime in all DataFrames if necessary</h4><pre># Assuming the DataFrames have a common column &#39;ds&#39; for the dates<br>sf_forecast[&#39;ds&#39;] = pd.to_datetime(sf_forecast[&#39;ds&#39;])<br>ml_forecast[&#39;ds&#39;] = pd.to_datetime(ml_forecast[&#39;ds&#39;])<br>timegpt_forecast[&#39;ds&#39;] = pd.to_datetime(timegpt_forecast[&#39;ds&#39;])<br>timesfm_forecast[&#39;ds&#39;] = pd.to_datetime(timesfm_forecast[&#39;ds&#39;])<br>test_df[&#39;ds&#39;] = pd.to_datetime(test_df[&#39;ds&#39;])<br><br># Print shapes to debug<br>print(&quot;sf_forecast shape:&quot;, sf_forecast.shape)<br>print(&quot;ml_forecast shape:&quot;, ml_forecast.shape)<br>print(&quot;timegpt_forecast shape:&quot;, timegpt_forecast.shape)<br>print(&quot;timesfm_forecast shape:&quot;, timesfm_forecast.shape)<br>print(&quot;test_df shape:&quot;, test_df.shape)<br><br># Check the first few dates<br>print(&quot;\nFirst dates:&quot;)<br>print(&quot;sf_forecast:&quot;, sf_forecast[&#39;ds&#39;].head(3).tolist())<br>print(&quot;ml_forecast:&quot;, ml_forecast[&#39;ds&#39;].head(3).tolist())<br>print(&quot;test_df:&quot;, test_df[&#39;ds&#39;].head(3).tolist())</pre><h4>Perform the merges</h4><p>Start with test_df to ensure we keep all test dates.</p><pre>merged_fcst = test_df[[&#39;ds&#39;, &#39;y&#39;, &#39;unique_id&#39;]].copy()<br>merged_fcst = pd.merge(merged_fcst, sf_forecast[[&#39;ds&#39;, &#39;AutoARIMA&#39;, &#39;AutoETS&#39;]], on=&#39;ds&#39;, how=&#39;left&#39;)<br>merged_fcst = pd.merge(merged_fcst, ml_forecast[[&#39;ds&#39;, &#39;LGBMRegressor&#39;, &#39;XGBRegressor&#39;, &#39;RandomForestRegressor&#39;]], on=&#39;ds&#39;, how=&#39;left&#39;)<br>merged_fcst = pd.merge(merged_fcst, timegpt_forecast[[&#39;ds&#39;, &#39;TimeGPT&#39;]], on=&#39;ds&#39;, how=&#39;left&#39;)<br>merged_fcst = pd.merge(merged_fcst, timesfm_forecast[[&#39;ds&#39;, &#39;timesfm&#39;]], on=&#39;ds&#39;, how=&#39;left&#39;)<br><br>print(&quot;\nMerged forecast shape:&quot;, merged_fcst.shape)<br>print(&quot;\nMerged forecast columns:&quot;, merged_fcst.columns.tolist())<br>print(&quot;\nFirst few rows of merged_fcst:&quot;)<br>print(merged_fcst.head())<br>print(&quot;\nNull counts:&quot;)<br>print(merged_fcst.isnull().sum())</pre><h4>Model Comparison</h4><p>Finally, we can calculate and compare error metrics for multiple forecasting models (AutoARIMA, AutoETS, LGBMRegressor, XGBRegressor, RandomForestRegressor, TimeGPT, and timesfm) on a given dataset. It first defines a function calculate_error_metrics that computes MAE, RMSE, and MAPE between actual and predicted values.</p><pre>import numpy as np<br><br>def calculate_error_metrics(actual_values, predicted_values):<br>    actual_values = np.array(actual_values)<br>    predicted_values = np.array(predicted_values)<br>    <br>    # Remove any NaN values<br>    mask = ~(np.isnan(actual_values) | np.isnan(predicted_values))<br>    actual_values = actual_values[mask]<br>    predicted_values = predicted_values[mask]<br>    <br>    if len(actual_values) == 0:<br>        print(f&quot;Warning: No valid data points after removing NaNs&quot;)<br>        return pd.DataFrame({&#39;Metric&#39;: [&#39;MAE&#39;, &#39;RMSE&#39;, &#39;MAPE&#39;], &#39;Value&#39;: [np.nan, np.nan, np.nan]})<br>    <br>    metrics_dict = {<br>        &#39;MAE&#39;: np.mean(np.abs(actual_values - predicted_values)),<br>        &#39;RMSE&#39;: np.sqrt(np.mean((actual_values - predicted_values)**2)),<br>        &#39;MAPE&#39;: np.mean(np.abs((actual_values - predicted_values) / actual_values)) * 100<br>    }<br>    <br>    result_df = pd.DataFrame(list(metrics_dict.items()), columns=[&#39;Metric&#39;, &#39;Value&#39;])<br>    return result_df<br><br># Use actual gold prices from merged dataframe<br>actuals = merged_fcst[&#39;y&#39;].values<br>error_metrics_dict = {}<br><br># Model columns to evaluate<br>model_columns = [&#39;AutoARIMA&#39;, &#39;AutoETS&#39;, &#39;LGBMRegressor&#39;, &#39;XGBRegressor&#39;, &#39;RandomForestRegressor&#39;, &#39;TimeGPT&#39;, &#39;timesfm&#39;]<br><br>for col in model_columns:<br>    if col in merged_fcst.columns:<br>        print(f&quot;\nEvaluating {col}...&quot;)<br>        predicted_values = merged_fcst[col].values<br>        print(f&quot;  Actuals shape: {actuals.shape}, Predictions shape: {predicted_values.shape}&quot;)<br>        print(f&quot;  Non-null predictions: {(~np.isnan(predicted_values)).sum()}&quot;)<br>        error_metrics_dict[col] = calculate_error_metrics(actuals, predicted_values)[&#39;Value&#39;].values<br>    else:<br>        print(f&quot;\nWarning: {col} not found in merged_fcst&quot;)<br><br>error_metrics_df = pd.DataFrame(error_metrics_dict)<br>error_metrics_df.insert(0, &#39;Metric&#39;, [&#39;MAE&#39;, &#39;RMSE&#39;, &#39;MAPE&#39;])<br><br>print(&quot;\n&quot; + &quot;=&quot;*80)<br>print(&quot;FINAL ERROR METRICS:&quot;)<br>print(&quot;=&quot;*80)<br>print(error_metrics_df)</pre><p>Executing the code step by step produces the final output as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/891/1*aijBBWRJXKLy6bEcn-8zBg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/862/1*rQN08lNwgrg--ltHxXzyTQ.png" /></figure><p>The complete code for the model comparison is available at the link below.</p><p><a href="https://colab.research.google.com/drive/1UAKc3DEPQzsOHov1gug1-1iWmpdRFiHe?usp=sharing">Google Colab</a></p><h3>Building NextGen Forecasting app using TimesFM</h3><p>In the previous section, various forecasting models were explored, including zero-shot prompting and fine-tuning of the TimesFM model to generate forecasts. In this section, a Streamlit-based application is developed for time series forecasting. The app allows users to upload CSV files, configure forecast settings, generate forecasts, visualize results, and interactively query the forecasts. It supports multiple aggregation methods, customizable forecast horizons, and provides confidence intervals along with a Q&amp;A interface to interpret forecast insights.</p><h4>Installing the dependencies</h4><ul><li>Initialize a uv project by executing the following command.</li></ul><pre>uv init timesfm_forecasting_tool<br>cd timesfm_forecasting_tool</pre><ul><li>Create and activate a virtual environment by executing the following command.</li></ul><pre>uv venv<br>source .venv/bin/activate # for linux<br>.venv\Scripts\activate    # for windows</pre><ul><li>Clone the github repository and move to the folder using the following command.</li></ul><pre>git clone https://github.com/google-research/timesfm.git<br>cd timesfm</pre><ul><li>Install timesfm by executing the following command.</li></ul><pre>uv pip install -e .[torch]</pre><ul><li>Install streamlit and matplotlib using uv.</li></ul><pre>uv add streamlit matplotlib</pre><ul><li>Navigate back to the root directory, create a file named app.py, and add the following code to it.</li></ul><pre>import streamlit as st<br>import pandas as pd<br>import numpy as np<br>import torch<br>import timesfm<br>import matplotlib.pyplot as plt<br><br># Page configuration<br>st.set_page_config(<br>    page_title=&quot;TimesFM Forecasting Tool&quot;,<br>    page_icon=&quot;📈&quot;,<br>    layout=&quot;wide&quot;<br>)<br><br># Initialize TimesFM model<br>@st.cache_resource<br>def load_model():<br>    torch.set_float32_matmul_precision(&quot;high&quot;)<br>    LOCAL_MODEL_PATH = &quot;./timesfm_model&quot;<br>    <br>    try:<br>        model = timesfm.TimesFM_2p5_200M_torch.from_pretrained(LOCAL_MODEL_PATH)<br>        st.success(f&quot;✓ Model loaded from local path: {LOCAL_MODEL_PATH}&quot;)<br>    except Exception as e:<br>        st.warning(f&quot;Failed to load from local path, trying HuggingFace...&quot;)<br>        model = timesfm.TimesFM_2p5_200M_torch.from_pretrained(&quot;google/timesfm-2.5-200m-pytorch&quot;)<br>        st.success(&quot;✓ Model loaded from HuggingFace&quot;)<br>    <br>    model.compile(<br>        timesfm.ForecastConfig(<br>            max_context=1024,<br>            max_horizon=256,<br>            normalize_inputs=True,<br>            use_continuous_quantile_head=True,<br>            force_flip_invariance=True,<br>            infer_is_positive=True,<br>            fix_quantile_crossing=True,<br>        )<br>    )<br>    return model<br><br>def load_and_prepare_data(df, target_column, date_column, aggregation, time_period):<br>    &quot;&quot;&quot;Prepare time series data from dataframe&quot;&quot;&quot;<br>    # Make a copy to avoid modifying original<br>    df = df.copy()<br>    <br>    # Convert date column to datetime with multiple format attempts<br>    date_formats = [<br>        &#39;%d-%m-%Y&#39;,  # 01-01-1985<br>        &#39;%m-%d-%Y&#39;,  # 01-01-1985 (alternative interpretation)<br>        &#39;%Y-%m-%d&#39;,  # 1985-01-01<br>        &#39;%m/%d/%Y %H:%M&#39;,  # 2/24/2003 0:00<br>        &#39;%d/%m/%Y %H:%M&#39;,  # 24/2/2003 0:00<br>        &#39;%Y-%m-%d %H:%M:%S&#39;,<br>        &#39;%Y-%m-%d %H:%M&#39;,<br>        &#39;%d-%m-%Y %H:%M:%S&#39;,<br>        &#39;%d-%m-%Y %H:%M&#39;,<br>        &#39;%m/%d/%Y&#39;,<br>        &#39;%d/%m/%Y&#39;,<br>        &#39;%Y/%m/%d&#39;,<br>        &#39;%d.%m.%Y&#39;,  # 01.01.1985<br>        &#39;%Y.%m.%d&#39;,  # 1985.01.01<br>    ]<br>    <br>    parsed_successfully = False<br>    successful_format = None<br>    <br>    for fmt in date_formats:<br>        try:<br>            test_parse = pd.to_datetime(df[date_column], format=fmt, errors=&#39;coerce&#39;)<br>            # Check if at least some dates were parsed<br>            if test_parse.notna().sum() &gt; 0:<br>                df[date_column] = test_parse<br>                parsed_successfully = True<br>                successful_format = fmt<br>                break<br>        except:<br>            continue<br>    <br>    # If no format worked, try automatic parsing<br>    if not parsed_successfully:<br>        try:<br>            df[date_column] = pd.to_datetime(df[date_column], errors=&#39;coerce&#39;, infer_datetime_format=True)<br>            if df[date_column].notna().sum() &gt; 0:<br>                parsed_successfully = True<br>                successful_format = &quot;auto-detected&quot;<br>        except:<br>            pass<br>    <br>    if not parsed_successfully:<br>        # Show first few non-null values to help debug<br>        sample_values = df[date_column].dropna().head(5).tolist()<br>        raise ValueError(f&quot;Could not parse dates in column &#39;{date_column}&#39;. Sample values: {sample_values}&quot;)<br>    <br>    # Remove rows with invalid dates or missing target values<br>    initial_count = len(df)<br>    df = df.dropna(subset=[date_column, target_column])<br>    removed_count = initial_count - len(df)<br>    <br>    if len(df) == 0:<br>        raise ValueError(f&quot;No valid data after removing {removed_count} rows with invalid dates or missing target values.&quot;)<br>    <br>    # Sort by date<br>    df = df.sort_values(date_column)<br>    <br>    # Create period column based on time_period<br>    if time_period == &quot;Day&quot;:<br>        df[&#39;period&#39;] = df[date_column].dt.strftime(&#39;%d-%b-%Y&#39;)<br>        period_format = &quot;day&quot;<br>    elif time_period == &quot;Week&quot;:<br>        df[&#39;period&#39;] = df[date_column].dt.strftime(&#39;Week %U, %Y&#39;)<br>        period_format = &quot;week&quot;<br>    elif time_period == &quot;Month&quot;:<br>        df[&#39;period&#39;] = df[date_column].dt.strftime(&#39;%B %Y&#39;)<br>        period_format = &quot;month&quot;<br>    elif time_period == &quot;Year&quot;:<br>        df[&#39;period&#39;] = df[date_column].dt.year.astype(str)<br>        period_format = &quot;year&quot;<br>    else:<br>        df[&#39;period&#39;] = df[date_column].dt.strftime(&#39;%B %Y&#39;)<br>        period_format = &quot;month&quot;<br>    <br>    # Group by period and aggregate<br>    if aggregation == &quot;Sum&quot;:<br>        grouped = df.groupby(&#39;period&#39;)[target_column].sum()<br>    elif aggregation == &quot;Mean&quot;:<br>        grouped = df.groupby(&#39;period&#39;)[target_column].mean()<br>    elif aggregation == &quot;Median&quot;:<br>        grouped = df.groupby(&#39;period&#39;)[target_column].median()<br>    elif aggregation == &quot;Count&quot;:<br>        grouped = df.groupby(&#39;period&#39;)[target_column].count()<br>    else:<br>        grouped = df.groupby(&#39;period&#39;)[target_column].sum()<br>    <br>    periods = grouped.index.tolist()<br>    values = grouped.values.tolist()<br>    <br>    return periods, values, period_format<br><br>def generate_forecast_labels(last_period, horizon, period_format):<br>    &quot;&quot;&quot;Generate forecast period labels&quot;&quot;&quot;<br>    forecast_labels = []<br>    <br>    if period_format == &quot;month&quot;:<br>        try:<br>            last_date = pd.to_datetime(last_period, format=&#39;%B %Y&#39;)<br>            for i in range(1, horizon + 1):<br>                next_date = last_date + pd.DateOffset(months=i)<br>                forecast_labels.append(next_date.strftime(&#39;%B %Y&#39;))<br>        except:<br>            forecast_labels = [f&quot;Forecast {i+1}&quot; for i in range(horizon)]<br>    elif period_format == &quot;day&quot;:<br>        try:<br>            last_date = pd.to_datetime(last_period, format=&#39;%d-%b-%Y&#39;)<br>            for i in range(1, horizon + 1):<br>                next_date = last_date + pd.Timedelta(days=i)<br>                forecast_labels.append(next_date.strftime(&#39;%d-%b-%Y&#39;))<br>        except:<br>            forecast_labels = [f&quot;Forecast {i+1}&quot; for i in range(horizon)]<br>    elif period_format == &quot;year&quot;:<br>        try:<br>            last_year = int(last_period)<br>            forecast_labels = [str(last_year + i) for i in range(1, horizon + 1)]<br>        except:<br>            forecast_labels = [f&quot;Forecast {i+1}&quot; for i in range(horizon)]<br>    else:<br>        forecast_labels = [f&quot;Forecast {i+1}&quot; for i in range(horizon)]<br>    <br>    return forecast_labels<br><br>def generate_chat_response(query, forecast_data, summary):<br>    &quot;&quot;&quot;Generate responses to user queries about the forecast&quot;&quot;&quot;<br>    query_lower = query.lower()<br>    <br>    if &quot;trend&quot; in query_lower or &quot;direction&quot; in query_lower:<br>        direction = summary[&#39;trend&#39;][&#39;direction&#39;]<br>        change = summary[&#39;trend&#39;][&#39;change_percent&#39;]<br>        return f&quot;The forecast shows a **{direction}** trend with a {abs(change):.2f}% change compared to historical data.&quot;<br>    <br>    elif &quot;highest&quot; in query_lower or &quot;maximum&quot; in query_lower or &quot;peak&quot; in query_lower:<br>        max_val = summary[&#39;forecast_stats&#39;][&#39;max&#39;]<br>        max_idx = forecast_data[&#39;point_forecast&#39;].index(max_val)<br>        max_period = forecast_data[&#39;periods&#39;][max_idx]<br>        return f&quot;The highest forecasted value is **{max_val:,.2f}** in **{max_period}**.&quot;<br>    <br>    elif &quot;lowest&quot; in query_lower or &quot;minimum&quot; in query_lower:<br>        min_val = summary[&#39;forecast_stats&#39;][&#39;min&#39;]<br>        min_idx = forecast_data[&#39;point_forecast&#39;].index(min_val)<br>        min_period = forecast_data[&#39;periods&#39;][min_idx]<br>        return f&quot;The lowest forecasted value is **{min_val:,.2f}** in **{min_period}**.&quot;<br>    <br>    elif &quot;average&quot; in query_lower or &quot;mean&quot; in query_lower:<br>        avg = summary[&#39;forecast_stats&#39;][&#39;mean&#39;]<br>        return f&quot;The average forecasted value is **{avg:,.2f}**.&quot;<br>    <br>    elif &quot;total&quot; in query_lower or &quot;sum&quot; in query_lower:<br>        total = summary[&#39;forecast_stats&#39;][&#39;total&#39;]<br>        return f&quot;The total forecasted value across all periods is **{total:,.2f}**.&quot;<br>    <br>    elif &quot;confidence&quot; in query_lower or &quot;interval&quot; in query_lower:<br>        ci_80 = summary[&#39;confidence_intervals&#39;][&#39;80_percent&#39;]<br>        return f&quot;The 80% confidence interval ranges from **{ci_80[&#39;lower&#39;]:,.2f}** to **{ci_80[&#39;upper&#39;]:,.2f}**.&quot;<br>    <br>    elif &quot;compare&quot; in query_lower or &quot;historical&quot; in query_lower:<br>        hist_mean = summary[&#39;historical_stats&#39;][&#39;mean&#39;]<br>        fore_mean = summary[&#39;forecast_stats&#39;][&#39;mean&#39;]<br>        diff = fore_mean - hist_mean<br>        pct = ((fore_mean - hist_mean) / hist_mean * 100) if hist_mean != 0 else 0<br>        return f&quot;Historical average: **{hist_mean:,.2f}**\nForecast average: **{fore_mean:,.2f}**\nDifference: **{diff:+,.2f}** ({pct:+.2f}%)&quot;<br>    <br>    else:<br>        return &quot;I can help you with questions about:\n- Trend and direction\n- Highest/lowest values\n- Average and totals\n- Confidence intervals\n- Comparing historical vs forecast data\n\nPlease ask a specific question!&quot;<br><br># Main UI<br>st.title(&quot;📈 TimesFM Forecasting Tool&quot;)<br>st.markdown(&quot;Upload your data and generate time series forecasts using Google&#39;s TimesFM model&quot;)<br><br># Load model<br>with st.spinner(&quot;Loading TimesFM model...&quot;):<br>    model = load_model()<br><br># Sidebar for controls<br>st.sidebar.header(&quot;⚙️ Forecast Configuration&quot;)<br><br># Data input method<br>input_method = st.sidebar.radio(&quot;Data Input Method&quot;, [&quot;Upload CSV&quot;, &quot;Paste Text Data&quot;])<br><br>df = None<br><br>if input_method == &quot;Upload CSV&quot;:<br>    uploaded_file = st.sidebar.file_uploader(&quot;Upload CSV File&quot;, type=[&#39;csv&#39;])<br>    if uploaded_file:<br>        # Try different encodings<br>        encodings = [&#39;utf-8&#39;, &#39;latin-1&#39;, &#39;iso-8859-1&#39;, &#39;cp1252&#39;, &#39;utf-16&#39;]<br>        df = None<br>        <br>        for encoding in encodings:<br>            try:<br>                uploaded_file.seek(0)  # Reset file pointer<br>                df = pd.read_csv(uploaded_file, encoding=encoding, on_bad_lines=&#39;skip&#39;)<br>                st.sidebar.success(f&quot;✓ Loaded {len(df)} rows (encoding: {encoding})&quot;)<br>                break<br>            except (UnicodeDecodeError, Exception) as e:<br>                continue<br>        <br>        if df is None:<br>            st.sidebar.error(&quot;❌ Could not read file. Please check the file encoding.&quot;)<br>else:<br>    text_data = st.sidebar.text_area(&quot;Paste CSV Data (with headers)&quot;, height=200)<br>    if text_data:<br>        from io import StringIO<br>        df = pd.read_csv(StringIO(text_data))<br>        st.sidebar.success(f&quot;✓ Loaded {len(df)} rows&quot;)<br><br># Main content area<br>if df is not None:<br>    # Show data preview<br>    with st.expander(&quot;📊 Data Preview&quot;, expanded=True):<br>        st.dataframe(df.head(10), use_container_width=True)<br>        st.info(f&quot;Total rows: {len(df)} | Columns: {len(df.columns)}&quot;)<br>    <br>    # Configuration<br>    col1, col2 = st.sidebar.columns(2)<br>    <br>    # Get numeric and date columns<br>    numeric_cols = df.select_dtypes(include=[np.number]).columns.tolist()<br>    all_cols = df.columns.tolist()<br>    <br>    with col1:<br>        date_column = st.selectbox(&quot;Date Column&quot;, all_cols)<br>    <br>    with col2:<br>        target_column = st.selectbox(&quot;Target Column&quot;, numeric_cols)<br>    <br>    col3, col4 = st.sidebar.columns(2)<br>    <br>    with col3:<br>        time_period = st.selectbox(&quot;Time Period&quot;, [&quot;Day&quot;, &quot;Week&quot;, &quot;Month&quot;, &quot;Year&quot;])<br>    <br>    with col4:<br>        aggregation = st.selectbox(&quot;Aggregation&quot;, [&quot;Sum&quot;, &quot;Mean&quot;, &quot;Median&quot;, &quot;Count&quot;])<br>    <br>    horizon = st.sidebar.slider(&quot;Forecast Horizon&quot;, min_value=1, max_value=24, value=6)<br>    <br>    # Forecast button<br>    if st.sidebar.button(&quot;🚀 Generate Forecast&quot;, type=&quot;primary&quot;, use_container_width=True):<br>        with st.spinner(&quot;Generating forecast...&quot;):<br>            try:<br>                # Prepare data<br>                periods, values, period_format = load_and_prepare_data(<br>                    df.copy(), target_column, date_column, aggregation, time_period<br>                )<br>                <br>                if len(values) &lt; 3:<br>                    st.error(f&quot;Not enough data points after aggregation. Got {len(values)}, need at least 3&quot;)<br>                else:<br>                    # Generate forecast<br>                    historical_array = np.array(values, dtype=np.float32)<br>                    point_forecast, quantile_forecast = model.forecast(<br>                        horizon=horizon,<br>                        inputs=[historical_array],<br>                    )<br>                    <br>                    forecast_values = point_forecast[0]<br>                    quantiles = quantile_forecast[0]<br>                    <br>                    # Generate labels<br>                    forecast_labels = generate_forecast_labels(periods[-1], horizon, period_format)<br>                    <br>                    # Convert to native types<br>                    forecast_values_list = [float(x) for x in forecast_values]<br>                    <br>                    # Store in session state<br>                    st.session_state[&#39;forecast_data&#39;] = {<br>                        &#39;periods&#39;: forecast_labels,<br>                        &#39;point_forecast&#39;: forecast_values_list,<br>                        &#39;quantiles&#39;: {<br>                            &#39;q10&#39;: [float(x) for x in quantiles[:, 1]],<br>                            &#39;q25&#39;: [float(x) for x in quantiles[:, 3]],<br>                            &#39;q50&#39;: [float(x) for x in quantiles[:, 5]],<br>                            &#39;q75&#39;: [float(x) for x in quantiles[:, 7]],<br>                            &#39;q90&#39;: [float(x) for x in quantiles[:, 9]]<br>                        },<br>                        &#39;horizon&#39;: horizon<br>                    }<br>                    <br>                    st.session_state[&#39;summary&#39;] = {<br>                        &#39;historical_stats&#39;: {<br>                            &#39;count&#39;: len(values),<br>                            &#39;mean&#39;: float(np.mean(values)),<br>                            &#39;std&#39;: float(np.std(values)),<br>                            &#39;min&#39;: float(np.min(values)),<br>                            &#39;max&#39;: float(np.max(values)),<br>                            &#39;total&#39;: float(np.sum(values))<br>                        },<br>                        &#39;forecast_stats&#39;: {<br>                            &#39;count&#39;: len(forecast_values),<br>                            &#39;mean&#39;: float(np.mean(forecast_values)),<br>                            &#39;std&#39;: float(np.std(forecast_values)),<br>                            &#39;min&#39;: float(np.min(forecast_values)),<br>                            &#39;max&#39;: float(np.max(forecast_values)),<br>                            &#39;total&#39;: float(np.sum(forecast_values))<br>                        },<br>                        &#39;trend&#39;: {<br>                            &#39;direction&#39;: &#39;increasing&#39; if np.mean(forecast_values) &gt; np.mean(values) else &#39;decreasing&#39;,<br>                            &#39;change_percent&#39;: ((np.mean(forecast_values) - np.mean(values)) / np.mean(values) * 100) if np.mean(values) != 0 else 0<br>                        },<br>                        &#39;confidence_intervals&#39;: {<br>                            &#39;80_percent&#39;: {<br>                                &#39;lower&#39;: float(np.mean(quantiles[:, 1])),<br>                                &#39;upper&#39;: float(np.mean(quantiles[:, 9]))<br>                            },<br>                            &#39;50_percent&#39;: {<br>                                &#39;lower&#39;: float(np.mean(quantiles[:, 3])),<br>                                &#39;upper&#39;: float(np.mean(quantiles[:, 7]))<br>                            }<br>                        }<br>                    }<br>                    <br>                    st.session_state[&#39;periods&#39;] = periods<br>                    st.session_state[&#39;values&#39;] = values<br>                    st.session_state[&#39;forecast_labels&#39;] = forecast_labels<br>                    st.session_state[&#39;quantiles&#39;] = quantiles<br>                    st.session_state[&#39;target_column&#39;] = target_column<br>                    <br>                    st.success(&quot;✓ Forecast generated successfully!&quot;)<br>                    st.rerun()<br>                    <br>            except Exception as e:<br>                st.error(f&quot;Error generating forecast: {str(e)}&quot;)<br>    <br>    # Display results if forecast exists<br>    if &#39;forecast_data&#39; in st.session_state:<br>        st.markdown(&quot;---&quot;)<br>        st.header(&quot;📊 Forecast Results&quot;)<br>        <br>        # Summary metrics<br>        col1, col2, col3, col4 = st.columns(4)<br>        <br>        with col1:<br>            st.metric(<br>                &quot;Historical Mean&quot;,<br>                f&quot;{st.session_state[&#39;summary&#39;][&#39;historical_stats&#39;][&#39;mean&#39;]:,.2f}&quot;<br>            )<br>        <br>        with col2:<br>            st.metric(<br>                &quot;Forecast Mean&quot;,<br>                f&quot;{st.session_state[&#39;summary&#39;][&#39;forecast_stats&#39;][&#39;mean&#39;]:,.2f}&quot;,<br>                delta=f&quot;{st.session_state[&#39;summary&#39;][&#39;trend&#39;][&#39;change_percent&#39;]:.2f}%&quot;<br>            )<br>        <br>        with col3:<br>            st.metric(<br>                &quot;Trend&quot;,<br>                st.session_state[&#39;summary&#39;][&#39;trend&#39;][&#39;direction&#39;].title()<br>            )<br>        <br>        with col4:<br>            st.metric(<br>                &quot;Forecast Periods&quot;,<br>                st.session_state[&#39;forecast_data&#39;][&#39;horizon&#39;]<br>            )<br>        <br>        # Visualization<br>        st.subheader(&quot;📈 Forecast Visualization&quot;)<br>        <br>        fig, ax = plt.subplots(figsize=(14, 6))<br>        <br>        periods = st.session_state[&#39;periods&#39;]<br>        values = st.session_state[&#39;values&#39;]<br>        forecast_labels = st.session_state[&#39;forecast_labels&#39;]<br>        forecast_values = st.session_state[&#39;forecast_data&#39;][&#39;point_forecast&#39;]<br>        quantiles = st.session_state[&#39;quantiles&#39;]<br>        <br>        # Create x-axis<br>        total_points = len(values) + len(forecast_values)<br>        all_x = list(range(total_points))<br>        hist_x = all_x[:len(values)]<br>        forecast_x = all_x[len(values)-1:]<br>        <br>        # Plot<br>        ax.plot(hist_x, values, label=&quot;Historical Data&quot;, color=&quot;#3498db&quot;, <br>                linewidth=2, marker=&#39;o&#39;, markersize=4)<br>        <br>        forecast_with_connection = [values[-1]] + forecast_values<br>        ax.plot(forecast_x, forecast_with_connection, label=&quot;Forecast&quot;, <br>                color=&quot;#2ecc71&quot;, linewidth=2, marker=&#39;s&#39;, markersize=4)<br>        <br>        # Confidence intervals<br>        quantiles_with_connection = np.vstack([[values[-1]] * quantiles.shape[1], quantiles])<br>        <br>        ax.fill_between(forecast_x, quantiles_with_connection[:, 1], <br>                        quantiles_with_connection[:, 9], alpha=0.2, <br>                        color=&#39;#2ecc71&#39;, label=&#39;80% Prediction Interval&#39;)<br>        <br>        ax.fill_between(forecast_x, quantiles_with_connection[:, 3], <br>                        quantiles_with_connection[:, 7], alpha=0.3, <br>                        color=&#39;#2ecc71&#39;, label=&#39;50% Prediction Interval&#39;)<br>        <br>        # Labels<br>        all_labels = periods + forecast_labels<br>        step = max(1, len(all_labels) // 12)<br>        tick_positions = list(range(0, len(all_labels), step))<br>        tick_labels = [all_labels[i] for i in tick_positions]<br>        <br>        ax.set_xticks(tick_positions)<br>        ax.set_xticklabels(tick_labels, rotation=45, ha=&#39;right&#39;)<br>        ax.legend(loc=&#39;best&#39;)<br>        ax.set_title(f&quot;TimesFM Forecast: {st.session_state[&#39;target_column&#39;]}&quot;, <br>                     fontsize=14, fontweight=&#39;bold&#39;)<br>        ax.set_xlabel(&quot;Period&quot;)<br>        ax.set_ylabel(st.session_state[&#39;target_column&#39;])<br>        ax.grid(True, linestyle=&#39;--&#39;, alpha=0.6)<br>        <br>        plt.tight_layout()<br>        st.pyplot(fig)<br>        <br>        # Forecast table<br>        st.subheader(&quot;📋 Forecast Values&quot;)<br>        <br>        forecast_df = pd.DataFrame({<br>            &#39;Period&#39;: st.session_state[&#39;forecast_data&#39;][&#39;periods&#39;],<br>            &#39;Forecast&#39;: st.session_state[&#39;forecast_data&#39;][&#39;point_forecast&#39;],<br>            &#39;Lower (10%)&#39;: st.session_state[&#39;forecast_data&#39;][&#39;quantiles&#39;][&#39;q10&#39;],<br>            &#39;Lower (25%)&#39;: st.session_state[&#39;forecast_data&#39;][&#39;quantiles&#39;][&#39;q25&#39;],<br>            &#39;Median&#39;: st.session_state[&#39;forecast_data&#39;][&#39;quantiles&#39;][&#39;q50&#39;],<br>            &#39;Upper (75%)&#39;: st.session_state[&#39;forecast_data&#39;][&#39;quantiles&#39;][&#39;q75&#39;],<br>            &#39;Upper (90%)&#39;: st.session_state[&#39;forecast_data&#39;][&#39;quantiles&#39;][&#39;q90&#39;]<br>        })<br>        <br>        st.dataframe(forecast_df.style.format({<br>            &#39;Forecast&#39;: &#39;{:,.2f}&#39;,<br>            &#39;Lower (10%)&#39;: &#39;{:,.2f}&#39;,<br>            &#39;Lower (25%)&#39;: &#39;{:,.2f}&#39;,<br>            &#39;Median&#39;: &#39;{:,.2f}&#39;,<br>            &#39;Upper (75%)&#39;: &#39;{:,.2f}&#39;,<br>            &#39;Upper (90%)&#39;: &#39;{:,.2f}&#39;<br>        }), use_container_width=True)<br>        <br>        # Chat interface<br>        st.markdown(&quot;---&quot;)<br>        st.subheader(&quot;💬 Ask Questions About Your Forecast&quot;)<br>        <br>        user_query = st.text_input(<br>            &quot;Ask me anything about the forecast:&quot;,<br>            placeholder=&quot;e.g., What is the trend? What is the highest forecasted value?&quot;<br>        )<br>        <br>        if user_query:<br>            response = generate_chat_response(<br>                user_query, <br>                st.session_state[&#39;forecast_data&#39;],<br>                st.session_state[&#39;summary&#39;]<br>            )<br>            st.info(response)<br>        <br>        # Example questions<br>        with st.expander(&quot;💡 Example Questions&quot;):<br>            st.markdown(&quot;&quot;&quot;<br>            - What is the trend?<br>            - What is the highest forecasted value?<br>            - What is the average forecast?<br>            - Show me the confidence interval<br>            - Compare historical and forecast data<br>            - What is the total forecasted amount?<br>            &quot;&quot;&quot;)<br><br>else:<br>    # Welcome screen<br>    st.info(&quot;👈 Please upload a CSV file or paste your data to get started&quot;)<br>    <br>    st.markdown(&quot;&quot;&quot;<br>    ### How to use this tool:<br>    <br>    1. **Upload Data**: Choose to upload a CSV file or paste text data<br>    2. **Configure**: Select your date column, target column, and forecast settings<br>    3. **Generate**: Click the &quot;Generate Forecast&quot; button<br>    4. **Analyze**: View the forecast visualization and ask questions<br>    <br>    ### Supported Features:<br>    <br>    - ✅ Multiple time periods (Day, Week, Month, Year)<br>    - ✅ Various aggregation methods (Sum, Mean, Median, Count)<br>    - ✅ Customizable forecast horizon (1-24 periods)<br>    - ✅ Confidence intervals (50% and 80%)<br>    - ✅ Interactive Q&amp;A about forecasts<br>    &quot;&quot;&quot;)</pre><p>Run the Streamlit application using the following command to view the results.</p><pre>streamlit run app.py</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*qD87vsd-nSeRxB6UWkqSqg.gif" /></figure><p><em>Thanks for reading this article !!</em></p><p>If you enjoyed this article, please click on the clap button 👏 and share to help others find it!</p><p>The full source code for this tutorial can be found here,</p><p><a href="https://github.com/codemaker2015/timesfm-experiments">GitHub - codemaker2015/timesfm-experiments: Forecasting time series data using timesfm</a></p><h4>References</h4><ul><li><a href="https://research.google/blog/a-decoder-only-foundation-model-for-time-series-forecasting/">A decoder-only foundation model for time-series forecasting</a></li><li><a href="https://github.com/google-research/timesfm">GitHub - google-research/timesfm: TimesFM (Time Series Foundation Model) is a pretrained time-series foundation model developed by Google Research for time-series forecasting.</a></li><li><a href="https://colab.research.google.com/drive/1UAKc3DEPQzsOHov1gug1-1iWmpdRFiHe?usp=sharing">Google Colab</a></li><li><a href="https://www.kaggle.com/datasets/kandij/electric-production/data">Time series analysis</a></li><li><a href="https://www.kaggle.com/datasets/odins0n/monthly-gold-prices">Monthly Gold Prices (1979-2021)</a></li><li><a href="https://www.kaggle.com/datasets/kyanyoga/sample-sales-data">Sample Sales Data</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=57486ebca761" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding DeepEval: A Practical Guide for Evaluating Large Language Models]]></title>
            <link>https://codemaker2016.medium.com/understanding-deepeval-a-practical-guide-for-evaluating-large-language-models-d7272b6c2634?source=rss-b7a60e472dfc------2</link>
            <guid isPermaLink="false">https://medium.com/p/d7272b6c2634</guid>
            <category><![CDATA[llm-testing]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[deepeval]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[llm-evaluation]]></category>
            <dc:creator><![CDATA[Vishnu Sivan]]></dc:creator>
            <pubDate>Tue, 09 Sep 2025 04:43:13 GMT</pubDate>
            <atom:updated>2025-09-12T04:04:43.697Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WyyyfS2tFEK7E17Daa9jvg.png" /></figure><p>The rapid evolution of Large Language Models (LLMs) has transformed the way we build intelligent applications, but with this growth comes an equally important challenge — how do we measure their true effectiveness? Traditional evaluation methods often fall short in capturing the diverse capabilities and limitations of LLMs. From reasoning and accuracy to coherence, bias, and ethical alignment, a robust evaluation framework is essential to ensure that these models are reliable and suitable for real-world use.</p><p>This is where DeepEval comes in. DeepEval is an open-source framework built to streamline LLM testing by offering a comprehensive suite of metrics, synthetic dataset generation, real-time evaluation, and seamless integration with popular testing frameworks like Pytest. By enabling easy customization, DeepEval empowers researchers and developers to benchmark models against tasks like MMLU, apply advanced metrics such as G-eval, and rigorously validate outputs for relevance and reliability.</p><p>In this tutorial, you’ll learn how to set up DeepEval, create a relevance test inspired by Pytest, evaluate LLM outputs using the G-eval metric, and run MMLU benchmarking on the TinyLlama model. By the end, you’ll have a clear workflow to systematically test and improve the performance of your LLM-powered applications.</p><h3>Getting Started</h3><h4>Table of contents</h4><ul><li><a href="#e4e6">What is DeepEval</a></li><li><a href="#e20d">Key Features</a></li><li><a href="#87e7">Getting started with evaluation of LLM models using DeepEval</a></li><li><a href="#99ab">Installing the dependencies</a></li><li><a href="#194a">Querying the Model &amp; Measuring Different Metrics</a></li><li><a href="#cf0f">Example 1: Answer Relevancy Metric</a></li><li><a href="#c81f">Example 2: G-Eval Metric</a></li><li><a href="#a7df">Example 3: Prompt Alignment Metric</a></li><li><a href="#c74f">Example 4: Json Correctness Metric</a></li><li><a href="#891a">Example 5: Summarization Metric</a></li><li><a href="#6d59">Example 6: LLM Integration</a></li><li><a href="#d772">Example 7: Hallucinations</a></li><li><a href="#f40c">Example 8: Faithfulness Metric</a></li><li><a href="#d194">Example 9: Chatbot Evaluation</a></li><li><a href="#31ce">Example 10: LLM Tracing</a></li><li><a href="#e0f8">Example 11: MCP Interactions</a></li><li><a href="#5cc3">MMLU benchmarking with DeepEval for custom LLMs</a></li></ul><h4>What is DeepEval</h4><p>DeepEval is an open-source evaluation framework designed for testing Large Language Models (LLMs) across multiple dimensions such as reasoning, accuracy, coherence, relevance, and ethical alignment. Unlike simple benchmarks, DeepEval goes beyond by offering custom metrics, real-time evaluation, synthetic dataset generation, and seamless integration with testing pipelines. It allows researchers and developers to systematically measure and monitor LLM performance both in experimentation and production.</p><h4>Key Features</h4><ul><li><strong>Extensive Metric Suite<br></strong>Provides 14+ research-backed metrics for LLM evaluation. <br>Includes advanced metrics like G-Eval (chain-of-thought reasoning), Faithfulness (accuracy &amp; reliability), Toxicity, Answer Relevancy, and Conversational Metrics such as knowledge retention and conversation completeness.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/611/1*-pCdH7BsRlX5s-Y6AsTr_w.png" /></figure><ul><li><strong>Custom Metric Development<br></strong>Allows users to define their own evaluation metrics tailored to specific use cases.</li><li><strong>Integration with LLMs<br></strong>Compatible with any LLM (including OpenAI models).<br>Supports benchmarking against popular datasets like MMLU and HumanEval.</li><li><strong>Real-Time Monitoring &amp; Benchmarking<br></strong>Enables continuous monitoring of LLMs in production.<br>Provides robust benchmarking capabilities to assess models efficiently.</li><li><strong>Simplified Testing with Pytest Integration<br></strong>Built with a Pytest-like architecture, making it easy to write unit tests for LLM outputs in just a few lines of code.</li><li><strong>Batch Evaluation Support<br></strong>Supports large-scale evaluations with batch processing, saving time when benchmarking custom LLMs.</li></ul><h3>Getting started with evaluation of LLM models using DeepEval</h3><p>In this session, we will explore how to evaluate Large Language Models (LLMs) with DeepEval. By default, DeepEval supports OpenAI, and we’ll be using the OpenAI GPT-4o-mini model. However, you can use any other LLM of your choice.</p><p>We will also use uv, a modern and fast Python package manager (instead of pip), to set up our environment and handle dependencies.</p><h4>Installing uv</h4><p>uv simplifies dependency management, virtual environments, and running scripts.</p><ul><li><strong>For Windows:</strong></li></ul><pre>powershell -ExecutionPolicy ByPass -c &quot;irm https://astral.sh/uv/install.ps1 | iex&quot;<br>set Path=C:\Users\Codem\.local\bin;%Path%</pre><ul><li><strong>For Linux / Mac:</strong></li></ul><pre>curl -LsSf https://astral.sh/uv/install.sh | sh</pre><p>Refer to the official website for detailed installation instructions.</p><p><a href="https://docs.astral.sh/uv/getting-started/installation">Installation | uv</a></p><h4>Installing the dependencies</h4><ul><li>Initialize a uv project by executing the following command.</li></ul><pre>uv init deepeval_demo<br>cd deepeval_demo</pre><ul><li>Create and activate a virtual environment by executing the following command.</li></ul><pre>uv venv<br>source .venv/bin/activate # for linux<br>.venv\Scripts\activate    # for windows</pre><ul><li>Install deepeval, langchain-openai, fastmcp, transformers, accelerate, bitsandbytes, datasets, pandas and python-dotenv using uv.</li></ul><pre>uv add deepeval langchain-openai fastmcp transformers accelerate bitsandbytes datasets pandas python-dotenv</pre><h4>Setting up the credentials</h4><ul><li>Create a file named .env. This file will store your environment variables, including the OpenAI key.</li><li>Open the .env file and add the following code to specify your OpenAI API key and Neo4j credentials.</li></ul><pre>OPENAI_API_KEY=sk-proj-C1K1hKug99wXxtj...</pre><h3>Querying the Model &amp; Measuring Different Metrics</h3><p>Now that the environment is set up, let’s start querying our LLM and measure the quality of its responses using different metrics.</p><h4>Example 1: Answer Relevancy Metric</h4><p>The Answer Relevancy Metric evaluates how relevant the model’s response is compared to the retrieval context. This is useful in RAG (Retrieval-Augmented Generation) systems or whenever you want to ensure the response aligns with supporting facts.</p><p>Create a file named test_relevancy.py and add the following code to it.</p><pre>from deepeval import assert_test<br>from deepeval.test_case import LLMTestCase<br>from deepeval.metrics import AnswerRelevancyMetric<br>from dotenv import load_dotenv<br>load_dotenv()<br><br>def test_relevancy():<br>    # Define the metric with a threshold<br>    relevancy_metric = AnswerRelevancyMetric(threshold=0.7, model=&quot;gpt-4o-mini&quot;)<br>    <br>    # Case 1: Partially relevant answer<br>    test_case_1 = LLMTestCase(<br>        input=&quot;Can I return these shoes after 30 days?&quot;,<br>        actual_output=&quot;Yes, you can return them. We offer a 30-day full refund. Do you have your original receipt?&quot;,<br>        retrieval_context=[<br>            &quot;All customers are eligible for a 30-day full refund at no extra cost.&quot;,<br>            &quot;Returns are only accepted within 30 days of purchase.&quot;,<br>        ],<br>    )<br>    <br>    # Case 2: Fully relevant answer<br>    test_case_2 = LLMTestCase(<br>        input=&quot;Can I return these shoes after 30 days?&quot;,<br>        actual_output=&quot;Unfortunately, returns are only accepted within 30 days of purchase.&quot;,<br>        retrieval_context=[<br>            &quot;All customers are eligible for a 30-day full refund at no extra cost.&quot;,<br>            &quot;Returns are only accepted within 30 days of purchase.&quot;,<br>        ],<br>    )<br>    <br>    # Run evaluation<br>    assert_test(test_case_1, [relevancy_metric])<br>    assert_test(test_case_2, [relevancy_metric])</pre><p><strong>Code explanation:</strong></p><ul><li><strong>Metric Defined</strong> → AnswerRelevancyMetric(threshold=0.7) sets the relevancy cutoff.</li><li><strong>Test Cases Created</strong> → Each LLMTestCase includes:<br>The user query (input), LLM output (actual_output) and retrieval context used for evaluation.</li><li><strong>Assertion</strong> → assert_test() automatically checks if the relevancy score passes the threshold.<br>In Test Case 1: the answer contradicts the context slightly (“Yes, you can return after 30 days” vs. rule of within 30 days).<br>In Test Case 2: the answer is fully aligned with the context.</li></ul><h4>Executing the test</h4><p>Use the following command to run the test.</p><pre>deepeval test run test_relevancy.py</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eYcP2gTol6EnvycOVGF68g.png" /></figure><h4>Example 2: G-Eval Metric</h4><p>G-Eval is an LLM evaluation framework that leverages chain-of-thought (CoT) reasoning to assess model outputs based on custom criteria. Unlike fixed metrics, G-Eval is highly flexible and can evaluate nearly any aspect of a response — such as factual accuracy, omissions, clarity, or adherence to instructions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/830/1*CIg2WHK8VfmXIcWP37GvGw.png" /><figcaption>Image Source: <a href="https://ar5iv.labs.arxiv.org/html/2303.16634">[2303.16634] G-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment</a></figcaption></figure><p>It works in two steps:</p><ol><li><strong>Generate Evaluation Steps</strong> — Uses CoT reasoning to break down the evaluation based on the given criteria.</li><li><strong>Determine Final Score</strong> — Applies those steps to score the LLM’s output.</li></ol><p>If evaluation steps are manually provided, G-Eval skips step one and directly uses them to calculate the score.</p><p>Create a file named test_geval_example.py and add the following code to it.</p><pre>from deepeval import assert_test<br>from deepeval.test_case import LLMTestCase, LLMTestCaseParams<br>from deepeval.metrics import GEval<br><br>from dotenv import load_dotenv<br>load_dotenv()<br><br>correctness_metric = GEval(<br>    name=&quot;Correctness&quot;,<br>    model=&quot;gpt-4o-mini&quot;,<br>    evaluation_params=[<br>        LLMTestCaseParams.EXPECTED_OUTPUT,<br>        LLMTestCaseParams.ACTUAL_OUTPUT],<br>    evaluation_steps=[<br>        &quot;Check whether the facts in &#39;actual output&#39; contradicts any facts in &#39;expected output&#39;&quot;,<br>        &quot;You should also lightly penalize omission of detail, and focus on the main idea&quot;,<br>        &quot;Vague language, or contradicting OPINIONS, are OK&quot;<br>    ],<br>)<br><br>first_test_case = LLMTestCase(input=&quot;What are the main causes of deforestation?&quot;,<br>                              actual_output=&quot;The main causes of deforestation include agricultural expansion, logging, infrastructure development, and urbanization.&quot;,<br>                              expected_output=&quot;The main causes of deforestation include agricultural expansion, logging, infrastructure development, and urbanization.&quot;)<br><br><br>second_test_case = LLMTestCase(input=&quot;Define the term &#39;artificial intelligence&#39;.&quot;,<br>                               actual_output=&quot;Artificial intelligence is the simulation of human intelligence by machines.&quot;,<br>                               expected_output=&quot;Artificial intelligence refers to the simulation of human intelligence in machines that are programmed to think and learn like humans, including tasks such as problem-solving, decision-making, and language understanding.&quot;)<br><br><br>third_test_case = LLMTestCase(input=&quot;List the primary colors.&quot;,<br>                              actual_output=&quot;The primary colors are green, orange, and purple.&quot;,<br>                              expected_output=&quot;The primary colors are red, blue, and yellow.&quot;)<br><br>test_cases = [first_test_case, second_test_case, third_test_case]<br>for test_case in test_cases:<br>    assert_test(test_case, [correctness_metric])</pre><p><strong>Code explanation</strong></p><ul><li>Define a G-Eval metric named “Correctness” using gpt-4o-mini.</li><li>Specify evaluation parameters: compare expected vs actual output.</li><li>Add custom evaluation steps: check contradictions, penalize omissions, allow vague language/opinions.</li><li>Create three LLM test cases with queries, actual outputs, and expected outputs.</li><li>Run all test cases using assert_test() with the correctness metric.</li></ul><h4>Executing the test</h4><p>Use the following command to run the test.</p><pre>deepeval test run test_geval_example.py</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S4RYJBd9ruoOnNF5LRPwIg.png" /></figure><h4>Example 3: Prompt Alignment Metric</h4><p>The prompt alignment metric evaluates whether an LLM’s generated output aligns with the instructions defined in the prompt template. It ensures that the model not only provides a relevant response to the query but also adheres to any specified formatting, style, or structural requirements.</p><p>Create a file named test_prompt_alignment.py and add the following code to it.</p><pre>from deepeval import evaluate<br>from deepeval.metrics import PromptAlignmentMetric<br>from deepeval.test_case import LLMTestCase<br>from langchain_core.prompts import ChatPromptTemplate<br>from langchain_openai import ChatOpenAI<br><br>from dotenv import load_dotenv<br>load_dotenv()<br><br>template = &quot;&quot;&quot;Question: {question}<br>Answer: Answer in Upper case.&quot;&quot;&quot;<br>prompt = ChatPromptTemplate.from_template(template)<br>model = ChatOpenAI(model=&quot;gpt-4o-mini&quot;)<br>chain = prompt | model<br>query = &quot;What is capital of India?&quot;<br>input_data = {&quot;question&quot;: query}<br># Invoke the chain with input data and display the response<br>actual_output = chain.invoke(input_data).content<br>print(&quot;actual_output:&quot;, actual_output)<br><br># Measuring prompt alignment<br>metric = PromptAlignmentMetric(<br>    prompt_instructions=[&quot;Reply in all uppercase&quot;],<br>    model=&quot;gpt-4o-mini&quot;,<br>    include_reason=True<br>)<br>test_case = LLMTestCase(<br>    input=query,<br>    actual_output=actual_output<br>)<br><br>metric.measure(test_case)<br>print(&quot;metric.score:&quot;, metric.score)<br>print(&quot;metric.reason:&quot;, metric.reason)<br><br># or evaluate test cases in bulk<br>evaluate([test_case], [metric])</pre><p>Run the test using deepval test run test_prompt_alignment.py command to check if the model output aligns with the prompt instructions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C1if2Rk97Uh9FOZuIpk1rQ.png" /></figure><h4>Example 4: Json Correctness Metric</h4><p>The JSON Correctness Metric evaluates whether an LLM’s generated output follows the correct JSON schema. Unlike other metrics that rely on an LLM for assessment, this metric simply checks the provided expected_schema and verifies if the actual_output can be successfully validated against it.</p><p>Create a file named test_json_correctness.py and add the following code to it.</p><pre>from deepeval import evaluate<br>from deepeval.metrics import JsonCorrectnessMetric<br>from deepeval.test_case import LLMTestCase<br>from langchain_core.prompts import ChatPromptTemplate<br>from langchain_openai import ChatOpenAI<br>from pydantic import BaseModel<br><br>from dotenv import load_dotenv<br>load_dotenv()<br><br>class ExampleSchema(BaseModel):<br>    name: str<br><br># Querying the model<br>template = &quot;&quot;&quot;Question: {question}<br>Answer:  Let&#39;s think step by step.&quot;&quot;&quot;<br>prompt = ChatPromptTemplate.from_template(template)<br>model = ChatOpenAI(model=&quot;gpt-4o-mini&quot;)<br>chain = prompt | model<br>query =&quot;Output me a random Json with the &#39;name&#39; key&quot;<br>input_data = {&quot;question&quot;: query}<br># Invoke the chain with input data and display the response<br>actual_output = chain.invoke(input_data).content<br>print(&quot;actual_output:&quot;, actual_output)<br><br># Measuring Json correctness<br>metric = JsonCorrectnessMetric(<br>    expected_schema=ExampleSchema,<br>    model=&quot;gpt-4o-mini&quot;,<br>    include_reason=True<br>)<br>test_case = LLMTestCase(<br>    input=query,<br>    actual_output=actual_output<br>)<br><br>metric.measure(test_case)<br>print(&quot;metric.score:&quot;, metric.score)<br>print(&quot;metric.reason:&quot;, metric.reason)<br><br># or evaluate test cases in bulk<br>evaluate([test_case], [metric])</pre><p>Run the test using deepval test run test_json_correctness.py command to check if the model output aligns with the instructions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ATxXO8z5ZuALOLUkurY0WA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mosCHrcc_4OxjZDPUxdDcw.png" /></figure><h4>Example 5: Summarization Metric</h4><p>The Summarization Metric evaluates whether an LLM generates factually accurate summaries that include the essential details from the original text. Its score is calculated using two components: the alignment_score, which checks if the summary avoids hallucinations or contradictions, and the coverage_score, which measures whether the summary captures all the necessary information from the source text.</p><p>Create a file named test_summarization.py and add the following code to it.</p><pre>from deepeval import evaluate<br>from deepeval.metrics import SummarizationMetric<br>from deepeval.test_case import LLMTestCase<br>from langchain_core.prompts import ChatPromptTemplate<br>from langchain_openai import ChatOpenAI<br>from pydantic import BaseModel<br><br>from dotenv import load_dotenv<br>load_dotenv()<br><br>class ExampleSchema(BaseModel):<br>    name: str<br><br># This is the original text to be summarized<br>text = &quot;&quot;&quot;<br>Rice is the staple food of Bengal. Bhortas (lit-&quot;mashed&quot;) are a really common type of food used as an additive too rice. there are several types of Bhortas such as Ilish bhorta shutki bhorta, begoon bhorta and more. Fish and other seafood are also important because Bengal is a reverrine region.<br>Some fishes like puti (Puntius species) are fermented. Fish curry is prepared with fish alone or in combination with vegetables.Shutki maach is made using the age-old method of preservation where the food item is dried in the sun and air, thus removing the water content. This allows for preservation that can make the fish last for months, even years in Bangladesh<br>&quot;&quot;&quot;<br><br>template = &quot;&quot;&quot;Question: {question}<br>Answer:  Let&#39;s think step by step.&quot;&quot;&quot;<br>prompt = ChatPromptTemplate.from_template(template)<br>model = ChatOpenAI(model=&quot;gpt-4o-mini&quot;)<br>chain = prompt | model<br>query =&quot;Summarize the text for me %s&quot;%(text)<br>input_data = {&quot;question&quot;: query}<br># Invoke the chain with input data and display the response in Markdown format<br>actual_output = chain.invoke(input_data).content<br>print(&quot;actual_output:&quot;, actual_output)<br><br>test_case = LLMTestCase(input=text, actual_output=actual_output)<br>metric = SummarizationMetric(<br>    threshold=0.7,<br>    model=&quot;gpt-4o-mini&quot;,<br>)<br><br>metric.measure(test_case)<br>print(&quot;metric.score:&quot;, metric.score)<br>print(&quot;metric.reason:&quot;, metric.reason)<br><br># or evaluate test cases in bulk<br>evaluate([test_case], [metric])</pre><p>Run the test using deepval test run test_summarization.py command to check if the summary is generated correctly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Wa8ywVYIMCtHcZMZUJuOlA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5RhIT0-PZK-h5RjWOg__xQ.png" /></figure><h4>Example 6: LLM Integration</h4><p>DeepEval is capable of integrating with any LLM and evaluating its performance across different metrics. In this example, we use LangChain’s ChatOpenAI with DeepEval to test the relevancy of a response. The model (gpt-4o-mini) is queried with <em>“What is the capital of India?”</em>, and the output is evaluated using the Answer Relevancy Metric.</p><p>Create a file named test_openai_llm.py and add the following code to it.</p><pre>from langchain_openai import ChatOpenAI<br>from deepeval.metrics import AnswerRelevancyMetric<br>from deepeval.test_case import LLMTestCase<br><br>from dotenv import load_dotenv<br>load_dotenv()<br><br># Initialize the model<br>chat = ChatOpenAI(model=&quot;gpt-4o-mini&quot;,temperature=0.7)<br><br># Get response<br>query = &quot;What is the capital of India?&quot;<br>response = chat.invoke(query).content<br>print(f&quot;User: {query} \nAssistant: {response}&quot;)<br><br>metric = AnswerRelevancyMetric(<br>    threshold=0.7,<br>    model=&quot;gpt-4o-mini&quot;,<br>    include_reason=True<br>)<br>test_case = LLMTestCase(<br>    input=query,<br>    actual_output=response<br>)<br><br>metric.measure(test_case)<br>print(metric.score)<br>print(metric.reason)</pre><p>Run the test using deepval test run test_openai_llm.py command to check the evaluation metrics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mNU8OR_5xUoreJddSHX-_Q.png" /></figure><h4>Example 7: Hallucinations</h4><p>DeepEval provides a Hallucination Metric that helps detect and score these cases. It compares the model’s actual output against the given context to check whether the response faithfully sticks to the facts provided. If the output strays beyond or invents unsupported details, the score decreases. This way, DeepEval helps ensure your LLM outputs remain accurate and grounded in the input context.</p><p>Create a file named test_hallucinations.py and add the following code to it.</p><pre>from deepeval import evaluate<br>from deepeval.metrics import HallucinationMetric<br>from deepeval.test_case import LLMTestCase<br><br>from dotenv import load_dotenv<br>load_dotenv()<br><br># Replace this with the actual documents that you are passing as input to your LLM.<br>context=[&quot;A man with blond-hair, and a brown shirt drinking out of a public water fountain.&quot;]<br><br># Replace this with the actual output from your LLM application<br>actual_output=&quot;A blond drinking water in public.&quot;<br><br>test_case = LLMTestCase(<br>    input=&quot;What was the blond doing?&quot;,<br>    actual_output=actual_output,<br>    context=context<br>)<br>metric = HallucinationMetric(threshold=0.5)<br><br># To run metric as a standalone<br># metric.measure(test_case)<br># print(metric.score, metric.reason)<br><br>evaluate(test_cases=[test_case], metrics=[metric])</pre><p>Run the test using deepval test run test_hallucinations.py command to check the evaluation metrics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*01_xA8l91Gk1qLQyndnzVQ.png" /></figure><h4>Example 8: Faithfulness Metric</h4><p>The FaithfulnessMetric in DeepEval evaluates whether an LLM’s output is accurately grounded in the provided context or source material. It measures if the response remains consistent with the facts without introducing fabricated or contradictory information. This metric is particularly useful for applications like question-answering or retrieval-augmented generation, where maintaining factual consistency is critical. A higher faithfulness score indicates that the model’s output reliably reflects the input context.</p><p>Create a file named test_faithfulness.py and add the following code to it.</p><pre>from deepeval import evaluate<br>from deepeval.test_case import LLMTestCase<br>from deepeval.metrics import FaithfulnessMetric<br><br>from dotenv import load_dotenv<br>load_dotenv()<br><br># Replace this with the actual output from your LLM application<br>actual_output = &quot;We offer a 30-day full refund at no extra cost.&quot;<br><br># Replace this with the actual retrieved context from your RAG pipeline<br>retrieval_context = [&quot;All customers are eligible for a 30 day full refund at no extra cost.&quot;]<br><br>metric = FaithfulnessMetric(<br>    threshold=0.7,<br>    model=&quot;gpt-4o-mini&quot;,<br>    include_reason=True<br>)<br>test_case = LLMTestCase(<br>    input=&quot;What if these shoes don&#39;t fit?&quot;,<br>    actual_output=actual_output,<br>    retrieval_context=retrieval_context<br>)<br><br># To run metric as a standalone<br># metric.measure(test_case)<br># print(metric.score, metric.reason)<br><br>evaluate(test_cases=[test_case], metrics=[metric])</pre><p>Run the test using deepval test run test_faithfulness.pycommand to check the evaluation metrics.</p><h4>Example 9: Chatbot Evaluation</h4><p>Chatbot Evaluation differs from standard single-turn evaluations because conversations occur over multiple turns. This requires the chatbot to maintain context awareness throughout the interaction, rather than simply providing accurate responses in isolation. <br>In DeepEval, chatbots are assessed through multi-turn interactions, which must be structured as test cases following OpenAI’s message format. Evaluating multi-turn conversations is challenging, as each AI response depends on the preceding user input and all prior turns in the conversation, making the evaluation inherently context-dependent.</p><p>Create a file named test_chatbots.py and add the following code to it.</p><pre>from deepeval.test_case import ConversationalTestCase, Turn<br>from deepeval.metrics import TurnRelevancyMetric, KnowledgeRetentionMetric<br>from deepeval import evaluate<br><br>from dotenv import load_dotenv<br>load_dotenv()<br><br>test_case = ConversationalTestCase(<br>    turns=[<br>        Turn(role=&quot;user&quot;, content=&quot;Hello, how are you?&quot;),<br>        Turn(role=&quot;assistant&quot;, content=&quot;I&#39;m doing well, thank you!&quot;),<br>        Turn(role=&quot;user&quot;, content=&quot;How can I help you today?&quot;),<br>        Turn(role=&quot;assistant&quot;, content=&quot;I&#39;d like to buy a ticket to a Coldplay concert.&quot;),<br>    ]<br>)<br><br>evaluate(test_cases=[test_case], metrics=[TurnRelevancyMetric(), KnowledgeRetentionMetric()])</pre><p>Run the test using deepval test run test_chatbots.pycommand to check the evaluation results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*njC7SocoQpVEixjxJT_iEg.png" /></figure><h4>Example 10: LLM Tracing</h4><p>LLM Tracing allows you to monitor the full execution of your application from start to finish. In DeepEval, the @observe decorator enables tracing and evaluation of any LLM interaction, regardless of the application’s complexity. By identifying individual components of your LLM workflow—such as functions that perform specific tasks or are invoked selectively—you can apply the @observe decorator to track their behavior. This provides detailed insights into how each part of your LLM application operates, making it easier to debug, optimize, and evaluate performance.</p><p>Tracing requires Confident AI credentials to see the traces. Create a basic account on Confident AI and add the given credentials in the .env file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9U8594EzTkb-l622exOy-A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uFpvgk8VdUvG7KPi5jb_Mw.png" /></figure><p>Confident AI is a comprehensive platform designed to evaluate and enhance the performance of large language models (LLMs). It leverages its open-source evaluation framework, DeepEval, to provide robust testing, benchmarking, and monitoring capabilities for LLM applications. Confident AI emphasizes observability, allowing teams to trace LLM interactions, conduct A/B testing, and gather real-time performance insights.</p><p><a href="https://www.confident-ai.com/">Confident AI - The DeepEval LLM Evaluation Platform</a></p><p>Create a file named test_llm_tracing.py and add the following code to it.</p><pre>from openai import OpenAI<br>from deepeval.tracing import observe<br><br>from dotenv import load_dotenv<br>load_dotenv()<br><br>client = OpenAI()<br><br>@observe()<br>def llm_app(query: str) -&gt; str:<br>    return client.chat.completions.create(<br>        model=&quot;gpt-4o-mini&quot;,<br>        messages=[<br>            {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: query}<br>        ]<br>    ).choices[0].message.content<br>    return<br><br># Call app to send trace to Confident AI<br>llm_app(&quot;Write me a poem.&quot;)</pre><p>Run the test using deepval test run test_llm_tracing.pycommand to see the tracing results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CIGhAjX1QHv1sYTmI2-aCg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cXM-tQeLQ8tfEbSUKUpkmw.png" /></figure><h4>Example 11: MCP Interactions</h4><p>The MCP Use Metric evaluates how effectively an MCP-based LLM agent utilizes the MCP servers it has access to. It leverages an LLM-as-a-judge approach to assess both the MCP primitives invoked and the arguments generated by the LLM application. This metric can be applied to a single-turn LLMTestCase containing MCP parameters, providing insights into the agent’s efficiency and correctness in interacting with the MCP environment.</p><ul><li>Create a basic MCP server using FastMCP. For that, create a file named mcp_echo_server.py and add the following code to it.</li></ul><pre>import asyncio<br>from fastmcp import FastMCP<br><br># Initialize FastMCP server<br>mcp = FastMCP(&quot;Simple Echo Server&quot;)<br><br>@mcp.tool()<br>def echo(message: str) -&gt; str:<br>    &quot;&quot;&quot;Echo back the provided message.&quot;&quot;&quot;<br>    return message<br><br>def main():<br>    &quot;&quot;&quot;Run the server.&quot;&quot;&quot;<br>    mcp.run()<br><br>if __name__ == &quot;__main__&quot;:<br>    main()</pre><ul><li>Run the server using the command</li></ul><pre>python mcp_echo_server.py </pre><ul><li>Create a test case to evalute the MCP. For that, create a file named test_mcp.py and add the following code to it.</li></ul><pre>from deepeval import evaluate<br>from deepeval.metrics import MCPUseMetric<br>from deepeval.test_case import LLMTestCase, MCPServer<br><br>from dotenv import load_dotenv<br>load_dotenv()<br><br>test_case = LLMTestCase(<br>    input=&quot;Hello&quot;, # Your input here<br>    actual_output=&quot;Hello&quot;, # Your LLM app&#39;s final output here<br>    mcp_servers=[MCPServer(server_name=&quot;Simple Echo Server&quot;)] # Your MCP server&#39;s data<br>    # MCP primitives used (if any)<br>)<br><br>metric = MCPUseMetric()<br><br># To run metric as a standalone<br># metric.measure(convo_test_case)<br># print(metric.score, metric.reason)<br><br>evaluate([test_case], [metric])</pre><p>Run the test using deepval test run test_mcp.pycommand to see the mcp evaluation results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OKgvEqsBouOP6MEvs1yvFg.png" /></figure><h3>MMLU benchmarking with DeepEval for custom LLMs</h3><p>MMLU (Massive Multitask Language Understanding) is a benchmark commonly used to evaluate large language models through multiple-choice questions. Covering 57 subjects ranging from math and history to law and ethics, it provides a thorough assessment of an LLM’s knowledge and reasoning skills. Its wide subject coverage and carefully designed questions have made MMLU a gold standard for measuring model performance.</p><p>In this guide, we will evaluate our custom LLM (TinyLlama-1.1B) on the MMLU dataset. Each entry in the dataset consists of an input prompt and multiple-choice answers (A, B, C, D). Model performance is measured by calculating the percentage of questions answered correctly.</p><h4><strong>Creating the Custom LLM Model Class</strong></h4><p>Define a custom class called TinyLlamaModel that extends DeepEvalBaseLLM to generate responses using the language model and tokenizer. The objective is to produce short outputs (two tokens) for a given prompt, while efficiently managing device allocation and handling preprocessing for both the input prompt and the generated output.</p><p>Create a file named test_custom_model.py and add the following code to it.</p><pre>import torch<br>from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig<br>from deepeval.models.base_model import DeepEvalBaseLLM<br>from deepeval.benchmarks import MMLU<br>from deepeval.benchmarks.tasks import MMLUTask<br>from typing import List<br><br>class TinyLlamaModel(DeepEvalBaseLLM):<br>    def __init__(self, model, tokenizer):<br>        self.model = model<br>        self.tokenizer = tokenizer<br>        self.device = &quot;cuda&quot; if torch.cuda.is_available() else &quot;cpu&quot;<br><br>    def load_model(self):<br>        return self.model<br><br>    def generate(self, prompt: str) -&gt; str:<br>        # Clean the prompt for multiple choice<br>        prompt = prompt.replace(&quot;Output &#39;A&#39;, &#39;B&#39;, &#39;C&#39;, or &#39;D&#39;. Full answer not needed.&quot;, &quot;&quot;)<br>        <br>        # Format the prompt for TinyLlama<br>        formatted_prompt = f&quot;### Instruction: Answer with just the letter (A, B, C, or D)\n\n### Question: {prompt}\n\n### Answer:&quot;<br>        <br>        model_inputs = self.tokenizer([formatted_prompt], return_tensors=&quot;pt&quot;).to(self.device)<br>        <br>        generated_ids = self.model.generate(<br>            **model_inputs,<br>            max_new_tokens=3,<br>            do_sample=False,<br>            temperature=0.1,<br>            pad_token_id=self.tokenizer.eos_token_id,<br>            repetition_penalty=1.1<br>        )<br>        <br>        # Extract only the new tokens<br>        generated_tokens = generated_ids[0][model_inputs[&#39;input_ids&#39;].shape[1]:]<br>        clean_output = self.tokenizer.decode(generated_tokens, skip_special_tokens=True).strip()<br>        <br>        # Extract just the letter (A, B, C, or D)<br>        for char in clean_output:<br>            if char in [&#39;A&#39;, &#39;B&#39;, &#39;C&#39;, &#39;D&#39;]:<br>                return char<br>        <br>        return clean_output[:1]  # Fallback: return first character<br><br>    async def a_generate(self, prompt: str) -&gt; str:<br>        return self.generate(prompt)<br><br>    def batch_generate(self, prompts: List[str]) -&gt; List[str]:<br>        &quot;&quot;&quot;Batch generate method required by MMLU benchmark&quot;&quot;&quot;<br>        results = []<br>        for prompt in prompts:<br>            try:<br>                result = self.generate(prompt)<br>                results.append(result)<br>            except Exception as e:<br>                print(f&quot;Error generating for prompt: {e}&quot;)<br>                results.append(&quot;&quot;)  # Fallback empty response<br>        return results<br><br>    async def a_batch_generate(self, prompts: List[str]) -&gt; List[str]:<br>        return self.batch_generate(prompts)<br><br>    def get_model_name(self):<br>        return &quot;TinyLlama-1.1B-Chat&quot;</pre><h4><strong>Loading the Model and Tokenizer</strong></h4><p>Create two functions to load the LLM model and tokenizer directly from local storage. The model will be loaded in 8-bit precision, and the tokenizer will be initialized with appropriate padding and special token configurations.</p><pre>def load_model(model_name: str):<br>    # Use light quantization for TinyLlama<br>    quant_config = BitsAndBytesConfig(<br>        load_in_4bit=True,<br>        bnb_4bit_quant_type=&quot;nf4&quot;,<br>        bnb_4bit_compute_dtype=torch.float16,<br>    )<br>    <br>    try:<br>        model = AutoModelForCausalLM.from_pretrained(<br>            model_name,<br>            quantization_config=quant_config,<br>            device_map=&quot;auto&quot;,<br>            dtype=torch.float16,<br>            trust_remote_code=True<br>        )<br>        return model<br>    except Exception as e:<br>        print(f&quot;Error loading quantized model, trying without quantization: {e}&quot;)<br>        # Fallback without quantization<br>        return AutoModelForCausalLM.from_pretrained(<br>            model_name,<br>            device_map=&quot;auto&quot;,<br>            dtype=torch.float16,<br>            trust_remote_code=True<br>        )<br><br>def load_tokenizer(model_name):<br>    tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)<br>    tokenizer.pad_token = tokenizer.eos_token<br>    tokenizer.padding_side = &quot;left&quot;<br>    return tokenizer</pre><h4>Building the custom LLM</h4><p>We will use the Hugging Face model TinyLlama/TinyLlama-1.1B-Chat-v1.0 to directly load both the model and tokenizer. These will then be passed into the custom LLM class to create an LLM response generator.</p><p>After loading the <strong>TinyLlama 1.1B</strong> model and tokenizer from Hugging Face and wrapping them in our custom class, we can test the response generation. The code first runs a single test prompt related to abstract algebra to verify that the model produces an output. It then performs batch generation with multiple prompts, such as arithmetic and geography questions, to validate that the model can handle multiple inputs efficiently. The results are printed for both single and batch generations, ensuring that our custom model class works as expected.</p><pre># Load TinyLlama 1.1B model from Hugging Face<br>tinyllama_model_name = &quot;TinyLlama/TinyLlama-1.1B-Chat-v1.0&quot;<br><br>print(&quot;Loading tokenizer...&quot;)<br>tokenizer = load_tokenizer(tinyllama_model_name)<br><br>print(&quot;Loading model...&quot;)<br>model = load_model(tinyllama_model_name)<br><br>print(&quot;Creating custom model...&quot;)<br>custom_model = TinyLlamaModel(model, tokenizer)<br><br># Test model generation<br>print(&quot;\nTesting model generation:&quot;)<br>prompt = &quot;&quot;&quot;<br>The following are multiple choice questions (with answers) about abstract algebra.<br><br>Find all c in Z_3 such that Z_3[x]/(x^2 + c) is a field.<br>A. 0<br>B. 1<br>C. 2<br>D. 3<br>Answer:&quot;&quot;&quot;<br><br>test_output = custom_model.generate(prompt)<br>print(f&quot;Generated output: &#39;{test_output}&#39;&quot;)<br><br># Test batch generation<br>print(&quot;\nTesting batch generation...&quot;)<br>test_prompts = [<br>    prompt,<br>    &quot;What is 2+2? A. 1 B. 2 C. 3 D. 4 Answer:&quot;,<br>    &quot;Capital of France? A. London B. Berlin C. Paris D. Rome Answer:&quot;<br>]<br><br>batch_outputs = custom_model.batch_generate(test_prompts)<br>for i, output in enumerate(batch_outputs):<br>    print(f&quot;Batch output {i+1}: &#39;{output}&#39;&quot;)</pre><h4><strong>Running the MMLU Benchmark</strong></h4><p>Finally, we will load the MMLU benchmark, define the tasks, and run the evaluation on the custom model. The results of each task can be reviewed using benchmark.task_scores, while benchmark.predictions provides detailed outputs showing which samples were answered correctly and which were not. This allows for a more granular analysis of the model’s performance.</p><pre># Run MMLU benchmark with very light settings<br>print(&quot;\nRunning MMLU benchmark...&quot;)<br>benchmark = MMLU(<br>    tasks=[MMLUTask.HIGH_SCHOOL_COMPUTER_SCIENCE],  # Only one task<br>    n_shots=2  # Reduced shots for smaller model<br>)<br><br>try:<br>    benchmark.evaluate(model=custom_model, batch_size=1)  # Batch size 1<br>    <br>    print(&quot;\nBenchmark Results:&quot;)<br>    print(f&quot;Task Scores: {benchmark.task_scores}&quot;)<br>    print(f&quot;Overall Score: {benchmark.overall_score}&quot;)<br>    <br>    # Print detailed predictions<br>    print(&quot;\nSample Predictions:&quot;)<br>    for i, (input_text, prediction) in enumerate(list(benchmark.predictions.items())[:3]):<br>        print(f&quot;Prediction {i+1}:&quot;)<br>        print(f&quot;Input: {input_text[:100]}...&quot;)<br>        print(f&quot;Prediction: {prediction}&quot;)<br>        print(&quot;---&quot;)<br>        <br>except Exception as e:<br>    print(f&quot;Error during benchmark evaluation: {e}&quot;)<br>    print(&quot;Trying with even smaller settings...&quot;)<br>    <br>    # Fallback: try with minimal settings<br>    benchmark = MMLU(<br>        tasks=[MMLUTask.HIGH_SCHOOL_COMPUTER_SCIENCE],<br>        n_shots=1<br>    )<br>    benchmark.evaluate(model=custom_model, batch_size=1)<br>    print(f&quot;Fallback overall score: {benchmark.overall_score}&quot;)</pre><h4>Final Code</h4><p>The complete implementation will look as follows:</p><pre>import torch<br>from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig<br>from deepeval.models.base_model import DeepEvalBaseLLM<br>from deepeval.benchmarks import MMLU<br>from deepeval.benchmarks.tasks import MMLUTask<br>from typing import List<br><br>class TinyLlamaModel(DeepEvalBaseLLM):<br>    def __init__(self, model, tokenizer):<br>        self.model = model<br>        self.tokenizer = tokenizer<br>        self.device = &quot;cuda&quot; if torch.cuda.is_available() else &quot;cpu&quot;<br><br>    def load_model(self):<br>        return self.model<br><br>    def generate(self, prompt: str) -&gt; str:<br>        # Clean the prompt for multiple choice<br>        prompt = prompt.replace(&quot;Output &#39;A&#39;, &#39;B&#39;, &#39;C&#39;, or &#39;D&#39;. Full answer not needed.&quot;, &quot;&quot;)<br>        <br>        # Format the prompt for TinyLlama<br>        formatted_prompt = f&quot;### Instruction: Answer with just the letter (A, B, C, or D)\n\n### Question: {prompt}\n\n### Answer:&quot;<br>        <br>        model_inputs = self.tokenizer([formatted_prompt], return_tensors=&quot;pt&quot;).to(self.device)<br>        <br>        generated_ids = self.model.generate(<br>            **model_inputs,<br>            max_new_tokens=3,<br>            do_sample=False,<br>            temperature=0.1,<br>            pad_token_id=self.tokenizer.eos_token_id,<br>            repetition_penalty=1.1<br>        )<br>        <br>        # Extract only the new tokens<br>        generated_tokens = generated_ids[0][model_inputs[&#39;input_ids&#39;].shape[1]:]<br>        clean_output = self.tokenizer.decode(generated_tokens, skip_special_tokens=True).strip()<br>        <br>        # Extract just the letter (A, B, C, or D)<br>        for char in clean_output:<br>            if char in [&#39;A&#39;, &#39;B&#39;, &#39;C&#39;, &#39;D&#39;]:<br>                return char<br>        <br>        return clean_output[:1]  # Fallback: return first character<br><br>    async def a_generate(self, prompt: str) -&gt; str:<br>        return self.generate(prompt)<br><br>    def batch_generate(self, prompts: List[str]) -&gt; List[str]:<br>        &quot;&quot;&quot;Batch generate method required by MMLU benchmark&quot;&quot;&quot;<br>        results = []<br>        for prompt in prompts:<br>            try:<br>                result = self.generate(prompt)<br>                results.append(result)<br>            except Exception as e:<br>                print(f&quot;Error generating for prompt: {e}&quot;)<br>                results.append(&quot;&quot;)  # Fallback empty response<br>        return results<br><br>    async def a_batch_generate(self, prompts: List[str]) -&gt; List[str]:<br>        return self.batch_generate(prompts)<br><br>    def get_model_name(self):<br>        return &quot;TinyLlama-1.1B-Chat&quot;<br>    <br>def load_model(model_name: str):<br>    # Use light quantization for TinyLlama<br>    quant_config = BitsAndBytesConfig(<br>        load_in_4bit=True,<br>        bnb_4bit_quant_type=&quot;nf4&quot;,<br>        bnb_4bit_compute_dtype=torch.float16,<br>    )<br>    <br>    try:<br>        model = AutoModelForCausalLM.from_pretrained(<br>            model_name,<br>            quantization_config=quant_config,<br>            device_map=&quot;auto&quot;,<br>            dtype=torch.float16,<br>            trust_remote_code=True<br>        )<br>        return model<br>    except Exception as e:<br>        print(f&quot;Error loading quantized model, trying without quantization: {e}&quot;)<br>        # Fallback without quantization<br>        return AutoModelForCausalLM.from_pretrained(<br>            model_name,<br>            device_map=&quot;auto&quot;,<br>            dtype=torch.float16,<br>            trust_remote_code=True<br>        )<br><br>def load_tokenizer(model_name):<br>    tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)<br>    tokenizer.pad_token = tokenizer.eos_token<br>    tokenizer.padding_side = &quot;left&quot;<br>    return tokenizer<br><br># Load TinyLlama 1.1B model from Hugging Face<br>tinyllama_model_name = &quot;TinyLlama/TinyLlama-1.1B-Chat-v1.0&quot;<br><br>print(&quot;Loading tokenizer...&quot;)<br>tokenizer = load_tokenizer(tinyllama_model_name)<br><br>print(&quot;Loading model...&quot;)<br>model = load_model(tinyllama_model_name)<br><br>print(&quot;Creating custom model...&quot;)<br>custom_model = TinyLlamaModel(model, tokenizer)<br><br># Test model generation<br>print(&quot;\nTesting model generation:&quot;)<br>prompt = &quot;&quot;&quot;<br>The following are multiple choice questions (with answers) about abstract algebra.<br><br>Find all c in Z_3 such that Z_3[x]/(x^2 + c) is a field.<br>A. 0<br>B. 1<br>C. 2<br>D. 3<br>Answer:&quot;&quot;&quot;<br><br>test_output = custom_model.generate(prompt)<br>print(f&quot;Generated output: &#39;{test_output}&#39;&quot;)<br><br># Test batch generation<br>print(&quot;\nTesting batch generation...&quot;)<br>test_prompts = [<br>    prompt,<br>    &quot;What is 2+2? A. 1 B. 2 C. 3 D. 4 Answer:&quot;,<br>    &quot;Capital of France? A. London B. Berlin C. Paris D. Rome Answer:&quot;<br>]<br><br>batch_outputs = custom_model.batch_generate(test_prompts)<br>for i, output in enumerate(batch_outputs):<br>    print(f&quot;Batch output {i+1}: &#39;{output}&#39;&quot;)<br><br># Run MMLU benchmark with very light settings<br>print(&quot;\nRunning MMLU benchmark...&quot;)<br>benchmark = MMLU(<br>    tasks=[MMLUTask.HIGH_SCHOOL_COMPUTER_SCIENCE],  # Only one task<br>    n_shots=2  # Reduced shots for smaller model<br>)<br><br>try:<br>    benchmark.evaluate(model=custom_model, batch_size=1)  # Batch size 1<br>    <br>    print(&quot;\nBenchmark Results:&quot;)<br>    print(f&quot;Task Scores: {benchmark.task_scores}&quot;)<br>    print(f&quot;Overall Score: {benchmark.overall_score}&quot;)<br>    <br>    # Print detailed predictions<br>    print(&quot;\nSample Predictions:&quot;)<br>    for i, (input_text, prediction) in enumerate(list(benchmark.predictions.items())[:3]):<br>        print(f&quot;Prediction {i+1}:&quot;)<br>        print(f&quot;Input: {input_text[:100]}...&quot;)<br>        print(f&quot;Prediction: {prediction}&quot;)<br>        print(&quot;---&quot;)<br>        <br>except Exception as e:<br>    print(f&quot;Error during benchmark evaluation: {e}&quot;)<br>    print(&quot;Trying with even smaller settings...&quot;)<br>    <br>    # Fallback: try with minimal settings<br>    benchmark = MMLU(<br>        tasks=[MMLUTask.HIGH_SCHOOL_COMPUTER_SCIENCE],<br>        n_shots=1<br>    )<br>    benchmark.evaluate(model=custom_model, batch_size=1)<br>    print(f&quot;Fallback overall score: {benchmark.overall_score}&quot;)</pre><h4>Executing the test</h4><p>Executing the custom LLM model benchmark test using the above code by running the following command.</p><pre>deepeval test run test_custom_model.py</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p89smFT-nNza_AXwlz0-7g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3U6UPA-I23yN04ORh4XsRw.png" /></figure><p><em>Thanks for reading this article !!</em></p><p><em>Thanks Gowri M Bhatt for reviewing the content.</em></p><p>If you enjoyed this article, please click on the clap button 👏 and share to help others find it!</p><p>The full source code for this tutorial can be found here,</p><p><a href="https://github.com/codemaker2015/deepeval-experiments">GitHub - codemaker2015/deepeval-experiments: This repository contains hands-on experiments with DeepEval, an open-source evaluation framework for testing Large Language Models (LLMs)</a></p><h4>References</h4><ul><li><a href="https://deepeval.com/docs/getting-started">Quick Introduction | DeepEval - The Open-Source LLM Evaluation Framework</a></li><li><a href="https://www.confident-ai.com/docs/llm-evaluation/quickstart">Quickstart | Confident AI Docs</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d7272b6c2634" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>