This project is a comprehensive, actionable blueprint for executing complex embedding migrations for large-scale Retrieval-Augmented Generation (RAG) systems. It provides functional, production-ready code examples and deployment scripts to guide you through the entire migration lifecycle.
Upgrading core embedding infrastructure without downtime or regressions is a significant challenge for senior AI/ML engineers. This framework demonstrates how to approach this problem systematically.
The project is structured as a series of steps (or phases), each building upon the last:
- Initial Generation (
01-generation/): (COMPLETED) Fully functional pipeline that ingests raw product data from BigQuery, generates initial embeddings usingtext-embedding-004(viagoogle-genaiSDK), and stores them in a highly-available AlloyDB for PostgreSQL vector database. Highly parallelizable execution using Cloud Run Jobs. - Semantic Search UI (
02-ui/): A Next.js application to demonstrate semantic search capabilities using the generated embeddings. - Embedding Migration (
03-migration/): Tools and scripts supporting batch migration of embeddings to a newer model (gemini-embedding-001).
- Compute: Google Cloud Run Jobs (for scalable, parallel batch processing)
- Database: AlloyDB for PostgreSQL (with
pgvectorfor vector storage) - AI/ML: Google Vertex AI (
text-embedding-005viagoogle-genaiSDK) - Data Source: BigQuery (
bigquery-public-data.thelook_ecommerce.products) - Infrastructure as Code: Terraform
To get started, navigate to the 01-generation directory and follow the instructions in its specific README to deploy the initial embedding pipeline.
This project is licensed under the Apache License 2.0.
This is not an officially supported Google product