Back to Guides9 min read • Updated Sept 2026
System Design

RAG Cost Architecture: Ingestion, Chunking & Vector Retrieval

Retrieval-Augmented Generation (RAG) splits costs between one-time corpus embedding and recurring query synthesis. Here is how to model both phases accurately without underestimating vector database and context amplification fees.

1. The Two-Phase RAG Cost Equation

RAG architecture economics are governed by two distinct cost vectors:

  1. Corpus Ingestion (One-Time / Periodic): Chunking your knowledge base and transforming it into vector embeddings via models like text-embedding-3-small ($0.02/1M tokens) or Voyage-3 ($0.12/1M tokens).
  2. Query & Generation (Recurring per search): Embedding the user's query, retrieving the top-K chunks, prepending them to the LLM system prompt, and generating an answer.

2. The Top-K Context Multiplication Effect

Many engineers budget for the user's 100-token question and the 300-token answer, forgetting that retrieving 5 chunks of 500 tokens adds 2,500 input tokens to every single query.

At 100,000 queries per month, that context amplification translates to 250,000,000 extra input tokens!

3. RAG Optimization Strategies

  • Prompt Caching on Golden Documents: If 80% of user queries search the same core documentation, cache the static reference corpus in memory.
  • Reranking (Cross-Encoders): Retrieve top-20 chunks cheaply with BM25/hybrid search, rerank with Cohere Rerank or BGE-Reranker, and only send the top-3 most relevant chunks to the expensive generation LLM.
← Model your RAG pipeline in our RAG CalculatorNext: AI Agent Cost Modeling →