The Hidden Cost of LLM Indexing: Why AI Search Infrastructure Bills Add Up

When companies build AI search features, they often focus on the cost of training or running the LLM itself. But there’s a quieter, steadily accumulating expense: the cost of indexing. Indexing is the process of preparing, chunking, embedding, and storing data so that a retrieval-augmented generation (RAG) system can find relevant information at query time. This layer, which connects your private data to a frozen model, carries its own infrastructure price tag.

For a million documents, embedding generation alone can cost between $20 and $130 in API fees, depending on the model. Storage for those vectors runs roughly $430–$580 per month on managed services. And if your data changes often, those costs recur. Understanding where the money goes is essential for anyone building AI search at scale—whether you’re a startup or an enterprise team.

What Actually Happens When You Index Data

Indexing isn’t training. The model weights stay frozen. Instead, you’re building a searchable representation of your data. The process involves several steps, each with its own cost drivers:

• Chunking and preprocessing: Splitting documents into pieces, cleaning text, deduplicating, extracting metadata. This is CPU-heavy but usually the cheapest part.
• Embedding generation: Running each chunk through an embedding model to convert it into a vector. This is where the significant compute or API costs appear.
• Vector storage: Storing those embeddings in a vector database like Pinecone, Weaviate, or pgvector. Costs scale with storage size, index type, and replication.
• Index maintenance: Updating, deleting, and re-embedding changed documents. For frequently updated corpora, this becomes a recurring expense.
• Retrieval compute: At query time, vector search plus optional reranking. This cost scales with query volume and index size.

The Numbers Behind a Million Documents

To make this concrete, consider a corpus of 1 million documents, each about 1,000 tokens long. That’s 1 billion tokens total.

Embedding generation costs:
– OpenAI’s text-embedding-3-small runs $0.02 per 1M tokens. For 1B tokens, that’s $20.
– The larger text-embedding-3-large costs $0.13 per 1M tokens, bringing the bill to $130.
– If you self-host on a GPU like an A10G, which can embed roughly 1M tokens per hour, you’d need about 1,000 GPU-hours. At $1.50/hour on AWS, that’s $1,500—one-time, but you also need to manage the infrastructure.

Vector storage costs:
– A 1M-document index with 1,536-dimension vectors (like OpenAI’s ada-002 style) consumes about 6–8 GB of storage.
– Pinecone’s serverless pricing is roughly $0.10 per GB-hour. That’s about $0.60–$0.80 per hour, which translates to $430–$580 per month—just for storage, before any query traffic.
– Self-hosting on a modest EC2 instance might run $100–$300 per month, but you’re responsible for uptime and scaling.

The bigger picture: For many production RAG systems, indexing is a one-time or amortized cost, while inference and query processing dominate at scale. But if your data is dynamic—news, legal filings, support tickets—indexing becomes a recurring line item that can rival inference costs.

Why Indexing Became Expensive

Pre-LLM search relied on inverted indexes (like BM25 or Elasticsearch)—cheap, deterministic, and well-understood. LLM-era search adds semantic embeddings, which are expensive to generate and store. The rise of RAG in 2023 made indexing a first-class concern: enterprises want LLMs to answer from their own data, which requires building and maintaining a private index.

Hybrid search is now the norm, meaning you often pay for both a traditional keyword index and a vector index. That’s double the storage and maintenance.

The Managed vs. Self-Hosted Tradeoff

Managed services like Pinecone, Weaviate Cloud, or Azure AI Search offer convenience and scalability, but you pay a premium. Self-hosting with open-source tools like Milvus or pgvector can cut costs, but you take on operational overhead—monitoring, scaling, and tuning.

A hybrid approach is common: use managed embeddings for quality, but store vectors in your own Postgres with pgvector to save on storage. Or vice versa. There’s no single right answer; it depends on your team’s expertise and the criticality of the system.

The ‘Indexing Tax’ in AI Search Startups

AI search products like Perplexity, You.com, or Glean must index the open web or enterprise corpora continuously. This creates a recurring infrastructure tax that traditional search engines didn’t have—they used cheaper lexical signals. Some startups have pivoted to cached or static indexes, or hybrid sparse-dense models like SPLADE or ColBERT, to reduce the burden.

This tax is a reason many AI search companies need venture funding at high valuations—they’re burning cash on infrastructure before revenue scales.

Practical Ways to Control Indexing Costs

• Choose the right embedding model: For many tasks, smaller models like text-embedding-3-small perform adequately at a fraction of the cost.
• Optimize chunking: Overlapping chunks increase storage and embedding costs. Fine-tune chunk size and overlap for your use case.
• Incremental indexing: Instead of re-embedding everything, only process changed documents. This can drastically reduce compute.
• Consider dimensionality reduction: Some vector databases support quantization or dimensionality reduction, cutting storage costs.
• Use a hybrid index: Pair a cheap keyword index with a smaller vector index to reduce costs without sacrificing quality.

The Bottom Line for Teams

Indexing costs are real, but they’re not prohibitive if planned carefully. For a small corpus of a few thousand documents, the cost might be pennies. At millions of documents, you’re looking at thousands of dollars per month, depending on your choices.

Don’t let the cost surprise you. Model your expected corpus size, update frequency, and query volume. Then compare managed vs. self-hosted options. With the right architecture, you can keep the index affordable while still delivering the semantic search your users expect.

Indexing is the silent partner in AI search—often overlooked until the bill arrives. By understanding its components and costs, you can make informed decisions that keep your infrastructure efficient. The key is to match your indexing strategy to your actual needs, not to the hype.

Summary

• Indexing is the process of preparing and embedding data for RAG systems—it’s separate from training and inference.
• For 1M documents, embedding costs range from $20 (using text-embedding-3-small) to $1,500 (self-hosted GPU), while vector storage runs $430–$580/month on managed services.
• Frequent data updates turn indexing into a recurring cost that can rival inference.
• Managed services offer convenience; self-hosting saves money but adds operational burden.
• Strategies like incremental indexing, model selection, and hybrid search can reduce expenses.

FAQ