Vector Databases FAQ
A vector database stores embeddings so you can retrieve similar text — and many RAG systems do not need a new database to start.
A vector database indexes embedding vectors so you can fetch the chunks most similar to a query. For RAG it is an index, not a source of truth; hybrid search with keywords is the production default, and Postgres with pgvector is enough for many first systems.
What a vector database is
If you already understand search indexes, you are most of the way there — with cosine similarity instead of only terms.
What is a vector database?
A vector database stores numeric embeddings — lists of floats that represent text, images, or other objects — and retrieves the nearest neighbours of a query vector. Approximate indexes (HNSW, IVF) make that search fast enough for interactive RAG. Metadata filters sit beside the vectors. It is a retrieval engine, not a replacement for your warehouse, wiki, or document store.
Do I need a vector database for RAG?
You need an embedding index. That can be a specialist vector database, or pgvector, or another search engine with a dense vector field. For a first workflow on thousands of chunks, Postgres or your existing search cluster is often enough. You need a dedicated store when latency, scale, or multi-tenancy exceeds what that cluster can hold — measured, not guessed.
Pinecone vs Weaviate vs pgvector — which should we pick?
Pick the one your operators can run and that hits recall and latency on your questions. pgvector wins when Postgres is already the platform. Pinecone, Weaviate, Qdrant, OpenSearch, and others win on managed scale, hybrid features, or ops you do not want to own. We are store-agnostic. A bake-off without a golden retrieval set will pick a logo. We will not pretend there is a universal winner.
How do embeddings get stored?
Each chunk is passed through an embedding model; the vector, the raw text (or a pointer to it), and metadata (source URL, heading path, ACL, timestamps) are upserted into the index. Deletes and updates follow the source’s change feed. You store enough text to cite; you do not treat the index as the only copy of the document. IDs must be stable or you will duplicate chunks on every sync.
What is hybrid search, and do we need it?
Hybrid search combines keyword matching (BM25) with vector similarity, then usually a re-ranker. You need it if people search with identifiers, rare proper nouns, or exact phrases — which in enterprises they do. Vector-only RAG fails those queries in week two. Reciprocal rank fusion is a common merge. The hybrid-retrieval systems page is the architecture; this is the store’s job in that pipeline.
How is a vector database different from a regular database?
A relational database answers exact predicates and joins. A vector index answers “nearest to this embedding,” approximately. Many products now do both in one engine (Postgres, OpenSearch). You still need transactions, backups, and access control from a real database for the system of record. Putting customer rows only in a vector store because “AI” is a category error.
How much data before we need a dedicated vector DB?
There is no honest universal cutoff. Tens of thousands of chunks often run fine on pgvector with a sensible index. Millions of chunks, tight p95 latency, or many tenants usually push you toward a specialist or a tuned search cluster. Measure recall@k and p95 on production-shaped queries. We will not sell a new database because the chunk count sounded large in a meeting.
How the index gets built
Chunking, metadata, and the change feed decide quality. The vendor logo decides less than the sales deck implies.
How do you choose a vector store?
Constraints first: must it stay in your VPC, who already operates it, do you need hybrid in one engine, what is the latency SLO. Then run your golden questions against two candidates. Ops and recall beat benchmark blog posts. Managed vs self-hosted is a staffing choice. ReinforcedX will implement against the one you can live with after we leave.
How do metadata filters and ACLs work?
Each chunk carries attributes — tenant, group, document ACL, language. At query time you filter to what the asking user can already open, then run vector/keyword search inside that subset (or filter after, with care). Filtering after a global search can leak existence. Identity comes from your IdP, not from a shared bot user. This is permissioned retrieval, not an optional extra.
How do you keep the vector index in sync?
Incremental upserts from the source change feed, not a nightly full crawl. Deletes must land or you will retrieve withdrawn policy. Freshness is an SLO: minutes from edit to searchable chunk. A vector DB that is perfectly tuned and a week stale is a wrong-answer machine. Webhooks, changes.list, or a short poll — whatever the source offers.
Do we have to care about HNSW, IVF, and dimensions?
Someone on the implementation does. Dimensions are fixed by the embedding model; mix models and you cannot compare vectors. HNSW and IVF are approximate indexes with recall/latency knobs (ef, nprobe, M). Wrong knobs look like “the database is bad.” We set them against your recall set. Application engineers should not have to tune this weekly if the runbook is right.
What happens when we change the embedding model?
Every stored vector is in the wrong space. You re-embed, usually with dual-write or a blue/green index, then switch queries and check recall. This is a migration, not a config flag. It is why we swap generators more freely than embedders. Budget compute and a freeze window. Partial re-embeds produce silently bad neighbours.
Can we just use Postgres and pgvector?
Yes, and we often do for a first RAG agent: one backup regime, one IAM story, hybrid via Postgres full-text plus vectors. You will outgrow it if p95 search or scale demands a dedicated engine — that is a measured swap, not a rewrite of the app if you hid the store behind a retriever interface. “Just use Pinecone” is not a requirement.
Which embedding model should we use?
Whichever scores best on your held-out retrieval set and you are willing to operate — we do not lock you to a vendor. Domain models help on jargon; they complicate re-embedding. Dimension count is a storage/latency trade, not a quality guarantee. The RAG FAQ covers the generator side; here the embedder is the schema of the index.
Running the store in production
Backups, tenancy, and p95 matter as much as demo recall. This is still a database.
Where does the vector database live?
In your cloud, next to the agent. Credentials stay in your secret manager. We do not host your corpus. Managed vector SaaS is acceptable when your security team accepts that data plane; otherwise self-host in the VPC. Contents are not used to train shared models. If a vendor cannot meet residency, they are off the list.
What happens if retrieval is slow?
The whole agent waits, or you time out and refuse. Causes: missing indexes, over-wide filters, huge k, a re-ranker on too many candidates, cold starts, or noisy neighbours from bad chunking. Traces must show retrieve latency separate from generation. Fix the index and k before buying GPUs for the generator. p95 is the number; average will lie.
How do you evaluate whether the vector DB is doing its job?
Recall of the known-relevant chunk at k, plus latency. Keep that separate from answer faithfulness. A golden set of real questions with labelled passages is the instrument. If you only score the final sentence, you cannot tell a store miss from a generator miss. Regression gates fail the release when recall drops after a reindex or parameter change.
How do you handle multi-tenancy in a vector store?
Tenant (or user) identifiers on every chunk, enforced in the query planner, not as a polite filter the app might forget. Some engines offer namespaces or collections per tenant; others rely on metadata. Cross-tenant search is a bug. For strict isolation, separate indexes. We test with a second tenant’s document and expect a no-result, not a redacted snippet.
What actually drives vector-database cost?
Stored dimensions × rows, replicas, query QPS, and re-embedding jobs. Hybrid and re-ranking add compute outside the store. We do not quote vendor list prices that will stale. You pay the cloud or the store vendor directly. The usual surprise is a full reindex after an embedding swap, not the first month of storage. Design the retriever so you can change stores without rewriting the agent.
Who owns the index after handover?
You do. Connection strings, schemas, backup policy, and the sync jobs are in your runbook. Thirty days of on-call is included. A store we introduced is still your database: snapshots, IAM, and deletion follow your standards. We do not keep a shadow copy of the embeddings.
When you should not add one
A new database is an ops commitment. Earn it with a recall number, not with a trend.
When should we not use a vector database?
When the job is exact lookup (order ID in a warehouse), when you have a handful of documents a prompt can hold, when keyword search already returns the right passage, or when nobody will operate another data store. Also when you cannot label a retrieval set — you will not know if the index works. Start simpler; graduate when measurements say so.
Do we need a vector DB if we only have 500 documents?
Usually no. Chunk them, embed into pgvector or even a local index, add BM25, and spend the effort on permissions and evals. Specialist vector SaaS for a small corpus is optional convenience, not architecture. The failure at that size is bad chunking and no golden questions, not missing HNSW.
Is a vector database for agents different than for search?
The engine can be the same. Agents add write-back of memories, tool-scoped retrieval, and stricter ACLs, and they issue more iterative queries (agentic RAG). Memory stores need write policies and forgetting; document indexes need freshness. Do not dump long-term agent memory into the same collection as the employee handbook without metadata that separates them.
Should we use a knowledge graph instead of a vector DB?
Sometimes, for multi-hop entity questions where chunks lose relationships. GraphRAG still often uses vectors for entry points. Graphs add extraction cost and drift. We add a graph when the golden set shows hop failures hybrid search cannot fix, not because “knowledge graph” appeared on a roadmap. Hybrid retrieval remains the default first system.
What usually goes wrong with vector databases in RAG?
Vector-only search, no ACLs, a full crawl that indexes drafts, never deleting, and changing embedders without a reindex. Teams then blame the model. The RAG FAQ lists the same four in the first 90 days. Rescue work starts by measuring recall on real questions, then deciding if the store was even the problem.
Vector databases store embeddings and run nearest-neighbour search. RAG uses them to fetch candidate chunks; it still needs BM25, filters, ACLs, and a re-ranker. You do not always need a specialist store: pgvector in Postgres is a sound start when operators already run it. Dedicated engines help at scale, with multi-tenancy, or with stricter latency. ReinforcedX is store-agnostic, deploys in your VPC, and measures recall on your questions before arguing brands. Re-embedding, stale indexes, and a single service account are the usual failures, not the missing vendor.
Keep reading
Ready to refine
your workflow?
Share your current process. We’ll help you identify what can be automated — and where efficiency can be reclaimed.