Embeddings FAQ
Embeddings turn text into vectors so you can retrieve by meaning — and they go stale the moment you change the model or the documents.
An embedding is a numeric vector that represents a piece of text (or an image) so that similar meanings sit close together in space. You use them to retrieve, cluster, and route — not to generate answers. Changing the embedding model means you must re-embed the corpus; mixing models in one index is how recall silently dies.
What embeddings are
If you only need one sentence: they are coordinates for meaning, useful only with a retrieval eval and a plan to rebuild them.
What are embeddings?
Vectors — lists of numbers — produced by a model so that texts with similar meaning have similar coordinates. You compare them with cosine similarity or inner product to retrieve neighbours. They do not “understand” in the way a person does, and they do not generate an answer. They are an index feature. If two invoices embed close together, retrieval will treat them as related; that can be useful or a permission incident.
What is an embedding model?
A model whose output is a vector, not a paragraph. OpenAI, Cohere, Voyage, E5, and many open weights ship them. Each has a dimension, a token limit, a language coverage, and a training distribution. You cannot mix their vectors. We are model-agnostic: we pick against your retrieval eval, not a vendor chart, and we run the chosen model in your account or VPC.
How are embeddings used in search and RAG?
You chunk documents, embed each chunk, store the vectors, embed the query, and fetch the nearest chunks for the generator to read. That is the retrieve half of RAG. Keyword search still sits beside it for identifiers. The RAG FAQ covers grounding, citations, and empty-retrieval behaviour. This page is the vector layer those systems sit on.
How many dimensions do we need?
Whatever the model emits, or a documented Matryoshka / truncation size that model was trained for. You do not pick 1536 because it sounds serious. Shorter vectors save storage and can be enough; shorter vectors from a model not trained to shorten will lose recall. We treat dimension as part of the model choice and measure recall@k on your questions, not on a theoretical bit rate.
Do we need a vector database?
You need a place to store vectors and search them. For many internal corpora that is Postgres with pgvector, or the warehouse you already run. Specialist stores (Pinecone, Weaviate, Qdrant, and others) earn their keep at scale, with hybrid features, or when ops wants them. Starting a new SaaS because a tutorial said “vector DB” is how you get a second permission system. Measure first.
When should we re-embed everything?
When you change the embedding model, its version, the chunking scheme, or a large slice of the corpus. Incremental embeds handle new documents. A silent vendor upgrade of the embedder is a full rebuild if the space moved. Plan the job: it is tokens, GPUs, and an eval to prove recall did not fall. We put that pass in the runbook rather than treating the index as immortal.
Are domain-specific embeddings worth it?
When generic models fail on your jargon, codes, or language mix, yes — a domain or fine-tuned embedder can lift retrieval where it is the bottleneck. When your miss is permissions, chunking, or missing files, a new embedder will not help. We A/B on your golden questions. Fine-tuning an embedder is a data project; you own the resulting weights.
Choosing and running them
The choice is an eval on your questions, not a leaderboard screenshot.
How do you choose an embedding model?
Run two or three candidates on a frozen set of real queries with labelled relevant chunks. Score recall@k, MRR, and latency. Include multilingual and identifier-heavy queries. Price is your API bill or GPU time, paid to the provider with no markup from us. A model that wins a public MTEB chart and loses on your SKUs is the wrong model.
Is hybrid search better than embeddings only?
In production, almost always. Dense retrieval misses exact tokens; keyword misses paraphrase. Reciprocal rank fusion plus a cross-encoder re-ranker is the usual stack. Pure embedding search is a prototype. If your corpus is short marketing blurbs, you might not need hybrid; if it has invoices and error codes, you do.
Do multilingual embeddings actually work?
They work when the model was trained for those languages and your eval includes them. Cross-lingual retrieval (question in French, chunk in German) is a separate, harder test. Do not assume an English-strong model is “fine for EU.” We split the golden set by language and refuse a single average that hides a dead locale.
Who owns the vectors?
You do. Indexes, chunk maps, and embedder configs are yours at handover, same as weights and runbooks. They live in your cloud. We do not keep a shadow copy to lock you in. If you leave, you still have an index your engineers can rebuild from the runbook.
What does it cost to embed a large corpus?
Tokens or GPU hours, at your provider rates, plus storage. We will not invent a per-million-token price here; those change. The surprise cost is re-embedding after a model swap, not the first pass. We quote implementation as a fixed fee and leave inference/embed spend on your bill with no markup. A cost forecast for the first rebuild is part of discovery.
Does chunking matter more than the embedding model?
Often yes. Tables split mid-row, 512-token windows that cut a clause, and chunks with no heading path will beat a better embedder. Structure-aware chunking, overlap, and parent-child retrieval are the usual fixes. Measure with the same golden questions when you change either chunking or the model, or you will not know which lever moved recall.
Can we mix embedding models in one index?
No. Different models, different spaces. You can run two indexes and fuse ranks, which is a retrieval-architecture choice with twice the embed cost. You cannot concatenate or average foreign vectors and expect meaning. If a vendor “upgrades” the model under the same name, treat it as a new model until the eval says the space is stable.
Embeddings in production
Production is ACL-filtered retrieval, a rebuild plan, and an eval that notices drift.
How do permissions work with embeddings?
Filter candidates by the asking user’s ACL at query time, then retrieve. A single service account over the whole vector store will leak neighbouring confidential chunks. Missing permission should look like a miss, not “a document exists but you cannot see it.” Identity comes from your IdP. This is the same rule as the rest of our RAG work.
What happens when the embedder updates?
Recall can drop without a generator change. Pin the model version, monitor retrieval metrics on a frozen query set, and rebuild when you intentionally move. If the vendor has no pin, freeze by hashing a canary set of vectors and paging on movement. Silent embedder updates are as real as silent chat-model updates.
How do you evaluate retrieval, not just answers?
Label which chunks should come back for real questions. Report recall@k, precision, and (separately) whether the generator stayed faithful to what was retrieved. Improving the chat score while recall falls is how you ship a fluent liar. Fifty or more real questions is the start of a golden set, not a nice-to-have.
Do we need late chunking or ColBERT, or is dense enough?
Dense embeddings plus hybrid and a re-ranker is the default we ship. Late chunking, ColBERT, or multi-vector retrieval help some long-document and identifier cases; they add ops cost. We add them when the golden set says dense-plus-hybrid has plateaued, not because a paper was trending. Start simple; keep the eval.
Is PII safe inside embeddings?
Not automatically. Embeddings can leak via inversion research and, more practically, via retrieved neighbours that still contain the raw text you stored next to the vector. Treat the chunk store as personal data if the source was. Redact before embed when your DPO requires it, knowing recall on names may fall. VPC and access control matter more than hoping the vector is “hashed.”
Where do embeddings live?
In your cloud, next to the chunk text and ACL metadata. We do not copy the corpus onto ReinforcedX infrastructure. Client data is not used to train a shared embedder. Thirty days of on-call covers the index after handover. Rebuild jobs run on your keys and your bill.
When vectors are the right tool
Not every search problem wants an embedding. Some want a catalogue, a graph, or a SQL warehouse.
Can we embed images and scanned PDFs too?
Yes, with multimodal or document embedders, plus OCR where you still need text. Citations should point at a page or figure, not a mystery vector. This is slower and more expensive than text-only RAG. If the PDF is already a clean text extract, start there. Do not buy a vision embedder to avoid fixing the parser.
Should we use a knowledge graph instead of embeddings?
Use a graph when the questions are about entities and relations you can extract reliably (“who owns this policy”). Use vectors when the questions are paraphrase over prose. Many systems do both: hybrid retrieval plus a small entity graph. GraphRAG is a programme of extraction quality; it is not a way to skip chunking.
Should we fine-tune the embedding model?
After you have labelled query–chunk pairs and a generic model has plateaued. Contrastive fine-tunes need real queries, not invented ones only. You own the resulting adapter. We will not fine-tune first; we will spend week one on chunking, hybrid search, and a golden set. That sequence is how you avoid training the wrong space.
Is product search the same as document RAG?
No. Product search cares about catalogue fields, faceting, stock, and business ranking. Document RAG cares about passages and citations. You can share an embedding model; you should not share the ranking stack blindly. We will say if you asked for an intranet agent and actually need a shop index.
When are embeddings the wrong tool?
When the question is an exact lookup, a SQL aggregation, or a workflow action. Embed the policy; query the warehouse with a tool. Also when you have twenty documents — grep and a prompt will beat an index. We will refuse a vector project whose real need is permissions or a missing API. Four weeks is for a retrieval system you can eval, not a fashion index.
Embeddings are the retrieval primitive under RAG and semantic search: chunk text, embed it, store vectors, retrieve nearest neighbours, often mixed with keyword search. Dimension count is a model property, not a quality slider you pick. Domain models help when generic embedders miss jargon. Re-embed when the model or the documents change. ReinforcedX builds the index in your VPC, with permissions at query time. You own the vectors and the eval questions. Postgres plus pgvector is enough for many corpora; a specialist store is a choice, not a religion.
Keep reading
Ready to refine
your workflow?
Share your current process. We’ll help you identify what can be automated — and where efficiency can be reclaimed.