Integration of AI into legal processes and tools has accelerated at a rapid pace over the last few years. It started with simple chat assistants and has now progressed to agentic AI. One thing that underpins both of these approaches and ensures quality and reliable outputs is retrieval.
Retrieval presents significant engineering and machine learning challenges. Many AI integrations succeed or fail because of it. Surfacing the right context is a necessary prerequisite for a correct result. This is why at Filevine we see retrieval as a cornerstone of our legal AI solutions.
In this article, we discuss retrieval from a machine-learning and quality standpoint, focusing on vector and keyword search. In follow-up articles, we’ll discuss how we scale this retrieval to millions of documents and how we built an agentic system on top of it.
How Your AI Actually Finds the Right Answer
No matter the AI tool you use to help you with your legal work, if you are working on a complex project, you will inevitably need a retrieval system to surface the most relevant content to your AI enabling it to present the user with most relevant information.
The traditional solution employed for this is called Retrieval Augmented Generation (RAG). In the RAG system, documents are converted into numerical representations (vectors) by using embedding models[1]. These vectors are then indexed in a vector database that is used to search semantically for the context during query time. A stronger reranker model then scores each candidate against the query to refine the results. This search flow is visualized in Figure 1.
Figure 1. Traditional RAG pipeline with reranker.
The Current Industry Standard
Semantic or vector search excels at finding similar concepts or topics with respect to your query. But this does not meet all the needs legal professionals have. Therefore, keyword search is used in conjunction with the vector search to enable precise lexical retrieval. The two result sets are merged using reciprocal rank fusion (RRF), which boosts documents that rank highly in both searches. This architecture is commonly described as hybrid retrieval and is illustrated in Figure 2.
Hybrid retrieval is a solid baseline, but at Filevine we have found it falls short of the standard the legal domain demands.
Figure 2. Industry standard hybrid search pipeline.
Going Beyond the Baseline
Handling Metadata
Metadata is information that describes a document apart from its content. Legal documents come with a variety of metadata, such as the number of citations or the court level. Many AI systems allow this metadata to be used as a “hard filter” when performing retrieval.
For most legal research, hard filtering’s limitations can lead to errors and missed documents. At Filevine we are able to support soft filtering which we believe is more sophisticated and aligns better with practitioners’ needs.
Imagine that you want to search for a document semantically but at the same time you want it to ideally come from a court of appeals court or higher. You cannot express such a search query with hard filters in a vector database, because we are trying to search both by the semantic meaning and metadata relationship [2].
Metadata Boosting
The simplest way to achieve this is to rerank the documents based on the relationship that we are interested in. So we first perform a similarity search and then proceed to prioritize documents from the court of appeal (represented by number 2) or higher. See this flow in Figure 3. Reranking by metadata is already supported in vector databases like Zilliz (Boost Ranker). But what if the similarity search only returns documents from the trial court because they are most similar? We end up with empty search results.
Figure 3. Naive metadata reranking of candidate documents.
Metadata Fusion
To solve this, at Filevine we can embed metadata information directly into the vector representation[3]. This way we are able to perform more expressive queries than traditional RAG pipelines. To achieve this we use specialized encoder models and create custom vector embeddings. Then, at search time, we can dynamically weigh the metadata to prioritize factors identified by legal professionals. See this flow in Figure 4.
Figure 4. Using metadata fusion to prioritize metadata during vector search.
Improving Keyword Search Capabilities
There are voices in the industry that claim that keyword search is all you need. At Filevine we do not find this to be the case, so we combine it with vector search. Moreover, the traditional search has a number of deficiencies that need to be actively addressed for the legal domain. Mainly, it does not support synonyms or homonyms and struggles with specific legal terminology.
To remedy this, we can employ learned sparse retrieval, which enables us to handle synonyms, homonyms, and special legal terminology. This retrieval enhances sparse keyword representations with information from embedding models. This provides two benefits: term weighting, which scores how important each word is in context, and term expansion, which surfaces related words absent from the text. See Figure 5. There are existing production-ready models like SPLADE, but these are general models that are not fine tuned for the legal domain. At Filevine we can leverage our own custom learned sparse retrieval models tuned specifically for legal work to deliver above-industry-standard lexical search.
Figure 5. Using information from embedding model to enhance sparse representation.
Making Vectors More Expressive and Explainable
Traditional embedding models produce a single vector for a given chunk of text. This final vector is obtained by pooling (averaging) the model's internal per-token vectors, which inherently loses information. ColBERT approach instead keeps a vector per token and scores a query against a document with a MaxSim operator: for each query token it takes the maximum similarity over all document tokens, then sums these. Because far less information is lost, the result is typically a more expressive and interpretable search. An illustration of interpretability between query and document is shown in Figure 6.
Figure 6. Illustration of how ColBert approach is able to provide retrieval interpretability.
Searching over multi-vector representation does not have to be done on a token level. Instead, we can operate on a paragraph level in a document, for example.
Traditionally, multi-vector search was used for reranking due to its higher latency compared to single-vector search. But with the introduction of the Google Muvera index, multi-vector representation is usable for default vector search without the latency issues.
At Filevine we are able to leverage multi-vector representation for both re-ranking and retrieval search that provides high-quality, interpretable results for our subscribers.
To see what advanced legal AI can do for you, try LOIS for free.
[1] Nowadays both encoder models (BERT) and decoder models (Qwen3) are used for vectorization.
[2] Mathematically, we want to search by this relationship cosine(query, doc) × 0.5 + rating × 0.5
[3] This way the relationship cosine(query, doc) × 0.5 + rating × 0.5 is directly fused into vectors. This process of fusion is introduced in Utilizing Metadata for Better Retrieval-Augmented Generation.