We've rebuilt the contract review system behind LOIS for Word three times in three years.
Our first version followed a fixed workflow. Then we built an agent that could reason through contracts on its own. Today, multiple agents can review and redline a contract before a lawyer ever opens Word.
We ended up with three very different architectures, each shaped by what AI could do at the time and what we'd learned from earlier versions.
Review Classic was fast but missed context
When a company negotiates a contract, its lawyers review the document for potential risks and propose changes to address those risks. They often follow a playbook that defines the company's preferred terms and how they handle compromises. We built our first version, Review Classic (“Classic”), around those playbooks.
Our first version used a rule-based LLM workflow to interpret legal language and suggest edits based on playbook requirements. The models could understand and edit contracts, but our code still controlled the review process. To make that work, we had to define how the system would apply each requirement.
We joke that lawyers’ favorite answer is “it depends.” Well, there’s a reason for that. Law is fundamentally semantic: the same guidance can apply differently across deals. But our customers needed consistent reviews before they could trust LOIS for Word to automate that work. We had to turn context-dependent judgment into a workflow with explicit, testable decisions.
Our key product insight was to distill these complex, loosely defined playbooks into explicit pass-or-fail rules that LLMs could apply to each contract. The model still had to interpret the legal language, but each rule specified what counted as passing. We could measure how reliably the system applied each rule and improve the review over time.
Before LLMs, legal tech companies had to train models on hundreds of examples to recognize particular legal concepts. When we built Classic in 2023, we could write rules in plain English and let the model figure out whether a contract satisfied them.
Every contract review ran in two steps:
- For each playbook rule, the model decided whether the contract passed, failed, or lacked enough information to tell. These results let us measure review quality and improve it over time.
- A separate model call generated suggestions for each failed check. We learned from lawyers’ edits to our suggestions and tailored future recommendations to their writing and negotiation styles.
Classic was fast. Most reviews took less than 15 seconds (compared to minutes to hours for competitors).
But our design had two big limitations.
- Playbooks were difficult to create and maintain. We assumed legal teams had clear guidelines for negotiating common contracts. But often, they didn't. Their existing guidance was usually outdated or rife with contradictions. And even if we perfectly reviewed a contract against a playbook, we could still miss risky terms that the playbook didn't cover yet.
- We used a fixed workflow to decide what the model would review and when. For each rule, we used semantic search to find the most relevant clauses and pass them to the model. But sometimes, search missed something in another clause that should have changed the result. The model couldn't know what it didn't know.
This workflow approach made sense when we first built Classic. Earlier models couldn't reliably decide what to do next, so our code made those decisions for them. But as models improved, we could give them the tools to navigate the contract and make edits on their own.
Agent Review gave the agents control over the review flow
As models got better at reasoning, we wanted to see how much of the review process we could hand over to an agent. Instead of prescribing each step in code, we gave a single agent the tools and autonomy to decide how to review a contract.
The agent chose what to read. In Classic, we relied on semantic search to find relevant text for each playbook rule. If it missed something important, like an exception elsewhere in the contract, the model had no way of knowing.
With Agent Review, an agent reviewing a liability cap could follow an exception into the indemnity section or check how a term was defined before suggesting a change. The playbook still defined the company's requirements, but the agent decided how to apply them.
The agent could flag risks the playbook didn't cover. Instead of only checking predefined requirements, it could reason about the contract and flag issues we hadn't explicitly told it to look for.
Agents could also look beyond the contract. Say a counterparty requires your company to conduct annual security audits, but your playbook doesn't say whether your company already does them. Rather than guess, the agent could decide to search your internal policies or ask you directly.
Customers no longer needed to codify everything they knew before getting a useful review.
The agent also decided where to make edits. In Classic, our code decided which parts of the contract the model could change. With Agent Review, the agent could find the right clause and edit it, even if we hadn't anticipated where the change was needed.
In production, wrong-section edits fell by more than 60%, and application errors dropped from 17% to 9%. Our internal evaluations showed the same reductions.
But giving the agent more autonomy also introduced new problems:
- Coverage. We needed a reliable way to track whether every playbook requirement had been reviewed.
- Consistency. We needed to make sure proposed edits didn’t conflict across the contract, especially when a change to one clause impacted another.
- Latency. A single agent worked through checks sequentially, leaving independent checks waiting on earlier work.
Review 2.0 combines parallel agents with durable orchestration
In Agent Review, a single agent worked through every issue in order. Review 2.0 lets separate agents review different requirements at the same time.
An orchestrator tracks which requirements have been reviewed using saved review records. A reconciliation step handles conflicting edits before they’re applied.
The orchestrator assigns three kinds of work:
- Playbook requirements
- Counterparty edits
- Issues agents discover independently
Each agent keeps its own context. On a 150-page contract, one agent might review confidentiality while another reviews liability. Each agent starts with relevant excerpts from the same saved version of the contract and can read more as needed. We cap how many workers can run at once, and each worker has a context budget.
Keeping progress outside agent conversations
The coordinating agent doesn’t have to remember what’s been completed. Each review item has a saved record of its findings and supporting evidence. Proposed edits are saved too.
Parallel agents read different passages from the same saved snapshot of the contract. Any additional text they request comes from the same snapshot. Stable location IDs link each finding to the contract language that supports it.
The orchestrator reads each finding with its explanation and supporting quotes, then decides which findings to keep or combine. It accounts for every saved worker finding and explains any decision to drop one. The server rejects submissions that leave a finding unaccounted for or count it twice.
- When the coordinating agent’s context fills up, we build a fresh context from the saved assignments and worker reports. We include the contract snapshot when it fits. Otherwise, we provide an outline with tools to read the relevant passages. We can restore saved work and read the same source text, but a fresh model call can still reach a different conclusion
- After a review stops midway, we resume from its saved progress. Suppose a worker flags an uncapped fee and proposes an edit. The orchestrator keeps the finding, but the process stops before committing the proposed edit to the review. We load the saved decision and finish committing the pending result. Completed items are skipped, and the worker doesn’t have to generate the proposed edit again
Lawyers can start reviewing findings as they arrive, while the agent continues working through the contract. The findings and proposed edits stay available after the session ends, so lawyers can reopen an earlier review without running it again.
Reconciling edits across agents
Parallel review gave us a new problem. Each agent could follow its instructions and still propose an edit that conflicted with another agent’s work.
Consider an NDA with a ten-year confidentiality obligation. One agent recommends shortening it to seven years. Another tries to apply the contract’s general two-year term to the same language. Both proposals target the same words. Whichever edit ran last would determine whether the revised clause said seven years or two.
To handle this, we build reconciliation into the editing process:
- Validate that the proposed edits match the original text
- Deduplicate by removing repeated edits
- Detectoverlaps where edits affect the same part of the contract
- Merge overlapping edits by having an agent review the competing suggestions and combine them into one edit
- Verify and apply each edit by checking that it targets the right text, then confirming that it was applied successfully
A failed merge leaves the conflicting edits unapplied. We preserve the available explanations as comments and explicitly mark the edits as unapplied. Other edits can still proceed.
We also built recovery into the editing process. An agent might recommend the right language but try to make the change in the wrong paragraph. The editing tools return an error that the agent can use to retry.
Measuring review quality and speed
We measured how reliably Review 2.0 found issues and produced usable edits.
- Issue detection. Across 50 benchmark contracts, Review 2.0 had 91% recall and 89% precision
- Redline quality. Users rated 87% of Review 2.0's redlines acceptable as written, up from 64% in Agent Review
- Edit application. 96% of accepted edits applied to the correct location on the first try
In a small benchmark with six conflicting edits, parallel merging took 5.5 seconds on average, compared with 30 seconds sequentially.
Reported failures help us build synthetic test cases that reproduce the issue without including client documents or confidential matter details. A legal engineer on our team confirms the expected result before each case enters our evaluation suite. Before release, we test updated agents against the case and related variations to check whether the failure happens again.
Taking review beyond Word
Each rebuild let agents take on more of the review. Today, our customers use LOIS to review and redline contracts arriving through email or Slack. By the time a user opens Word, the first pass is done.
The next chapter is to bring review into every place legal work happens. Lawyers and agents should be able to work on the same contract. When someone picks up the work, they should understand what has happened and how the proposed edits fit together.
As agents take on more work, we also need to rethink what it means for a lawyer to supervise it. A lawyer may be asked to approve a review they didn’t direct. The record should make the agent’s work and the lawyers’ decisions clear enough to review and audit.
We want agents to carry reviews from intake through revisions, asking lawyers for decisions along the way. A firm’s accumulated knowledge should help guide each new review, with enough context to judge when an earlier answer applies. A lawyer shouldn’t have to be in every conversation for the team to benefit from their experience.
I expect we’ll rebuild parts of the system again as models get better. The frontier keeps moving! We'll keep building products that can move with it.
Acknowledgements
Thank you to the following members of Filevine’s team for their work on Playbook Review: Samir Khoja, Matthew Davison, Sona Sulakian, Nicholas Norrell, Niki Khokale, Jake Kohn, and Caroline Farrell.