Case study
Brokers were searching 30,000 PDFs or asking whoever was free. We built an assistant that answers only from the brokerage's own library, shows the passage and version it used, and says so plainly when the answer is not there.
Named only with client approval Figures marked are illustrative Reply within one business day
Copilot · fine-tuned on your data
Live0.94
F1 score
86ms
Inference
12k/d
Requests
OlDevs built an internal assistant for a national commercial insurance brokerage that answers policy-wording questions using only the firm's own library of wordings, endorsements and carrier bulletins.Every answer shows the exact passage and document version it drew from, and the assistant declines to answer when it cannot find support in the source material. It runs inside the brokerage's own cloud tenancy, so no client or carrier data leaves their environment, and an audit log records every question, answer and citation. 78% of wording questions are now resolved without escalating to a senior broker, and 100% of answers are traceable to a document version.
Key facts
The challenge
Brokers answered policy-wording questions by searching a shared drive of 30,000 PDF wordings, endorsements and carrier bulletins, or by asking whichever senior colleague happened to be free. New hires took months to become confident in that material. The compliance team, meanwhile, had no way to know which document version a broker had relied on when advising a client.
The fastest route to a wording answer was asking whichever senior broker happened to be available. That put a hard ceiling on how many questions could be answered in a day, concentrated knowledge in a few people, and made the quality of an answer depend on who picked up.
Thirty thousand PDF wordings, endorsements and bulletins sat on a shared drive. Searching it could not reliably tell a superseded wording from the edition in force, so a broker either already knew where to look or asked someone who did.
When compliance reviewed a file, there was no way to establish which version of a wording the broker had actually read. The reasoning lived in an email thread or in somebody's memory, which is a difficult position for a regulated firm to defend.
New brokers took months to become confident in the material, and much of that time was spent learning where documents lived and which colleague to ask. Onboarding cost senior time twice: once in teaching, and again in answering the questions teaching did not cover.
What we built
We built an internal assistant on a large language model that answers only from the brokerage's own document library and shows the exact passage and version it drew from. The retrieval layer indexes wordings, endorsements and bulletins as they land, keeps carrier-specific permissions intact, and declines to answer when it cannot find support in the source material. It runs inside the brokerage's own cloud tenancy, so no client or carrier data leaves their environment, and every question, answer and citation is written to an audit log.
New wordings, endorsements and bulletins are picked up as they land, split by clause rather than by page, and indexed with the carrier and the document version so retrieval can tell one edition from another.
The assistant retrieves candidate passages first and generates only from what it found. Each answer shows the exact passage it drew from, so a broker verifies the wording itself rather than trusting a summary of it.
When retrieval returns nothing that supports an answer, the assistant declines rather than filling the gap. That behaviour was specified up front, tested against questions the library genuinely cannot answer, and monitored after launch.
Compliance sees every question asked, the answer given, the passage cited and the document version it came from. Reviews now start from a record of what was read instead of from reconstruction.
Process
Question inventory before any model work
We collected the wording questions brokers actually ask, including ones that should not have a confident answer, and used them as the set the assistant would be measured against.
Retrieval built and measured on its own
Retrieval quality was proven before generation was added. If the right clause is not in the retrieved set, no amount of prompting rescues the answer, so that is what we measured first.
Deployed inside the client's tenancy
The system was built to run in the brokerage's own cloud environment, with private networking and the carrier-specific permissions carried through from the document library so nothing leaves the firm's boundary.
Weekly review with senior brokers
Senior brokers graded answers each week, marking not only correctness but whether the citation supported the claim and whether a refusal had been the right call.
Launch with refusal and citation monitoring
The assistant went live with usage, citation and refusal rates monitored, so thresholds were tuned against the questions brokers were genuinely asking rather than against test traffic alone.
Stack
Retrieval and indexing
Models and generation
Application
Security and governance
Delivery and operations
78%
Of wording questions resolved without escalating to a senior broker
11 min → 40 s
Median time to a cited answer
100%
Of answers traceable to a document version
Outcomes
78% of wording questions are resolved without escalating to a senior broker. The people who were the firm's search engine are back on client work, and the questions that do reach them are the ones that genuinely need judgement.
Median time to a cited answer moved from 11 min to 40 s. A broker on a call can check a clause while the client is still on the line, and check the wording itself rather than a paraphrase of it.
100% of answers are traceable to a document version. Every question, answer and citation is logged, so a file review starts from a record of what was read rather than from asking people to remember.
Newer staff can look something up without deciding whether the question is worth interrupting a colleague for, which removes the social cost of asking and shortens the stretch where confidence lags competence.
In their words
We were nervous about an AI giving confident wrong answers. What we got is a tool that says it cannot find something when it should, and shows its work when it can.
Chief Compliance Officer, national insurance brokerage
Insurance & financial services
FAQ
It generates only from passages retrieved out of the brokerage's own library, and it declines when nothing in the source material supports an answer. Refusal was a specified behaviour, not an afterthought: we tested it against questions the library genuinely cannot answer, reviewed the results with senior brokers each week, then monitored refusal rates after launch.
No. The assistant runs inside the brokerage's own cloud tenancy, with private networking and carrier-specific permissions carried through from the document library. Documents, questions and answers stay in the client's environment, and the audit log lives there too.
Documents are indexed with the carrier and the document version, and split by clause rather than by page, so retrieval can tell an edition in force from one that has been replaced. Every answer names the version it drew from, which is what makes the citation useful in a review.
The ingestion pipeline picks up new wordings, endorsements and bulletins as they land and indexes them with their carrier and version, so the assistant answers from current material without anyone running a manual import. Permissions on the source document are preserved.
Fourteen weeks, covering the ingestion pipeline, the broker-facing web app and the compliance review screens, with a working demo every week. Request a quote and we will scope a comparable build for your own document library.
Let’s connect
Tell us what is not working. We’ll reply within one business day with how we would approach it and a tailored quote.