Overview
| Client | National dealer-network distributor — 140 dealers | 6 support staff |
|---|---|
| Industry / Technology | Distribution / Assistant |
| Period | Mar-May 2026 (9 weeks) |
| Scope | Retrieval pipeline | admin console | messenger integration |
| Stack | FastAPI, pgvector, BGE-m3, cross-encoder reranker, Langfuse |
| Disclosure grade | L2 Anonymised |
What was the problem?
There were 320 documents covering product specifications, returns policy and promotion terms, with several versions in circulation. Support staff spent about half of each enquiry simply establishing which document was current.
Why this approach?
We prioritised suppressing wrong answers over raising the answer rate. The similarity threshold was set high enough to leave a deliberate refusal rate, and a refusal hands the dealer to a person. In distribution support, one wrong answer costs more than ten unanswered ones.
What we built
- 01Reconciled 320 documents by version and attached validity metadata
- 02Hybrid retrieval (vector + BM25) to keep part-number lookups accurate
- 03Tuned against a 100-question golden set in repeated QA rounds
- 04Embedded the widget in the dealer messenger channel
Before to After (measured in-house)
First-line CS per month
1,240
480
Average response time
18 min
9 sec
Golden-set accuracy
-
92%
Wrong answers reported per month
-
2
Figures were measured on the same basis before the engagement and after go-live, and the calculation records are retained.
“Building it to say "I do not know" is what earned the trust. Dealers now reach for the assistant first.”
Support team lead | quoted with consent