'/%3e%3ccircle%20cx='98'%20cy='41'%20r='9'%20fill='%2322D3EE'/%3e%3c/svg%3e){kind=small}
'/%3e%3cellipse%20cx='520'%20cy='260'%20rx='360'%20ry='132'%20stroke='%238B93B0'%20stroke-opacity='0.18'%20stroke-width='1'%20transform='rotate(-16%20520%20260)'/%3e%3ccircle%20cx='520'%20cy='260'%20r='46'%20fill='%23111A38'%20stroke='%2322D3EE'%20stroke-opacity='0.35'/%3e%3ccircle%20cx='920'%20cy='80'%20r='6'%20fill='%2322D3EE'/%3e%3c/svg%3e)
A support assistant that knows when to hand over
Challenge
A support team was buried in repetitive questions answerable straight from the help center — but a naive bot would answer the harder, unanswerable ones just as confidently, which is exactly how automated support loses customers' trust.
Approach
We built retrieval-augmented answering over the help-center content with citations back to the source passages, then put a confidence gate in front of every reply: when retrieval score or answer groundedness falls below threshold, the question routes straight to a human with the full conversation and retrieved context attached. An evaluation set of real questions guards against regressions each time the content or model changes.
Architecture
Question → retrieval over help center → groundedness check → cited answer OR confidence-gated handover to an agent
Python · LangGraph · vector search · FastAPI — groundedness evaluation, confidence-gated escalation
Results
- Around 45% of tickets resolved without a human, the rest escalated with context*
- Every answer cites the source passages it came from
- Escalations reach agents with the conversation and retrieved context attached
* Illustrative figure from a representative engagement, shown to convey typical scope. Replaced with client-verified numbers before publication.