UBCIInternship project2026
Banking document search
An AI assistant that finds the right banking documents and answers with sources you can check.
The problem
A fluent answer is not useful when the supporting document is wrong. In a banking setting, retrieval quality, traceability, and latency are product requirements—not offline model trivia.
What I built
The final workflow joined document validation, hybrid retrieval, reranking, generation, and source inspection in one product path.
The result
Teacher reranking raised BCT Hit@1 by 12.6 percentage points.
In context
Hit@1 measures retrieval relevance, not complete answer correctness or user comprehension.
Explore the technical details
Question & ownership
Can a banking assistant retrieve the right source first without making every CPU request depend on a slow, heavyweight reranker?
I owned the retrieval architecture, benchmark design, teacher-to-student reranking experiments, API workflow, and the source-grounded user experience.
Data & validation
The corpus combined heterogeneous banking material, public regulatory publications, and permission-gated internal documents. Evaluation used labeled query sets plus held-out and robustness variants.
- Normalize heterogeneous documents into reviewable retrieval units.
- Keep internal material permission-gated throughout ingestion and publication.
- Separate benchmark queries from held-out and paraphrased robustness checks.
- Preserve source identity through retrieval so every answer can remain traceable.
Key decisions
Hybrid retrieval over a single search signal
Lexical search protects exact banking terms, references, and codes; dense retrieval handles semantic paraphrases. Either signal alone left predictable blind spots.
Dense-only and lexical-only retrieval were retained as comparison baselines.
Rank fusion before learned reranking
RRF combines rankings without pretending that lexical and dense similarity scores share the same scale.
Raw-score blending would require fragile score normalization across retrievers.
Distill quality instead of serving the teacher
The teacher established the quality ceiling; the student retained nearly all of the gain at a CPU latency that fits an interactive product.
Serving the teacher produced stronger cost and latency pressure for every query.
Expose sources in the interface
Users need to inspect why an answer was returned, especially when the material informs operational or compliance work.
A citation-free chat response would make retrieval errors harder to detect and correct.
Evaluation
The benchmark centered first-result relevance because the top retrieved document has disproportionate influence on both the answer and user trust.
- Teacher reranking raised BCT Hit@1 by 12.6 percentage points.
- The student retained 98.6% of the teacher's Hit@1 result.
- The optimized CPU path reduced latency by approximately 18×.
Limitations & next steps
- Hit@1 measures retrieval relevance, not complete answer correctness or user comprehension.
- The private corpus prevents a public live demo and limits reproducibility outside the authorized environment.
- Document and policy changes can shift the benchmark, so evaluation must be rerun as the corpus evolves.
- Add explicit answer-faithfulness and citation-correctness evaluation.
- Use reviewed user feedback to expand the hard-query set without leaking private content.
- Track retrieval drift as new regulatory publications enter the index.
Sanitized case study