Overview
Compliance teams answer legal questions every day, and "probably right" is not good enough. A generic chatbot invents section numbers, mixes old and new versions of a rule, ignores who may see which document, and leaves no audit trail. The design rule here is simple: no verifiable evidence, no answer. It covers the PDPA, the Anti-Money Laundering Act, the CDD ministerial regulation, and Bank of Thailand and SEC notifications, alongside internal policies.
What I built
- Verified citations: the model must return a verbatim quote for every citation, and code checks that the quote exists in the cited passage. If none verify, the user sees the source text instead of the answer.
- Abstention: if the best reranker score falls below a calibrated threshold, the language model is never called.
- Hybrid retrieval: BGE-M3 dense vectors, Thai-segmented BM25 and explicit section lookup, fused and re-ordered by a cross-encoder, with a glossary that maps the words staff use to the words in the law.
- Thai legal documents: repairs broken Thai text from government PDFs, parses the มาตรา/ข้อ structure and amendment footnotes, and falls back to OCR for scanned pages.
- Time-aware answers: each answer follows the version in force on a chosen date, while older versions stay searchable.
- Access control, twice: clearance filters inside the retrieval SQL and PostgreSQL row-level security under a read-only role. A test deletes the SQL filter and confirms nothing leaks.
- Privacy and audit: personal data is masked before it reaches the model (Thai ID checksum, card numbers, phones, names), prompt injection is screened, and every exchange goes into a hash-chained audit log that is re-verified hourly.
- Policy gap analysis: checks each clause of an internal policy against the law and produces a reviewable worklist with verified quotes.
Results
218 evaluation questions in three sets, scored with deterministic checks rather than an LLM judge:
- Baseline (fixed-size chunks + vector search): Hit@6 of 51.6–57.5%. Splitting documents by legal unit (มาตรา/ข้อ) alone lifted it to 90%.
- Held-out set of 21 questions, first blind run: Hit@6 94.1% and correct behaviour 85.7% — it exposed three real defects, which were then fixed.
- Extended set of 146 questions, first blind run: Hit@6 92.2% and correct behaviour 94.5%.
- Confidential-document leakage was 0 in every set, cited quotes were found verbatim in the source 95.9–98% of the time, and end-to-end answers took 3.5 s at p50 and 8.8 s at p95.
What these numbers do not prove
All test questions were written by engineers and none is confirmed by a compliance expert yet; the internal policies are synthetic samples; and the system has not been used by real users or penetration-tested. The repository says so openly and ships the tooling to close that gap.
Tools
Python and FastAPI, PostgreSQL with pgvector, BGE-M3 with a cross-encoder reranker, the Gemini API, Next.js with shadcn/ui, Docker with a production compose file, GitHub Actions running 550 tests on every push, and Prometheus with 12 alert rules about answer trustworthiness.