Key takeaways
- Canada’s Directive on Automated Decision-Making already requires pre-launch bias testing (6.3.1) and scheduled outcome monitoring (6.3.2). OSFI Guideline E-23 extends the same discipline to AI models at banks and insurers, effective May 1, 2027.
- A weekend-sized RAG evaluation on Canadian regulatory text shows what monitoring catches: a chunking and top-k change lifted faithfulness from 0.95 to 1.00 and context precision from 0.71 to 0.86 on 12 questions.
- Three open-source frameworks cover the job: Ragas for metric depth, DeepEval for test-driven CI, TruLens for production tracing.
- Action: add a 12-question golden set and three Ragas metrics to your next RAG change; re-run before every deploy.
The regulatory backdrop
Two Canadian instruments now share one expectation: test an AI system before launch and keep testing it after.
The Treasury Board Directive on Automated Decision-Making has required this for years. Section 6.3.1 says departments must test the data their systems use for unintended biases before going into production. Section 6.3.2 says they must monitor outcomes on a scheduled basis to catch unintentional results and verify compliance. With section 6.2.3’s right to a meaningful explanation of each decision, the picture is clear: an untested generative system is a compliance risk as much as a quality one.
OSFI is bringing the same logic to the financial sector. Its updated Guideline E-23 on model risk management takes effect May 1, 2027 and covers AI and machine learning models at all federally regulated financial institutions. It expects documented monitoring standards (evaluation criteria, monitoring measures, escalation procedures) plus independent review of whether models work as intended and fit their purpose. A RAG assistant answering staff or customer policy questions is a model, and E-23’s lifecycle discipline applies to it.
Neither instrument names a tool. That part is engineering judgment, so I built a small test to find out.
The mini-benchmark
I assembled 14 passages of public regulatory text: the TBS Directive on Automated Decision-Making and a professional summary of OSFI Guideline E-23’s expectations. I wrote 12 test questions with hand-written reference answers, each tied to the passage that answers it, from “When did the Directive take effect?” to “What lifecycle stages must FRFIs govern under E-23?”
The pipeline is deliberately ordinary: TF-IDF retrieval over character chunks, with gemini-3.5-flash-lite answering in 2-3 sentences strictly from the retrieved passages. I ran two variants on the same questions:
- Baseline: 400-character chunks, no overlap, top-3 retrieval.
- Improved: 900-character chunks with 150-character overlap, top-6 retrieval.
Ragas 0.4.3 scored three metrics: faithfulness (are the answer’s claims supported by the retrieved context?), context precision (are the retrieved passages relevant?), and answer relevancy (does the answer address the question?). gemini-3.5-flash-lite judged, with structured-output parsing.
| Metric | Baseline (400-char chunks, top-3) | Improved (900-char chunks + overlap, top-6) |
|---|---|---|
| Faithfulness | 0.95 | 1.00 |
| Context precision | 0.71 | 0.86 |
| Answer relevancy | 0.29 | 0.28 |
Means over 12 questions; faithfulness and context precision judged by gemini-3.5-flash-lite, relevancy by local TF-IDF.
The pattern is the useful part. The improved variant lifted faithfulness from 0.95 to 1.00 and context precision from 0.71 to 0.86: larger overlapping chunks kept each passage self-contained, so claims stayed grounded and the top-6 set held fewer near-misses than the baseline’s top-3. Answer relevancy was flat (0.29 vs 0.28), which is itself a finding: with local TF-IDF embeddings, that metric rewards word overlap over meaning, so paraphrase swings it. Two metrics say the change helped, one says it is noise, and the operator decides what to ship. That logged, repeated judgment call is what the Directive’s scheduled monitoring duty (6.3.2) looks like in practice: one good run proves nothing; the trend does. The habit is what matters: a golden set, three metrics, a before/after comparison on every change.
Ragas vs DeepEval vs TruLens
| Ragas | DeepEval | TruLens | |
|---|---|---|---|
| Core idea | Reference-free and reference-based LLM-judge metrics for RAG | Pytest-style testing framework for LLM apps | Observability and tracing for LLM apps in production |
| Strength | Deep metric set (faithfulness, context precision/recall, answer relevancy) with LangChain/LlamaIndex integrations | Test cases, synthetic data generation, and CI-friendly pass/fail gates | Feedback functions (groundedness, relevance) computed over live traces, with dashboards |
| Best fit | Diagnosing why a RAG pipeline’s answers drift | Regression-testing prompts, models, and retrieval on every change | Continuous monitoring of a deployed assistant |
Pragmatically: DeepEval-style gates in CI, TruLens-style feedback in production, Ragas metrics underneath. Pre-launch testing and scheduled monitoring map to CI gates and production traces; all three are open source, and the real cost is judge-LLM tokens.
The actionable takeaway
You do not need a research budget to evaluate like a regulator expects. This week: write 12 questions your RAG system must answer, with one-line reference answers. Score three Ragas metrics on a baseline, change one thing (chunking or top-k), and re-run. If the numbers move the way mine did, you have the seed of the monitoring program the Directive and E-23 describe, and a defensible answer when someone asks how you know the system still works.
Method note: 12 questions, 14 public-text passages, TF-IDF retrieval, gemini-3.5-flash-lite throughout, local TF-IDF embeddings for relevancy. Per-question scores ship with the data file. Small sample: treat deltas as directional.
The corpus, the benchmark script, and the per-question scores are public: movahedi-ca/rag-eval-demo.