An SLO-aware retrieval router for web and product search.
One thesis: spend the scarce budget where it buys the most relevance. Every query should not pay for the same expensive retrieval pipeline. A controller picks the cheapest route that still answers it.
This runs the actual trained model in your browser. The same eight-feature extraction and linear decision the Go service uses in production, loaded from the model the Python pipeline trained on 1,039 real ESCI queries. Nothing is sent anywhere.
loading model...
Measured on 1,500 real Amazon ESCI test queries (~38 graded candidates each). Running the expensive cross-encoder on every query buys almost nothing over BM25 and costs 17x the latency. An oracle that routes per query beats it on both axes.
| policy | nDCG@10 | rr_exact | latency (ms) | utility |
|---|---|---|---|---|
| bm25_only | 0.7465 | 0.7193 | 2.00 | 0.7385 |
| always_rerank | 0.7510 | 0.7361 | 35.00 | 0.6110 |
| rule_router | 0.7479 | 0.7280 | 18.93 | 0.6722 |
| oracle | 0.7769 | 0.7783 | 4.26 | 0.7599 |
Learned router vs always-rerank on held-out test, paired bootstrap with the query as the unit: utility +0.133, 95% CI [+0.127, +0.139], win/tie/loss 449/0/12, significant. Latencies are modeled until measured under load; nDCG is real on the ESCI judgments.
query -> features -> route controller -> { lexical(BM25) | dense(ANN) | hybrid(RRF) | rerank(cross-encoder) } -> ranking
|
utility = nDCG - latency_penalty - violation_penalty - timeout_penalty
Python owns research and trains the router. The Go service loads the exact frozen model and applies the same decision, so the deployed router equals the evaluated one. OpenSearch holds the BM25 and HNSW indexes; Redis caches; Prometheus and a k3s autoscaler run the serving path. The web track reuses the same controller with a crawl budget instead of a latency budget.