From 35b3e455cf57ac1e2caecb7f44bb870e22f00b9e Mon Sep 17 00:00:00 2001 From: Kshitij <160704796+kshitij-ka@users.noreply.github.com> Date: Mon, 4 May 2026 15:46:42 +0530 Subject: [PATCH] docs: update README. --- README.md | 37 +++++++++++++++++++------------------ 1 file changed, 19 insertions(+), 18 deletions(-) diff --git a/README.md b/README.md index e0407b8..a71d107 100644 --- a/README.md +++ b/README.md @@ -13,10 +13,10 @@ | Metric | Target | **Our Score** | |---|---|---| | Hit Rate @3 | > 80% | **100%** (10/10) | -| MRR @5 | > 0.7 | **1.000** | -| Avg Latency | < 5 s | **~19 ms** | +| MRR @5 | > 0.7 | **0.950** | +| Avg Latency | < 5 s | **~18 ms** | -All 10 public queries returned the expected standard at rank 1. Average query latency is 19 ms after the index warms up — 250× faster than the 5 s target. +All 10 public queries returned the expected standard in the top 2 results. Average query latency is 19 ms after the index warms up — 250× faster than the 5 s target. --- @@ -28,7 +28,7 @@ Indian Micro and Small Enterprises (MSEs) spend weeks manually searching BIS SP- 2. **Get ranked BIS standards** with matched sections and relevance scores in milliseconds 3. **Read AI explanations** of why each standard applies, generated by Groq LLM -The system covers all **573 unique standards** across **25 building material categories** from BIS SP-21 (Summaries of Indian Standards for Building Materials). +The system covers all **586 unique standards** across **25 building material categories** from BIS SP-21 (Summaries of Indian Standards for Building Materials). --- @@ -39,8 +39,8 @@ The system covers all **573 unique standards** across **25 building material cat ``` data/raw/dataset.pdf (BIS SP-21, 929 pages) → src/parse_bis_pdf.py - → data/processed/standards.json 573 structured records [committed] - → data/processed/standards_chunks.json 1,236 RAG-ready chunks [committed] + → data/processed/standards.json 586 structured records [committed] + → data/processed/standards_chunks.json 1,269 RAG-ready chunks [committed] → inference.py --build → data/processed/embeddings.npy dense vectors [gitignored — rebuild locally] → data/processed/faiss.index FAISS index [gitignored — rebuild locally] @@ -70,7 +70,7 @@ Browser / API Client - Pass 4: truncates next-standard content bleed at a second `1. Scope` marker - Each standard is further split by section with **50-word overlap** to prevent context loss at boundaries - Weak chunks (<30 words) are merged with their neighbour -- Result: 1,236 chunks from 573 standards (avg 2.2 chunks/standard) +- Result: 1,269 chunks from 586 standards (avg 2.2 chunks/standard) **Hybrid Retrieval** (`inference.py`): - **Dense**: FAISS `IndexFlatIP` with `all-MiniLM-L6-v2` embeddings (384-dim cosine similarity) @@ -83,6 +83,7 @@ Browser / API Client - +0.25 if ≥60% of significant title words appear in the query (strong title match) - +0.20 if an exact IS ID from the query matches this standard - +0.35 / -0.40 grade discriminator: boosts/penalises OPC-grade standards (33/43/53) when query names a specific grade +- +0.30 / -0.20 part-number discriminator: boosts matching Part N and penalises non-matching parts when query explicitly names a part number (handles "Part – 1", "PART2" etc.) - -0.15 penalty for very short chunks (<40 body words) **Post-grouping Part disambiguation**: when multiple parts of the same IS base number survive into the candidate set with identical titles, IDF-weighted discriminating keyword scores break the tie — rarer corpus terms (e.g. "lightweight") carry proportionally more weight. @@ -95,7 +96,7 @@ Browser / API Client |---|---| | Persistent Python daemon | FAISS index load takes ~18 s cold. Spawn once at boot, queue all requests through a single process — zero cold start per query. | | `inference.py` never modified | Bridge pattern: `bridge/retrieve.py` imports `inference.py` as a module. Judges run `inference.py` directly; the web server uses the bridge. Both paths are identical. | -| In-memory data | 573 standards + 1,236 chunks fit comfortably in RAM. No database dependency, no I/O per request. | +| In-memory data | 586 standards + 1,269 chunks fit comfortably in RAM. No database dependency, no I/O per request. | | LLM fallbacks everywhere | Every Groq call is wrapped with a timeout (8 s) and a safe default return. `Promise.allSettled` for parallel calls. Server starts and retrieval works without a `GROQ_API_KEY`. | | Weighted BM25 document | Repeating title tokens ×4 makes exact IS-standard name queries dominant over body-text noise — critical for the BIS domain where standard names are precise. | @@ -105,14 +106,13 @@ Browser / API Client ``` SpecForge/ -├── inference.py # Entry point for judges — do not modify +├── inference.py # Entry point for judges ├── requirements.txt # All Python dependencies -├── scripts/ -│ └── eval_script.py # Provided evaluation script (Hit@3, MRR@5, latency) +├── eval_script.py # Provided evaluation script (Hit@3, MRR@5, latency) ├── data/ │ └── processed/ -│ ├── standards.json # 573 parsed standards (committed) -│ ├── standards_chunks.json # 1,236 RAG chunks (committed) +│ ├── standards.json # 586 parsed standards (committed) +│ ├── standards_chunks.json # 1,269 RAG chunks (committed) │ ├── public_test_set.json # 10 public evaluation queries │ └── retrieval_results.json # Our results on public test set ├── src/ @@ -211,7 +211,7 @@ source .venv/bin/activate python inference.py --build ``` -Encodes 1,236 chunks, writes `embeddings.npy` + `faiss.index` to `data/processed/`. Takes **~2 min on CPU**. Subsequent starts load from cache — no rebuild needed unless chunks change. +Encodes 1,269 chunks, writes `embeddings.npy` + `faiss.index` to `data/processed/`. Takes **~2 min on CPU**. Subsequent starts load from cache — no rebuild needed unless chunks change. ### Step 4 — Node.js dependencies @@ -341,7 +341,7 @@ python inference.py \ --output data/processed/retrieval_results.json # Step 2: score -python scripts/eval_script.py \ +python eval_script.py \ --results data/processed/retrieval_results.json ``` @@ -350,8 +350,8 @@ Targets and our results on the public set: | Metric | Formula | Target | Achieved | |---|---|---|---| | Hit Rate @3 | correct queries where expected std in top-3 / total | > 80% | **100%** | -| MRR @5 | Σ(1/rank_i) / N | > 0.7 | **1.000** | -| Avg Latency | total_time / num_queries | < 5 s | **~0.019 s** | +| MRR @5 | Σ(1/rank_i) / N | > 0.7 | **0.950** | +| Avg Latency | total_time / num_queries | < 5 s | **~0.018 s** | --- @@ -422,7 +422,7 @@ All 25 material categories sorted alphabetically. ### `GET /api/stats` ```json -{ "standards": 573, "chunks": 1236, "categories": 25 } +{ "standards": 586, "chunks": 1269, "categories": 25 } ``` --- @@ -433,6 +433,7 @@ All 25 material categories sorted alphabetically. |---|---| | **Hybrid RAG retrieval** | FAISS (dense, 60%) + BM25 (sparse, 40%) fused and re-ranked | | **Re-ranking** | Keyword overlap, title match, exact IS-ID match, short-chunk penalty | +| **Part-number disambiguation** | Explicit Part N in query boosts matching part ±0.30, penalises siblings ±0.20; handles em-dash/PART2 variants | | **AI explanations** | Groq `llama-3.1-8b-instant` — parallel, fallback-safe | | **Query rewriting** | LLM expands natural language to IS-standard vocabulary (optional) | | **Chunk-grounded QA** | Question answered from the most relevant chunk of a specific standard |