Benchmark · 20 October 2026
NyayaVeda AI Legal Accuracy Benchmark
600 questions on Indian law, 100 in each of 6 areas, asked on 20 October 2026 of the production NyayaVeda AI pipeline (release r91). 488 were answered, 111 declined because the sources did not support an answer, and 1 timed out. Every figure below is a share, k of n, with its 95% interval, and the questions, answers and grades behind it are kept for independent verification.
87.6%
Citations grounded in the sources given
1,964 of 2,242
98.0%
Hindi and Hinglish questions
98 of 100
86.0%
IPC/CrPC to BNS/BNSS drafts
86 of 100
97.0%
Fabricated citations rejected
97 of 100
Results, 20 October 2026
| Area | Result | k of n | 95% interval |
|---|---|---|---|
| Citations grounded in the sources givenOf the citations an answer makes (precedents and provisions), the share found in the sources it was written from. (higher is better) | 87.6% | 1,964 of 2,242 | 86.0% to 89.2% |
| IPC/CrPC to BNS/BNSS transitionDrafts for an offence dated around 1 July 2024 that charge under the code the offence date requires and register under the code in force at filing. (higher is better) | 86.0% | 86 of 100 | 77.9% to 91.5% |
| Fabricated citations and sections rejectedQuestions naming a citation or section that cannot exist, which the answer rejects rather than treating as law. (higher is better) | 97.0% | 97 of 100 | 91.5% to 99.0% |
| Hindi and Hinglish questionsAnswers that identify the provision a Hindi or Hinglish question names. (higher is better) | 98.0% | 98 of 100 | 93.0% to 99.5% |
| Personal data repeated in the answerAnswers that repeat an injected (synthetic) Aadhaar, PAN, phone number or victim's name. (lower is better) | 0.0% | 0 of 95 | 0.0% to 3.9% |
| Overruled law relied on as good lawAnswers that rely on a judgment a larger bench displaced, as if it were current law. (lower is better) | 12.0% | 9 of 75 | 6.4% to 21.3% |
| Overruled and per incuriam authority recognisedAnswers that state a displaced authority to be displaced and do not rely on it. (higher is better) | 52.0% | 52 of 100 | 42.3% to 61.5% |
| Split benches: which opinion bindsAnswers whose verdict on which opinion of a split Supreme Court bench binds under Article 141 is right. (higher is better) | 47.0% | 47 of 100 | 37.5% to 56.7% |
A declined answer is the system stating that its sources do not support an answer, which it does rather than guess; in the accuracy areas a decline counts as a miss. Time per answer: median 32 s, 95th percentile 50.4 s.
Compared with 1 October 2026
The 1 October run asked 1,959 questions from the same frozen plan; the 600 questions of 20 October 2026 are its first 100 of each area.
| Area | 1 Oct 2026 (1,959 questions) | 20 October 2026 (600 questions) |
|---|---|---|
| Citations grounded in the sources given | 76.0% (5,880 of 7,732) | 87.6% (1,964 of 2,242) |
| IPC/CrPC to BNS/BNSS transition | 76.2% (234 of 307) | 86.0% (86 of 100) |
| Fabricated citations and sections rejected | 99.5% (379 of 381) | 97.0% (97 of 100) |
| Hindi and Hinglish questions | 95.5% (364 of 381) | 98.0% (98 of 100) |
| Personal data repeated in the answer | 19.3% (70 of 363) | 0.0% (0 of 95) |
| Overruled law relied on as good law | 9.2% (21 of 229) | 12.0% (9 of 75) |
| Overruled and per incuriam authority recognised | 50.3% (150 of 298) | 52.0% (52 of 100) |
| Split benches: which opinion binds | 52.6% (111 of 211) | 47.0% (47 of 100) |
Personal data on 1 October was measured before a masking fix; re-measured on 3 October 2026 after it, 0.3%.
How it was measured
The questions are generated, not hand-written: each combines verified legal facts (a statute section and its replacement, a Supreme Court bench and its split, an overruling chain), every fact carrying the authority it was read from. Section numbers come from the IPC to BNS and CrPC to BNSS concordance, validated against India Code. A fact two credible sources dispute only produces questions that ask for the dispute to be surfaced. The question set was frozen before any answer was seen.
The six areas: drafting across the 1 July 2024 transition; which opinion of a split Supreme Court bench binds under Article 141; recognising overruled and per incuriam authority; rejecting fabricated citations and sections; Hindi and Hinglish questions; and questions carrying synthetic Aadhaar, PAN, phone numbers and victims' names.
An answer that must be read is read by an automated reader that quotes it; code checks that every quote occurs in the answer and names the judgment or section, and applies the question's gold. The readers were validated against hand reading before use.
What this benchmark does not show
- No advocate has reviewed the grades yet. We invite advocates, law faculties and firms to review any of them.
- Citation precision measures grounding in the sources given, not whether an authority exists: a real authority cited from memory counts as a miss.
- With 100 questions an area, a difference between two runs smaller than about 8 to 14 points is not established.
- Retrieval recall against a gold set, the provision in force on a given date, state amendments and writing quality are not measured here.
Questions about the benchmark
How accurate is NyayaVeda AI?
On 600 Indian-law questions asked on 20 October 2026, 87.6% of the citations its answers made (1,964 of 2,242) were found in the sources the answer was written from. It identified the provision in 98.0% of Hindi and Hinglish questions, drafted the correct code for 86.0% of offences dated around the 1 July 2024 transition, rejected 97.0% of fabricated citations and sections, and repeated personal data in 0 of 95 answers. It was weakest on split benches (47.0%) and on recognising overruled authority (52.0%), which are stated here as they were measured.
Does NyayaVeda AI make up case law?
Every citation in an answer is checked against the sources the answer was written from, and an answer marks what it could not verify. 87.6% of citations were found in those sources. In 97 of 100 questions naming a citation or section that cannot exist, it rejected the fabricated authority rather than stating law for it.
Does NyayaVeda AI handle the IPC to BNS and CrPC to BNSS transition?
Yes. Asked to draft for an offence dated around 1 July 2024, its drafts charged under the code the offence date requires and registered under the code in force at filing in 86 of 100 cases (86.0%), against 76.2% on 1 October 2026.
Can NyayaVeda AI answer in Hindi?
Yes. Questions written in Devanagari Hindi and in Roman-script Hinglish were answered with the right provision identified in 98 of 100 cases (98.0%).
How was the benchmark scored?
The questions are generated from verified legal facts, each carrying the authority it was read from, and frozen before any answer is seen. Answers that must be read are read by an automated reader that quotes the answer; code checks every quote occurs in the answer and names the judgment or section, and applies the question's gold. The readers were validated against hand reading before use. No advocate has reviewed the grades yet, and that review is invited.
Can I verify these results?
Yes. Every question, every answer exactly as the user received it, every grade with the quotes it rests on, and every score with its population and interval are kept for verification and shared with researchers, advocates, law faculties and firms on request at support@nyayaveda.co.in. The one-page report is on this page.
Verify it yourself
Every question, every answer as the user received it, every grade with the quotes it rests on, and every score with its population and interval are kept for independent verification, with checksums. Researchers, advocates, law faculties and firms can request access.
Cite as: NyayaVeda AI Legal Accuracy Benchmark, 20 October 2026. Tripathi, Abhay. NyayaVeda AI. https://www.nyayaveda.co.in/benchmark