# Is Lilavati SOTA?

Pending benchmarks. Announced Oct 16, 2026; no independent results yet.

As of Nov 3, 2026, Lilavati is pending benchmarks on isitsota: announced Oct 16, 2026, with no independent results on the boards we track yet.

Udgam Labs' 95M-parameter English model that solves grade-school maths word problems by writing a short Python program, which is run in a sandbox to give the answer. Trained from scratch and released under Apache-2.0 with Lilavati-Checker, a 95M companion model that rates programs so many tries can be combined.

Canonical: https://isitsota.com/m/lilavati
Last updated: 2026-10-24
Tracking since: 2026-10-24

## Announced facts

| Fact | Value |
| --- | --- |
| Lab | Udgam Labs |
| Announced | Oct 16, 2026 |
| API | Not yet |
| Open weights | Published |
| Parameters | 95M (94.7M; 12 layers, width 640) |
| Input → output | English word problem → a Python function whose return value is the answer |
| Training | Trained from scratch on public synthetic maths datasets with Python solutions (TinyGSM and a TinyGSM-style set), general English text, the GSM8K training split and LLM-generated problems kept only if their solutions executed |
| Context | 4,096 tokens |
| Companion model | Lilavati-Checker (95M, same architecture and tokenizer) estimates how likely each program is right; the lilavati package runs 8 tries by default with a checker-weighted vote |
| Release | udgamlabs/Lilavati and udgamlabs/Lilavati-Checker on Hugging Face; code, sandbox and evaluation on GitHub (udgamlabs/lilavati) |
| License | Apache-2.0 (weights and code) |

Tags marked unsure are not stated on the lab's own pages.

## What would let us call it

Verdicts use only independent boards we already track. Two consecutive snapshots are required before any crown.

- [x] Publicly announced with a source
- [x] Usable by the public (API or weights)
- [ ] Listed on an independent board we track
- [ ] Enough results for a rank in a category
- [ ] Holds across two snapshots (crowns only)

## What the lab claims

Self-reported. Shown for context; not used in any verdict.

- Lab claim: GSM8K test 54.7% with one greedy try (program executed); 72.3% with the checker-weighted vote over 48 tries.
- Lab claim: SVAMP 69.9%, ASDiv (numeric) 74.0%, GSM-Hard 43.7%, GSM-Plus testmini 39.2% with one greedy try.
- Lab claim: about 2.7× the GSM8K score of the strongest other open model up to 150M parameters that Udgam tested; Udgam notes TinyGSM's unreleased 125M models reported higher numbers under their own protocol.

## Sources

- LAB: [Lilavati, a 95M model that solves maths word problems by writing Python (Udgam Labs blog)](https://udgamlabs.com/blog/lilavati) (2026-10-16)
- LAB: [Lilavati model page (Udgam Labs)](https://udgamlabs.com/models/lilavati) (2026-10-16)
- REPO: [udgamlabs/Lilavati model card (Hugging Face)](https://huggingface.co/udgamlabs/Lilavati)
- REPO: [udgamlabs/Lilavati-Checker model card (Hugging Face)](https://huggingface.co/udgamlabs/Lilavati-Checker)
- REPO: [udgamlabs/lilavati (GitHub)](https://github.com/udgamlabs/lilavati)

Tracking continues until an independent board lists it.

## Related

- Lab: [Udgam Labs](https://isitsota.com/labs/udgam-labs.md)
- All pending releases: https://isitsota.com/releases.md
- JSON: https://api.isitsota.com/v1/models/lilavati