Is Lilavati state of the art?Pending benchmarks.Announced Oct 16, 2026; no independent results yet.

The details so far

Lab

Udgam Labs

Announced

Oct 16, 2026

API

Not yet

Open weights

Published

Parameters

95M (94.7M; 12 layers, width 640)

Input → output

English word problem → a Python function whose return value is the answer

Training

Trained from scratch on public synthetic maths datasets with Python solutions (TinyGSM and a TinyGSM-style set), general English text, the GSM8K training split and LLM-generated problems kept only if their solutions executed

Context

4,096 tokens

Companion model

Lilavati-Checker (95M, same architecture and tokenizer) estimates how likely each program is right; the lilavati package runs 8 tries by default with a checker-weighted vote

Release

udgamlabs/Lilavati and udgamlabs/Lilavati-Checker on Hugging Face; code, sandbox and evaluation on GitHub (udgamlabs/lilavati)

License

Apache-2.0 (weights and code)

Tags marked unsure are not stated on the lab's own pages.

What would let us call it

Verdicts use only independent boards we already track. Two consecutive snapshots are required before any crown.

• Publicly announced with a source
• Usable by the public (API or weights)
• Listed on an independent board we track
• Enough results for a rank in a category
• Holds across two snapshots (crowns only)

What the lab claims

Self-reported. Shown for context; not used in any verdict.

Lab claim: GSM8K test 54.7% with one greedy try (program executed); 72.3% with the checker-weighted vote over 48 tries.

Lab claim: SVAMP 69.9%, ASDiv (numeric) 74.0%, GSM-Hard 43.7%, GSM-Plus testmini 39.2% with one greedy try.

Lab claim: about 2.7× the GSM8K score of the strongest other open model up to 150M parameters that Udgam tested; Udgam notes TinyGSM's unreleased 125M models reported higher numbers under their own protocol.

Sources