Is WorldSonus state of the art?Pending benchmarks.Announced Oct 6, 2026; no independent results yet.
Noiz AI and HKUST's streaming video-to-audio model for world models and video: it generates 48 kHz stereo audio in 100 ms chunks from past and current frames, with optional text instructions that can change mid-stream. Open weights for non-commercial use (CC BY-NC 4.0).
The details so far
- Lab
- Noiz AI
- Announced
- Oct 6, 2026
- API
- Not yet · None announced
- Open weights
- Published · Hugging Face FF2416/WorldSonus (linked from NoizAI's GitHub README), not gated; inference code at NoizAI/WorldSonus on GitHub
- Type
- Causal autoregressive diffusion (rectified-flow) video-to-audio model; each chunk sees only past and current frames
- Output
- Stereo 48 kHz audio in 100 ms chunks, as a WAV file or streamed raw PCM
- Control
- Optional text prompt; prompts can change at chunk boundaries during a stream
- Components
- Frozen DINOv3 visual encoder, T5Gemma 2 text encoder and a causal stereo audio VAE; DINOv3 and T5Gemma 2 keep their upstream licenses
- Training data
- 1,465 hours of audio: 999 hours of paired stereo video-audio and 466 hours of audio only
- License
- CC BY-NC 4.0 (non-commercial use with attribution)
- Paper
- WorldSonus: Bringing Sound to Worlds (arXiv 2610.08760), Noiz AI with HKUST, MetaX and Shanghai Jiao Tong University
- Requirements
- Python 3.10+, a CUDA GPU, FFmpeg and ffprobe
- Parameters
- Not stated
Tags marked unsure are not stated on the lab's own pages.
What would let us call it
Verdicts use only independent boards we already track. Two consecutive snapshots are required before any crown.
- Publicly announced with a source
- Usable by the public (API or weights)
- Listed on an independent board we track
- Enough results for a rank in a category
- Holds across two snapshots (crowns only)
What the lab claims
Self-reported. Shown for context; not used in any verdict.
Lab claim: 41.2 ms of compute per 100 ms audio chunk (real-time factor 0.41) on one NVIDIA H100.
Lab claim: VGGish FAD of 1.73 on 5-second stereo VGGSound clips and 2.03 on 30-second interactive videos, matching or exceeding offline bidirectional video-to-audio baselines on acoustic quality and stereo balance.
Sources
PAPERWorldSonus: Bringing Sound to Worlds (arXiv)arxiv.org/abs/2610.08760 · Oct 6, 2026LABWorldSonus project pagenoizai.github.io/WorldSonus/REPONoizAI/WorldSonus on GitHubgithub.com/NoizAI/WorldSonusREPOFF2416/WorldSonus on Hugging Facehuggingface.co/FF2416/WorldSonusTracking continues until an independent board lists it.