Skip to content

26 s6-ear: a decision model that listens ​

Code ready and tested end to end on a small sample (3 November 2026). The first s6-ear-nano was trained in November 2026 and failed three of its four gates: it failed its gate; its file is published as a result (owner's decision of 15/11/2026). The result is at the end of this chapter.

Every model so far reads its state as text. s6-ear reads audio: the last 10 seconds of speech, at 16 kHz. It answers typed questions about them with the same engine, the same GGUF format, the same early exits and the same WASM build as the game models. These are the decisions a voice agent needs every 80 ms: has the speaker finished, which word was said, and (designed, never run) whether to keep listening, acknowledge, speak or yield.

The idea: keep the semantic tokens, drop the codec ​

The starting point is a talk by Neil Zeghidour (Kyutai). A neural codec such as SoundStream or Mimi turns audio into "almost characters", a few codes per frame, and a language model can then predict those codes like text. That is how AudioLM, Moshi and their full-duplex agents work. One remark in it fits skycmd well: a small model holds the conversation and calls a large one when it does not know.

The codecs themselves are not small:

parameters
Mimi encoder (Kyutai)about 48M
SNAC19.8M
EnCodec15M
s6-ear-nano, the whole decision model1.5M (1,532,449 trained; 1.6M with Parallax, which the s6 ablation did not adopt)

Two published results settle the design:

  • Continuous features beat discrete indices for classification (DASB, arXiv 2406.14294; DualTurn, arXiv 2603.08216).
  • Mimi's first codebook is semantic. Kyutai distilled it from WavLM, and its codes for LibriSpeech are already published under CC-BY-4.0 (shangeth/librispeech-mimi-codes).

So s6-ear never runs a codec. It reads cheap log-mel features and learns to predict Mimi's semantic codes during pretraining. The teacher's knowledge goes into 1.6M ternary-friendly weights, and Mimi never runs on our machine.

A front-end you can read in one sitting ​

crates/sky-infer/src/audio.rs has no dependency:

  • Spectrum: a 25 ms periodic Hann window, a 10 ms hop and a 512-point radix-2 FFT.
  • Bands: 40 mel bands from 20 Hz to 8 kHz, then y = (log10(max(mel, 1e-8)) + 2) / 4.
  • Frames: 8 hops are stacked into one state frame of 320 values every 80 ms, exactly Mimi's 12.5 Hz frame rate. The state is the last 128 frames (10.24 s).

The same file runs natively, in the browser and in the data pipeline that writes the training features. Training therefore sees exactly the frames the engine computes at inference.

The smart detail: a log without libm. The first WASM parity test failed by 3e-5. The cause was f32::sin in the test signal and, potentially, log10 in the front-end: libm differs by one ULP between macOS and WASM. The front-end now computes its log from IEEE basic operations only (an atanh series), and the test signal is generated by a rotation recurrence. Native and WASM features now agree to 1e-12, and the decisions to 2.4e-7.

The model: s6-nano with an ear ​

configs/s6-ear-nano.json keeps the s6-nano shape: d = 144, 2 blocks looped twice, four early exits, Parallax attention. The only change is the state stream:

  • The frame projection. audio_proj · frame + bias ([144, 320], float, like s6-cast's patch projection) replaces the token table, with one state position per 80 ms frame.
  • The question stays text. The question and its options are still BPE text in the static stream, so you can ask about words the model never heard in training. A classic keyword spotter has a fixed output layer and cannot do that.
  • The format. Two GGUF keys, state_kind = "audio" and audio_frame = 320. A file without them is an ordinary text model, byte for byte.

Parameters (fixed by a test): 1,615,681 for the decision model, which is s6-nano plus the 46,224 of audio_proj. That is 1/5 of Smart Turn v3.2, the 8M end-of-turn detector we compare against. The trained model has 1,532,449, because the s6 ablation did not adopt Parallax and its probe rows.

Pretraining: "Mimi as the teacher" ​

sky-train s2-pretrain --audio runs BEST-RQ / HuBERT-style masked prediction:

  1. Crops of up to 128 frames, 30% of them masked in spans of 3.
  2. Each masked frame's projection is replaced by a learned mask vector.
  3. A small head predicts the frame's Mimi codebook-0 id, out of 2,048 classes.

The pretraining stack has no token table at all, and the 2,048-class head is dropped when the decision model starts. In the smoke run, the loss left the uniform value (7.6) within 30 steps, at about 100,000 frames per second on an M5.

The audio jev ​

sky-data fetch-audio and sky-data audio-jev turn public CC-BY-4.0 clips into decision records, one per question:

questiontypedata
"Has the speaker finished their turn?"noulSmart Turn's data (23 languages, French included)
"Which word was said?", 12 optionschoiceSpeech Commands v0.02, the standard benchmark
the same question on words never trained onchoice, 10 optionsMSWC, English and French: 50 words per language held out
designed, never run: "What should I do now?" (listen, backchannel, speak, yield)choiceAMI, by a pre-registered rule (the AMI downloader was not written)
designed, never run: questions about the contentnoul / choiceGLM-5.3-flash reads the LibriSpeech transcripts, and s6-ear learns to answer from the audio (no transcript was labelled)

The last row is cross-modal distillation. The teacher that already labels text for s6-nano also labels what was said, and the student learns to answer from the sound. That replaces the spoken-intent datasets, which are all non-commercial (Speech-MASSIVE, SLURP).

Two choices made for honesty, before any run ​

  • The right test set. The plan drew the end-of-turn test clips from smart-turn-data-v3-train. That is Smart Turn's own training set, so the bar would have been measured on data the baseline had memorized. The test is now Pipecat's official held-out set, smart-turn-data-v3.2-test, and every train clip with a test id is removed.
  • A fair zero-shot test. Each held-out word gives exactly 20 test questions, so the right answer is uniform over the held-out words, and no frequency prior can pass for hearing.

Both choices are written into the pre-registration before any full download or training.

System 1, with a guarantee ​

Two thresholds per question type, both chosen by conformal risk control on the val split (scripts/ear-gate.py crc):

  • Early exit, as for the games: answer at the first exit that is confident enough. The answer may differ from full depth in at most α = 2% of the cases.
  • Hand-off to System 2: below a second threshold, s6-ear says "I don't know" and passes the clip to a larger model (Whisper, Smart Turn). The error among the decisions it keeps is bounded by 5%.

This is the routing tested on LFM3-1.7B, applied to voice. A tiny frontal model decides every 80 ms, and the large model is called only when an answer is really needed. Kyutai's MoshiRAG does the same with a 7B frontal model.

The gates ​

Fixed in advance, read on the ternary model at full depth:

  1. End of turn at least as accurate as Smart Turn v3.2 on the same clips.
  2. Speech Commands (12 classes): at least 0.90.
  3. Zero-shot keywords: at least 2 × chance (0.20) in English and in French.

A control is trained alongside: the same model and budget without the Mimi pretraining, so the report shows what the distillation brings.

Try it ​

bash
MAX_CLIPS=200 scripts/train-ear.sh data      # smoke data: 200 clips per set
scripts/train-ear.sh data pretrain decide eval

The Listen page runs it on your microphone or on sample clips once a model passes its gates. In your own page:

js
const r = JSON.parse(engine.decideAudio('noul', 'Has the speaker finished their turn?',
  JSON.stringify(['false: still talking', 'true: finished']), pcm16k))  // Float32Array

To try the page with a model of your own: build the site and serve the build (npm --prefix site run docs:build, then npm --prefix site run docs:preview), because under docs:dev Vite refuses to import the WASM module from /public, as on the other demo pages. Copy the model into site/public/ear/dev/ (ignored by git) and open /skycmd/ear/?model=/ear/dev/<file>.gguf: the model parameter replaces the s6-ear-nano file of the page's second demo (the decision model with keywords).

The measured costs in the smoke run, both at full depth:

  • Native: 0.6 ms per decision for 1 s of audio and 5 ms for 10 s, log-mel included.
  • WASM: 0.67 ms per decision (median over 20 clips).

The first run: three gates failed ​

Trained in November 2026 in a braked mode, so the budgets count working minutes: 48,626 pretraining steps, then 48,205 decision steps. A first decision run of 31,480 steps was discarded before any evaluation because it ran in swap (amendment 4). The control, the same model without the Mimi pretraining, did 49,131 steps. Test split, ternary, full depth (runs/s6-ear-nano/eval/gate.json):

gates6-ear-nanocontrolbarverdict
end of turn (31,527 clips)0.56810.76470.9263, Smart Turn v3.2 (Pipecat's report; the v1 harness read 0.8518)fail
Speech Commands, 12 classes0.88850.92170.90fail
zero-shot keywords, English0.21000.30200.20pass
zero-shot keywords, French0.13300.15200.20fail
  • The pretraining task was learned. Codebook-0 accuracy on masked validation frames reached 0.2196, 45 times the technical bar of 10/2048.
  • It did not transfer. The control, starting from random weights, is more accurate on every set: in this run, the Mimi pretraining brought nothing to the decisions.
  • The verdicts do not change at the conformal thresholds. There the model uses 3.42 of its 4 passes on average.
  • Not computed by the run: the precision and recall of its end-of-turn answers, which the pre-registration asks for. Smart Turn v3.2 has 0.783 and 0.971.
  • Latency, read afterwards from the run files (final report): 3,435 µs per decision natively (median over all test sources) and 6,696 µs in WASM (median over 20 test clips).

It failed its gate; its file is published as a result (owner's decision of 15/11/2026), as evaluated: the Listen page loads it under a notice with these numbers and answers at full depth. Its control is not published.

The second version passed. s6-ear2-nano does one job, end of turn, as a 632 KB student of Smart Turn v3.2 that reads the last 4 s of audio: 0.8993 on the same 31,527 test clips (bar 0.85; the teacher 0.9260 through the same fixed harness), a false cut-off of 0.0847 and 2.39 ms per decision in WASM. It is weaker on real conversations, where it answers "finished" on 0.5253 of the real ends of turn of eot-bench at its cautious threshold (results, the article).

The third version: trained on real conversations, gate failed ​

s6-ear v3 (pre-registration) kept v2's graph and front end and changed only the training, to attack the weakness above. Two things were missing from v2's data: real conversations, and the silences the demo actually asks about (the energy VAD asks 0.2 s into a pause, then every 0.2 s, while v2's training clips end at the end of speech).

  • Labels from the other speaker. otoSpeech (about 280 h of spontaneous English, gated, CC BY 4.0) records each speaker on a separate channel. The demo's energy VAD and policy, ported to Rust and checked bit for bit against turn.ts, are replayed on one channel; each pause they would ask about becomes a record. Its label comes from the other channel: end of turn when the other speaker takes the floor (speaks for at least 1 s, not a backchannel) within 4 s, hold when the same speaker resumes first. No model writes a label.
  • Checked before any training. A second, neural detector (Silero VAD) relabelled the validation records and agreed on 0.9509 of them (bar 0.85); the silence after a pause, alone, did not predict the label (AUC at most 0.4396, bar 0.60), so the other voice does not leak into the channel.
  • Trained from v2 for 120 working minutes, half on those records cut 0.2 to 2.4 s into each pause, half on Smart Turn's clips extended by their own background noise.

The gate asked for a measurable gain on eot-bench, real speech addressed to an agent, at the policy's first question. It failed (E1, E2 and E3):

s6-ear3-nanos6-ear2-nanoSmart Turn v3.2
eot-bench AUC, 0.2 s into each pause (16,230 cuts, 14 languages)0.71970.74480.7873
held-out human-to-human conversations, AUC (30,000 records, reported)0.68780.60310.6430
Smart Turn's test clips, accuracy (31,527)0.89780.89930.9260

The change on eot-bench is −0.0252 [−0.0336, −0.0167], where at least +0.02 with a lower bound above 0 was asked. The pre-registration had written down, before the run, how to read this case: since v3 improved on its own held-out conversations, the human-to-human signal was learned and does not transfer to speech addressed to an agent. v3 was designed after v2's eot-bench reading, and its label cross-check was amended after a measurement on five training sessions; both are disclosed with the result. Its file is published as a result, with a Space; s6-ear2-nano stays the file to use (results, the article).

Apache-2.0. Civil use only. No trackers, no cookies: scores stay in your browser.