Real-time Speech Recognition on SL2610-Series

Automatic speech recognition (ASR) enables intelligent, voice-enabled devices across a wide range of applications. Thanks to advances in AI models and hardware acceleration like the Torq NPU from Synaptics, developers can now deploy real-time speech recognition in low-power products. This unlocks more intuitive ways for users to interact with products. If you're interested in bringing real-time speech recognition to your own edge AI applications, this blog will show you how. You can also try it out using a demo app.

What you'll need for the demo:

1. Astra Machina SL2619 or Synaptics Coralboard running Astra SDK v2.4 or later Linux OS.

2. USB microphone

If you want to jump straight to the demo, click here.

Moonshine V2 - Now with Streaming!

Moonshine is a family of speech recognition models designed for low-latency transcription developed by Moonshine AI (Jeffries et al., 2024; Kudlur et al., 2026). They provide tiny models, as small as 34 million parameters (~130MB), that you can run on Astra devices in real-time even in floating-point precision.

The base Moonshine model is based off the classic encoder-decoder transformer model (Vaswani et al., 2017). It consists of an encoder that looks at the full length of the speech to capture the context and passes this "encoding" onto the decoder that autoregressively produces the transcription. Moonshine V2 later released in 2026 improved by allowing a sliding window attention in the encoder part of the model. This means we no longer need to wait for the full-length of speech to start emitting the first token, reducing the time-to-first-token (TTFT) as the encoder can now "stream" the encoding frames to the decoder.

The Astra SL2610 and the Torq NPU

The SL2610 is a heterogeneous SoC:

• Arm Cortex-A55 cores (CPU)

• Mali-G31 GPU

• Torq T1 NPU (1 TOPS)

Torq is Synaptics' silicon and compiler stack built around Google's Coral NPU, an open-source RISC-V NPU core with native support for AI workloads. By design, the NPU is optimized for matrix operations prevalent in the AI era.

We will be running the full ASR system on the chip, placing the Moonshine V2 model on the NPU. See the performance gains table on why we want to do this.

Preparing the model using Torq

So we want to put the model on the NPU. Here's the list of steps:

1. Get ONNX model components

2. Modify the model components - Make them static and apply optimizations

3. Use Torq toolchain to get binaries

Get ONNX model components

In my case, I downloaded the model.safetensors file from Huggingface (UsefulSensors/moonshine-streaming-tiny) and exported the onnx components using torch.onnx.export(dynamo=True). Sometimes you will be able to get ONNX models directly from Huggingface but working from the PyTorch level and the weights means, you get the ability to apply edits at this level. For instance, we can design our own model interfaces, modify component functions, and fuse/decompose model components much easier.

Moonshine Voice splits the full model into 5 ONNX components: frontend, encoder, adapter, cross_kv, and decoder_kv.

Figure 1: ONNX model components of Moonshine V2.

Component
frontend	Stateful audio front-end that takes a raw audio chunk + rolling conv buffers, runs the strided causal conv stack, outputs feature embeddings + updated buffers.
encoder	The transformer encoder layers (stateful) that turn frontend features into contextualized hidden states, using a causal/sliding-window attention mask so it can run chunk-by-chunk.
adapter	A small projection (pos_emb + proj) that reshapes encoder hidden states into the form the decoder's cross-attention expects.
cross_kv	Precomputes the cross-attention K/V projections from the encoder hidden states once per chunk, so the decoder doesn't need to recompute them at every generated token.
decoder_kv	The autoregressive decoder: takes the previous token + self-attention KV cache + the precomputed cross-attention KV, emits the next token and updated self-attention cache.

Modifying the model components

Reduce unnecessary data transfers by fusing some components

For a given chunk size of the input audio, the components prior to the decoder_kv can work in sequence. We can optimize the split by fusing them. This will limit the data transfers from the device to the host (NPU - CPU), keeping the full model in place[1]. We shall call this fused part the "encoder" and the decoder_kv simply the "decoder".

Convert dynamic dimensions to fixed-length (static)

Since the Torq T1 NPU supports fixed-length I/O and operations, we will need to update the dynamic dimensions to some fixed size. The pipeline currently uses ONNX surgery, ONNX_graphsurgeon, to do this.

Decompose exotic operations into common

Additionally, in the scenario that certain exotic ops are not supported by the current version of the torq compiler (make sure you pull the latest updates! They get updated often), we can temporarily support them by decomposing them into simpler ops[2]. In the frontend, for example, I had to fit a polynomial to the asinh ops, and the layernorms in the encoder need to be decomposed due to issues with MLIR lowering.

Compiling the model components to VMFB files (binaries for the NPU)

Once we have prepared static ONNX files, we can now pass it onto the torq-compiler to generate the binaries to run on the Astra SL2610. You can follow the documentation in that repo to "lower" ONNX models. First, we use IREE tools to generate an optimized MLIR file before passing it off to Torq which outputs binaries .vmfb.

Full export pipeline from Synaptics (automating the conversion process)

The torq-tools repo's export pipeline already automates the entire process to modify, optimize, and compile the model components. We can run the full export using the following (make sure to read the readme and set up the env first):

torq-export-model moonshine_streaming --chunk-len 1280 --input-seconds 8 --extract-embeddings --export-attention --convert-dtypes

Here are some useful arguments:

• --chunk-len (required, no default): audio samples per streaming chunk fed to the encoder (e.g. 1280 samples = 80ms @16kHz).

• -i/--input-seconds (default 5): bounds the decoder's input (Cross KV-cache) and its Self KV-cache. In other words, this limits the total audio length (context) that the static decoder will be able to attend to at a time.

• --extract-embeddings: switches the decoder's first input from an integer token id to a float inputs_embeds tensor, moving the embedding-table lookup host-side (paired with a separately-exported decoder_token_embeddings.npy).

• --convert-dtypes: precision conversion. Make sure to have this when generating .vmfb files.

For our current demo, I chose 8 seconds of maximum transcript length with the encoder processing 1280 samples of audio (80ms) at a time. This should produce encoder.vmfb and decoder.vmfb files inside

torq-tools/models/UsefulSensors/moonshine-streaming-tiny/export/torq/converted/static

Also notice the five files: adapter_pos_emb.npy, decoder_token_embeddings.npy, streaming_config.json, tokenizer.json, and config.json generated in the directory below:

torq-tools/models/UsefulSensors/moonshine-streaming-tiny/export/onnx/float/static

Running the demo

Here's a simple demo of real-time speech transcription using the Moonshine V2 model, adapted from the Moonshine Voice repo, that runs fully onboard the Coralboard or Machina Kit. As you speak into the mic connected to the board, the transcript updates as you speak, with what we call a "preview", which updates itself as more audio comes in. As we mentioned before, this is only possible due to the improvement of V2, the streaming version, where the encoder uses a sliding window attention to produce embeddings into a growing buffer (instead of re-processing the whole clip every time).

Ensure your development board is running the sl2619_oobe_scarthgap image of Astra SDK (Yocto Project Linux) version scarthgap_6.12_v2.4.0 or later. Check out the Astra SDK Documentation and Astra SDK Releases for more details.

Follow along with the README.md files in torq-examples, and under the moonshine_streaming directory, run the following command on your board.

python src/infer.py -m ../models/Synaptics/moonshine-streaming-tiny-torq \
    --hw-type astra_machina --device <mic> --vad-silence 8 --vad-threshold 0.010 --preview-every 5

Make sure you have a mic connected to the board (I use a USB extension to keep the mic away from the buzzing on the board). You can check your --device number by running:

python src/infer.py --list-device

You can also provide a pre-recorded WAV file and simulate the real-time transcription:

python src/infer.py -m ../models/Synaptics/moonshine-streaming-tiny-torq \
    --vad-silence 0.5 --wav ../../librispeech_clean.wav --realtime

Figure 2: Transcription output

The transcription is quite accurate compared to the groundtruth:

Yesterday you were trembling for a health that is dear to you. Today you fear for your own. Tomorrow it will be anxiety about money. The day after tomorrow, the diatribe of a slanderer. The day after that, the misfortune of some friend. Then the prevailing weather. Then something that has been broken or lost. Then a pleasure with which your conscience and your vertebral column reproach you. Again, the course of public affairs.

As you might have noticed, I set the --vad-silence to 0.5 seconds since the audio is crisp with low noise and it's easy for the VAD to pick up on the speech. When testing with your own mic, make sure to adjust this knob accordingly since the background noise can make your transcription break up too often.

Below is the workflow of the algorithm:

Figure 3: Real-time streaming transcription pipeline.

A worker thread continuously pushes raw audio into a queue. A separate thread resamples the audio to 16kHz and chunks it into fixed 80ms (1280-sample) slices and then runs a lightweight energy-based VAD to determine the current phase of speech. If you are speaking, it feeds the chunk to the encoder that runs on the NPU. The encoder requires some lookahead so we need to throw away some warmup encoder output frames, but every audio chunk appends a fixed batch of new frames onto a cross-attention buffer (cross-KV cache). We then launch the decoder every so often (say every 15 chunks) to preview what has been said.

Design Considerations

Here are some design issues that come up:

1. The model components require static shapes: Since the decoder needs a fixed cross-KV input and self-KV buffer, we need to fix the utterance length. Moreover, each decode will run on the full utterance.

2. The memory boundary between the host CPU and the device NPU: We want to limit the memory transfers across, so we need to make the cross-KV and self-KV caches device resident.

3. When and how do I run the decoder? Should we re-decode the full audio from the start of the utterance?

The easy part: making the KV caches device resident

This is a key optimization. We need to keep the data resident on the NPU instead of shuttling it back and forth over the slow memory connection:

1. Cross-attention KV cache is constant for each decode call.

2. Self-attention KV cache is reusable across the forward passes of the decode call.

Just by keeping both these caches device resident, we can significantly cut down the expensive memory transfers. Additionally, because the model components and I/O are static, implementing this feature is simple.

The tricky part: deciding when to "commit" to a token (word fragment)

The most obvious and accurate way to get a live preview is to simply re-decode the whole transcript from scratch. After all, the decoder will have better context of the full-length audio, that is, the maximized cross-KV cache. But as you can see, this can be prohibitively slow as the cost of compute grows with the square of the input audio length. Moreover, because we have static model components, each decoder call is much more expensive with the masked inputs.

Instead, we can cut this cost down by committing the prefix of the preview as the audio progresses. We are betting on the idea that with enough context, the initial transcriptions are accurate and can now be finalized (it's just a waste to redo them since they would just be the same tokens).

So I currently use two heuristics to decide when to commit a token:

1. It stops changing (LocalAgreement-2): the token is consistent across the last two consecutive decodes.

2. It is old: the token is at least 3 seconds of audio behind the frontier.

Both of these heuristics can be tuned to your particular use case by just trying some numbers out for each separately and seeing how consistent they are with the "golden" full-context decode.

The harder part: continuous streaming

With my current setup, the transcription is not truly a streaming transcription with infinite audio length. In particular, we are limited to streaming within the fixed utterance limit (set to 8 seconds).

Even so, we can have a pseudo-streaming user experience by coordinating with the VAD at the system level. When we detect long pauses, we can reset the transcription pipeline and the model components' states and their buffers. Provided we don't have long utterances past the 8-second fixed window, we will experience a smooth streaming transcription.

Other strategies like sliding windowing of the cross KV cache, or buffering and stitching across utterance boundaries may affect the accuracy, but are worth exploring.

You are invited to investigate how to coordinate the model at the system level to provide the best streaming transcription.

Summary

In this blog, we took a look at:

• a brief overview of the Moonshine V2 streaming model,

• a short guide on how to export and prepare the model from PyTorch,

• a simple demonstration of the real-time streaming speech transcription, and

• some design problems and their remedies.

What's next?

• As we discussed earlier, there is still some work to be done to enable better coordination at the system level to handle the breaks across two utterances.

• For further improvements on latency and memory footprint, we can employ INT8 quantization, particularly on the decoder, at the cost of some accuracy loss.

References

• Jeffries, N., King, E., Kudlur, M., Nicholson, G., Wang, J., & Warden, P. (2024). Moonshine: Speech recognition for live transcription and voice commands. arXiv preprint arXiv:2410.15608.

• Kudlur, M., King, E., Wang, J., & Warden, P. (2026). Moonshine v2: Ergodic streaming encoder asr for latency-critical speech applications. arXiv preprint arXiv:2602.12241.

Additional Notes

1. As we shall see later, we can put the i/o all on the device as well but fusing them removes the hassle here.
2. Be careful not to overdo this since it can either mess with the accuracy or the latency.

Some performance numbers

Tiny model

Component	CPU (ms)	NPU (ms)	Speedup	DRAM: CPU → NPU (MB)	Memory: CPU → NPU (MB)
Encoder	81.6	23.4	3.49×	122.3 → 82.9	90.1 → 41.0
Decoder	37.9	28.3	1.34×	170.5 → 131.3	138.0 → 68.2

Table 1: Performance gains by using the NPU over the CPU on the Coralboard for the 34 million model (tiny).

Small model

Component	CPU (ms)	NPU (ms)	Speedup	DRAM: CPU → NPU (MB)	Memory: CPU → NPU (MB)
Encoder	409.0	174.2	2.35×	361.1 → 526.3	328.9 → 262.9
Decoder	99.0	68.6	1.44×	391.4 → 313.4	358.9 → 164.2

Table 2: Performance gains by using the NPU over the CPU on the Coralboard for the 123 million model (small).

Figure 4: With full re-decode, the number of steps grows quadratically with the audio length.

Figure 5: Hence, the latency of the decode calls when using the full re-decode strategy grows much faster than the incremental strategy.

Figure 6: The queue depth barely catches up to to real time with the full re-decode.