Skip to main content

Music Generation: Jellectronica on SL2610-Series

Finish The Getting Started Guide First

Complete the Getting Started with SL2610 guide first. It covers setting up your board, cloning the examples, and creating the Python environment. Activate the environment with source .venv/bin/activate in each new terminal.

Jellectronica Lite turns a video stream into ambient music in real time. A YOLOv8 object detector runs on the Torq NPU, and the position of each detected object is mapped onto a musical grid. As creatures move through the frame they trigger notes, chords and arpeggios.

By default the demo watches a bundled jellyfish video. It can also use a live YouTube stream such as the Monterey Bay Aquarium moon jelly cam, an IP camera, or any video file.

An optional MelodyRNN layer listens to the triggered notes and generates an accompanying melody with a pre-trained Magenta LSTM. Everything runs on the board: NPU inference, melody generation, audio synthesis and rendering.

Jellectronica

How it works​

video source ─→ Torq NPU (YOLOv8 int8, 320x320) ─→ tracker (8x4 grid)
│
┌───────────────────────────┼──────────────────┐
▼ ▼ ▼
SoftSynth MelodyRNN Display
(USB audio out) (optional LSTM) (DSI or browser)

When a detected object crosses a cell boundary in the 8x4 grid, it triggers a note. The four rows map to different instruments (arpeggios, pads, chords and bass), so more objects in the frame make a richer sound. The synthesizer and MelodyRNN are pure Python and NumPy, so no TensorFlow or Magenta package is required.

Run Jellectronica on Astra​

Hardware​

  • Synaptics Coralboard or Astra Machina SL2619 kit
  • A USB speaker or headset (or a Bluetooth speaker, see below)
  • Optional: a Waveshare 7" DSI display. Without a display the app streams to a browser.

Install the example-specific dependencies and download the models:

cd jellectronica
pip install -r requirements.txt
python setup_demo.py

To use a Bluetooth speaker instead of USB audio, pair it using the Astra Linux Bluetooth guide.

Music from video on the edge​

Headless mode: stream to a browser​

python3 run_cli.py

Open http://<board-ip>:5002 in a browser on the same network. The video and detection overlay stream to the browser and the music plays through the board's speaker. Use --port to change the port.

Display mode: DSI or HDMI screen​

Set up the display environment from the Getting Started guide (XDG_RUNTIME_DIR and WAYLAND_DISPLAY), and add:

export WESTON_DISABLE_GBM_MODIFIERS=true

For portrait mode on an 800x480 display:

export ORIENTATION=portrait
export DISPLAY_HEIGHT=800
export DISPLAY_WIDTH=480

Then start the app, which renders directly to the display:

python3 run_gui.py

Choose a video source​

SourceOption
Bundled jellyfish video (default)no option needed
YouTube livestream (jelly cam)--youtube
Another YouTube video--youtube <URL>
Local file, IP camera or HTTP stream--video <path-or-URL>
python3 run_cli.py --youtube
python3 run_cli.py --video rtsp://192.168.1.100:554/stream

If YouTube is unavailable, or the stream goes black, the app falls back to the bundled video.

Additional options to try​

  • --no-ai, turn off the MelodyRNN accompaniment
  • --model <file>.vmfb, use a different detection model
  • --audio-driver alsa --alsa-device hw:0,0, pick the audio output device

Make it your own​

The musical grid works with any YOLOv8 detection model, and all detected objects trigger notes regardless of class. To make your dog, a bird feeder or street traffic play music:

Follow the steps in the blog Deploy a Custom YOLO Model on the Synaptics Torq NPU to create your own model. Here is a short summary:

  1. Train a YOLOv8 model on your objects with Ultralytics.
  2. Convert it for the NPU with the Torq compiler.
  3. Copy the .vmfb file to the board.
  4. Run python3 run_cli.py --model ../models/your_model.vmfb.

Models​

This tutorial uses the following pre-compiled model for the SL2610-Series. See the model page for details, performance and download links:

Browse all models on the Models page.

Python walkthrough​

  • detector.py, runs YOLOv8 on the Torq NPU
  • tracker.py, tracks objects and maps them to the grid
  • music_engine.py and soft_synth.py, the five-channel audio engine and built-in synthesizer
  • melody_rnn.py, NumPy LSTM inference for the AI accompaniment

See the example's docs/architecture.md for more detail.

note

The jellyfish video is provided courtesy of the Monterey Bay Aquarium for non-commercial use only.

Explore other applications​