Skip to main content

Physical AI in Action: Autonomous Robotic Arm with Synaptics Astra® SL2610

· 21 min read
Nicholas Christoffersen
Nicholas Christoffersen
Embedded AI @ Synaptics

This guide covers the end-to-end process of teaching a low-cost robot arm to do a pick-and-place task and then running a vision and motor control policy on a Synaptics Astra SL2610-Series development kit — no cloud, no GPU, and no host PC needed at inference time. This is a hands-on example of Physical AI: models that perceive and act in the real world, running entirely on-device.

Video 1: Autonomous Pick-and-Place with Astra SL2619 - No Cloud, No PC, No GPU

Hardware​

Models / Policies​

We use Action Chunking with Transformers (ACT), a lightweight imitation-learning policy — the standard first test for simple manipulation tasks on arms like the SO-101.

Each inference takes:

  • One image per camera
  • The robot's 6 motor positions

Then it predicts the next 100 motor positions — an "action chunk", ~3.3 seconds of motion at 30 Hz. Committing to a chunk of actions rather than a single action gives the policy a forward time horizon: it follows through on a motion instead of re-deciding every step and dithering in place. It also means the policy only has to re-plan a few times per second rather than at every 33 ms control tick, so the arm keeps moving smoothly while the next inference (~320 ms on the board) is still in flight — and it opens the door to asynchronous re-planning, which we use in Step 6.

The ACT policy has two pieces:

  • A ResNet-18 backbone per camera: 320×240 frame → feature vector
  • A small transformer (4 encoder layers, 1 decoder layer): image features + encoded motor positions → action chunk

Software​

Tested with
  • Host: x86-64 Ubuntu 24.04, Python 3.12, lerobot==0.6.1
  • uv 0.12.13 on both host and board
  • Torq v2.2.0 compiler and runtime wheels, torq-tools @ d605a4d
  • Board: Astra Machina SL2619, Astra SDK image sl2619_scarthgap_6.12_v2.5.0, uv-managed Python 3.12
  • Example scripts: sl2610-act-example

Performance​

For the models (datatype: bf16) running on the NPU:

  • Inference time: ~320 ms, ~6× faster than the CPU[1]
  • Joint fidelity: under 1 degree (0.08° mean per-joint error against the fp32 reference)

The Journey​

  1. Acquire parts
  2. Record demonstrations — ~1 hr, 50 episodes
  3. Train the model (ACT) — a few hours on a GPU
  4. Get familiar with the Torq AI Platform — ~30 min
  5. Compile for the Torq NPU — a handful of commands
  6. Run on the board — live @ 30 Hz
  7. Going faster — Coming in part two of this blog!

Steps 1–3 are standard LeRobot. If you've trained an SO-101 policy before, skip to Step 4.

Steps 4–6 are new. They cover how to take a PyTorch policy and get it running fast on Synaptics Astra. Step 4 introduces the toolchain; Steps 5–6 get the policy running on the board.

1. Acquire the parts​

WhatNotes
SO-101 arm (follower + leader)3D-printed + hobby servos. Print it yourself or buy a kit. Follower does the work; leader is the puppet you demonstrate with.
2× USB camerasOne wrist cam (on the gripper) + one top/overview cam. Get cameras that natively output QVGA (320×240) MJPG — see the note below.
Synaptics Astra Machina SL2610 (or Synaptics Coralboard™ SL 2GB)The board that runs inference. It has the NPU we target.
A dev system (for training only)A modern CPU + GPU combination trains this in less than an hour (<30 min on an AMD Ryzen 9 5900X + Nvidia 5060 Ti).
Pick native QVGA cameras

We found that QVGA offered sufficient resolution for the simple pick-and-place task, and significantly increased model speed. Prefer cameras that stream compressed MJPG: the two cameras share a USB 2.0 bus, and two uncompressed (YUYV) streams may overload the USB bandwidth, especially at VGA resolution. A VGA-only (640×480) camera also works by downscaling to QVGA as a preprocessing step. However, if you apply any image preprocessing, it should be consistent across training and inference.

A note on the GPU

Older or smaller cards (a GTX 1060, say) will train, just slower. If your GPU for training is modest, drop --steps (a single well-demoed task converges in 5-30K steps in our experience) or rent a cloud GPU by the hour. LeRobot also runs on Apple Silicon (mps).

Build the arm and install LeRobot on your laptop as per the SO-101 ARM Github and LeRobot SO-101 Guide. The SO-101 uses Feetech servos, so you need that extra plus the robot-workflow and training extras -- a bare pip install lerobot is missing the motor and CLI deps:

# uv is much faster at the dependency resolver than native pip; install the pinned version
# (other install methods: https://docs.astral.sh/uv/#installation)
curl -LsSf https://astral.sh/uv/0.12.13/install.sh | sh
source $HOME/.local/bin/env   # put uv on PATH in this shell
uv venv -p 3.12 venv_lerobot   # lerobot 0.6.1 requires Python >= 3.12
source venv_lerobot/bin/activate
uv pip install 'lerobot[feetech,core_scripts,training]==0.6.1'
# feetech      → SO-101 servo driver + serial deps
# core_scripts → lerobot-record / -replay / -calibrate / -teleoperate
# training     → lerobot-train (+ wandb, accelerate)

Find your motor ports and try to teleoperate the robot -- it will have you calibrate the robot automatically if you haven't already done so:

lerobot-find-port                         # tells you which /dev/ttyACM* is which
lerobot-teleoperate \
  --robot.type=so101_follower --robot.port=<FOLLOWER_PORT> --robot.id=my_follower \
  --teleop.type=so101_leader --teleop.port=<LEADER_PORT> --teleop.id=my_leader

If the teleoperate command fails with a permission error on the port, add yourself to the group that owns serial devices:

sudo usermod -aG dialout $USER   # Ubuntu/Debian; on Arch the group is uucp

Logout/login for changes to take effect and try again.

note

Keep that follower calibration file; the board needs the exact same one later so the policy's joint angles map to the real motors.

2. Record demonstrations​

Video 2: Teaching the Policy Through Imitation

You teach the policy by doing the task yourself: move the leader arm by hand, the follower mirrors it, and both cameras + all joint angles get recorded. This is imitation learning: no reward functions, no simulation, just show it what "good" looks like. For detailed instructions, see LeRobot Imitation Learning Guide.

First find out which cameras are yours:

lerobot-find-cameras

Then start recording episodes:

# to push your dataset to the Hugging Face Hub (the hf CLI comes with lerobot)
hf auth login

lerobot-record \
  --robot.type=so101_follower --robot.port=<FOLLOWER_PORT> --robot.id=my_follower \
  --teleop.type=so101_leader  --teleop.port=<LEADER_PORT>  --teleop.id=my_leader \
  --robot.cameras="{ wrist: {type: opencv, index_or_path: 0, width: 320, height: 240, fps: 30, fourcc: MJPG},
                     top:   {type: opencv, index_or_path: 2, width: 320, height: 240, fps: 30, fourcc: MJPG} }" \
  --dataset.repo_id=<HF_USERNAME>/pick_place_demo \
  --dataset.push_to_hub=true \
  --dataset.single_task="Pick up the block and place it in the bin" \
  --dataset.num_episodes=50
fourcc: MJPG

Asks the camera for a compressed MJPG stream, check your specific cameras for supported resolution/format combinations.

Tips that actually matter:

  • Vary the starting position of the object every episode. Consistency in how you do the task, variety in where things start is what makes the policy robust.
  • ~50 episodes is enough for a small task -- e.g. place location fixed with varying pick location. For more complex tasks (i.e. both the pick and place location varying), consider collecting more data if you don't achieve good results quickly.
  • QVGA (320×240) is deliberate. Small frames keep the model small and the board fast. You don't need 1080p to pick up a block.
  • The dataset uploads to the Hugging Face Hub automatically (often with a timestamp); use the 'visualize dataset' tool on Hugging Face to review episodes (highly recommended).

3. Train the model​

We train an ACT (Action Chunking with Transformers) policy: a ResNet-18 image backbone (shared by both cameras) feeds a small transformer (4 encoder layers + 1 decoder) that predicts a 100-step chunk of future joint actions at once. It's often the first "proof of concept" model one uses on a new robot, and has first-class support in LeRobot.

If you train on a different machine, repeat the venv setup from Step 1 there. The training extra already includes wandb; log in once to watch training stats (or drop --wandb.enable=true).

wandb login
lerobot-train \
  --policy.type=act \
  --dataset.repo_id=<HF_USERNAME>/pick_place_demo \
  --policy.repo_id=<HF_USERNAME>/act_pick_place \
  --policy.push_to_hub=true \
  --dataset.eval_split=0.10 \
  --eval_steps=1000 \
  --batch_size=8 \
  --wandb.enable=true --wandb.disable_artifact=true \
  --steps=20000

We stuck with the defaults given by LeRobot (as of v0.6.1): dim_model=512, n_heads=8, dim_feedforward=3200, chunk_size=100, a ResNet-18 backbone with ImageNet weights, use_vae=true. The above command holds out the last 10% of recorded episodes as evaluation episodes. We found that training 5-20K steps gives the best results. On a single NVIDIA 5060 Ti 16 GB, this trained in about 25 minutes.

Before touching the board, confirm the policy works on your laptop/desktop driving the real arm, again made simple by LeRobot.

lerobot-rollout \
  --strategy.type=base \
  --policy.path=<HF_USERNAME>/act_pick_place \
  --robot.type=so101_follower --robot.port=<FOLLOWER_PORT> --robot.id=my_follower \
  --robot.cameras="{ wrist: {...}, top: {...} }" \
  --duration=30

If the policy is not completing the task (a good initial goal is 70-90% accuracy), review your training data in HuggingFace's visualize dataset feature. A few good questions to ask:

  • Are objects in view of your top camera?
  • Did you always reach/approach in the same way, from the same angle, etc?
  • Do you have enough training data?

If it runs well with inference done by your laptop/desktop, you have a good policy. Now make it run on the edge.

4. Get familiar with the Torq AI Platform​

Everything from here on runs through Torq, Synaptics' toolchain for the Torq NPU. It's the piece that makes "run the model on the board" real. It is based on MLIR / IREE and it is open source on GitHub under the synaptics-torq org. You don't have to reverse-engineer a vendor blob; you can read exactly how your model gets lowered by inspecting the output of the various Intermediate Representations.

Two repos matter:

RepoWhat it is
torq-compilerThe compiler itself: takes an ONNX/TFLite model → MLIR → a .vmfb module the Torq NPU runs. Built on IREE/MLIR. It can also simulate the hardware, so you can test a compile without a board on your desk.
torq-toolsThe surrounding toolbox: model exporters, dtype conversion (convert_dtype, e.g. fp32→bf16), and the glue that drives the compiler. This is where the "prepare the model" scripts live.

Full docs — install, quickstart, the torq-compile CLI reference — are in the Torq Documentation.

The mental model for the whole flow:

 any trained model (onnx, tflite, etc.)  ──►  MLIR  ──►  Torq (compile)  ──►  .vmfb  ──►  runs on the Torq NPU

Our ACT policy is just one thing you can push through it. The rest of this guide is that pipeline applied to our robot.

5. Compile the model for the Torq NPU​

A PyTorch policy can't run on the Torq NPU directly — it has to be lowered to a .vmfb module the accelerator understands, which is exactly what the Torq compiler does.

  1. Set up the Torq compiler and torq-tools (see the Torq Documentation for details) in a separate venv on an x86-64 Linux host. The onnx extra adds the ONNX importer; torq-tools adds torq-cleanup-model.
uv venv -p 3.12 venv_torq
source venv_torq/bin/activate
# torq-tools pulls in torch; the CPU index keeps the CUDA packages out
uv pip install "torq-compiler[onnx] @ https://github.com/synaptics-torq/torq-compiler/releases/download/v2.2.0/torq_compiler-2.2.0-cp312-cp312-manylinux_2_28_x86_64.whl" \
  "torq-tools @ git+https://github.com/synaptics-torq/torq-tools@d605a4d" \
  --extra-index-url https://download.pytorch.org/whl/cpu --index-strategy unsafe-best-match
  1. Get the two model-specific helper scripts (ONNX export and normalization constants) plus the board scripts for Step 6 from GitHub:
git clone https://github.com/synaptics-astra-demos/sl2610-act-example.git

# your trained checkpoint: the training run's checkpoints/last/pretrained_model directory,
# or `hf download <HF_USERNAME>/act_pick_place --local-dir act_pick_place`
MODEL_DIR=<path_to_your_trained_checkpoint>
  1. Run the pipeline. Every step is one command, and everything after the export is model-agnostic; this is how any exported ONNX model gets onto this hardware. Run the two helper scripts in your LeRobot environment; run the rest in the Torq venv from step 1.
Why the export is the one model-specific step

The conversion itself is PyTorch's standard, generic torch.onnx.export, but it needs a function with plain tensor inputs and outputs, and their concrete shapes. A LeRobot policy doesn't expose that directly: select_action() hides Python-side control flow (the action-chunk queue), normalization lives in separate pre/post-processors, and the VAE encoder only exists for training. So the export script is a small wrapper that selects the deployable subgraph — call policy.model with two 320×240 frames and the 6-value state, return just the action chunk — and hand it to the standard exporter. Swap in a different policy — another architecture, or a different camera setup — and that wrapper is the only thing that changes; the rest of the pipeline below is the same for any ONNX model.

# --- LeRobot env ---
source venv_lerobot/bin/activate
# the ONNX exporter needs these; lerobot doesn't install them
uv pip install 'onnx==1.23.0' 'onnxscript==0.7.2'
# export the trained policy to a single self-contained fp32 ONNX
python sl2610-act-example/build/onnx_01_export.py "$MODEL_DIR" -o act_fp32.onnx
# bake the normalization constants (image/state/action mean-std) the board applies around the model
python sl2610-act-example/build/export_norm_params.py "$MODEL_DIR" norm_params.npz

# --- Torq venv (torq-compile + torq-tools) ---
source venv_torq/bin/activate
# undo the PyTorch->ONNX exporter artifacts (works on any ONNX model):
# merges unrolled per-token Concats back into one op, evaluates constant
# subgraphs offline, and fuses each BatchNorm into the preceding conv.
# --verify re-runs both models and checks the outputs still match.
torq-cleanup-model onnx act_fp32.onnx -o act_clean.onnx --verify

# convert fp32 -> bf16 (fp32 I/O preserved via Cast)
torq-convert-dtype onnx -i act_clean.onnx -o act_bf16.onnx -d bf16

# ONNX -> MLIR -> Torq NPU module
python -m iree.compiler.tools.import_onnx act_bf16.onnx -o act.mlir --data-prop
torq-compile act.mlir -o act_full.vmfb --torq-hw=SL2610 --torq-disable-css --torq-disable-host

This produces a single act_full.vmfb (~70 MB): the image backbone + the full transformer, one module. Inputs are the two camera frames + the robot state; output is the 100-step action chunk (main_graph(image_wrist, image_top, state) → [1,100,6]).

That's it for the default path — a workaround-free pipeline, no hand-editing of the graph. The result is a single ~70 MB module that runs at ~320 ms/inference on the board. The next step shows you how to integrate the vmfb into the board for motor control.

Visualization

It is also informative to use a visualization tool like netron to visualize the onnx version of your model. Compare the original act_fp32.onnx with act_clean.onnx to see what the cleanup script did!

Fidelity

Against the fp32 reference on 30 real frames, the bf16 module tracks to 0.46 % relative-L2 on the action chunk — under 0.1° mean error per joint (worst single time-step 0.91°). That's the pure bf16 rounding floor; no accuracy tuning was applied.

6. Run the model on the board​

Push the compiled module (vmfb), the normalization constants (npz), and your follower calibration to the board (json), then run the control loop. The CPU owns the cameras and motors; the Torq NPU owns inference.

Prepare the board (once)

The board needs one venv with LeRobot (motors and cameras) and the Torq runtime (runs the vmfb on the NPU). On the board (adb shell or ssh; the board needs internet access):

# from the board
curl -LsSf https://astral.sh/uv/0.12.13/install.sh | sh
source $HOME/.local/bin/env   # put uv on PATH in this shell
# --managed-python: the image's own Python lacks stdlib modules we need
# (fcntl for the motor serial port, statistics for the control loop)
uv venv -p 3.12 --managed-python /home/root/act-venv
source /home/root/act-venv/bin/activate
# the CPU-only torch index keeps uv from pulling in CUDA packages the board can't use
uv pip install --extra-index-url https://download.pytorch.org/whl/cpu --index-strategy unsafe-best-match \
  'lerobot[feetech]==0.6.1' 'pyusb==1.3.1' \
  'https://github.com/synaptics-torq/torq-compiler/releases/download/v2.2.0/torq_runtime-2.2.0-cp312-cp312-manylinux_2_28_aarch64.whl'

BOARD_PY in board_config.sh already points at this venv.

The motor serial port. The board image has no USB-serial kernel driver, so plugging in the follower's motor adapter does not create a /dev/ttyACM* (and lerobot-find-port has nothing to find). Instead, ch343_pty_bridge.py (shipped with the board scripts) talks to the adapter from userspace through pyusb and exposes it as /dev/ttyFOLLOWER, which LeRobot opens like any serial port. The adapter must be a CH343 (USB id 1a86:55d3); check with cat /sys/bus/usb/devices/*/idProduct on the board.

note

Then fill in sl2610-act-example/board/board_config.sh with your board address (adb or user@host), the two camera device paths, the camera rotation you recorded with (if any), the motor port, and your follower calibration file.

# from the host — deploys over USB (adb) or ssh, per board_config.sh
bash sl2610-act-example/board/deploy_to_board.sh act_full.vmfb norm_params.npz

The deploy script prints the exact on-board commands. Run three escalating steps — always dry-run before you let the arm move (source ./board_config.sh first so the cameras/motor/calibration are set; the ./ matters in the board's sh):

cd /home/root/act_run_2cam   # BOARD_DEST from board_config.sh
source ./board_config.sh

# motor serial bridge (once per boot) -> /dev/ttyFOLLOWER
"$BOARD_PY" ch343_pty_bridge.py --link "$MOTOR_PORT" > /tmp/ch343_bridge.log 2>&1 &
sleep 2; cat /tmp/ch343_bridge.log   # expect "BRIDGE UP ... /dev/ttyFOLLOWER"

# 0) seam check — cameras → Torq NPU → action, no motion. Confirms the pipeline is wired right.
"$BOARD_PY" board_probe_2cam.py

# 1) dry run — reads motors, computes targets, moves NOTHING.
"$BOARD_PY" board_act_loop_2cam.py --duration 15 --fps 30

# 2) LIVE — the arm moves. Clear the workspace. Start slow (add --vmax 2.5 for a gentler first run).
"$BOARD_PY" board_act_loop_2cam.py --live --duration 60 --fps 30

The loop runs a motor thread and a Torq NPU inference thread in parallel, continuously re-planning the action chunk (~2–3 Hz) and blending overlapping chunks — the same async-chunking scheme LeRobot uses, but with inference on the accelerator instead of a GPU.

Watch what the model sees. Add --stream-port 8080 and open <Board_IP_Address>:8080 in your web browser on the host machine (over USB: adb forward tcp:8080 tcp:8080, then localhost:8080) to watch a real time video feed of what the robot sees during inference, motor measurements and actions, as well as inference speed.

Safety is on by default

The loop is dry-run unless you pass --live; joint targets are clamped to the training envelope; there's a startup ramp and a per-step jump cap (--vmax, default 5°/step); Ctrl-C releases torque. Lower --vmax for the first real run.

At this point you have a working robot running entirely on the board!

7. Going faster​

Quantize to int8 (Coming in part two!)

The bf16 build is great, but quantization lets you trade a little accuracy for speed. A follow-up blog will cover quantizing the ACT model for the best accuracy/speed tradeoff and running that model on Astra with Torq.

What "feasible on the edge" actually means here​

Running on the SL2610 Torq NPU, latency vs. the CPU on the same chip (all measured on the same SL2610 board, same session):

ConfigurationLatency / inferenceAccuracy (joint error)vs. CPU
torq vmfb (bf16) (Steps 5–6)~320 ms0.46% relL2, <0.1°/joint mean~6× faster
fp32 ONNX (ONNX Runtime) on the CPU~1.75 s—1×

The policy drove the physical SO-101 arm through the full pick-and-place task at a 30 Hz control loop, with joint tracking well under a degree — running entirely on the board. No GPU. No cloud. No host in the loop.

That's the point of this whole exercise: real imitation-learning robotics, self-contained on a single low-power edge accelerator.

Troubleshooting​

  • The arm jitters or lags in --live. Raise --vmax (default 5°/step; at 2.5 we saw ~18% of steps capped) so the arm can keep up with what the model wants.
  • Joint angles are wrong / the arm lunges. The board must use the same follower calibration file you made in Step 1. The deploy script copies the host one — don't recalibrate separately on the board.
  • The loop asks "Press ENTER to use provided calibration file…". The motors' stored calibration doesn't match the file you pushed (a different arm, or re-calibrated since). Press ENTER to write the pushed file to the motors; type c only if you really want to re-calibrate on the board.
  • could not open port /dev/ttyFOLLOWER. The motor bridge isn't running (it doesn't survive a reboot) — start it as shown above. If it prints device 1a86:55d3 not found, the motor adapter isn't plugged in or isn't a CH343; Resource busy means another bridge already holds the adapter (kill it first).
  • Cameras swapped. Model input order is (wrist, top). If the arm behaves as if it's looking the wrong way, your two cameras are swapped in the loop config.
  • torq-compile prints tile-and-fuse: failed to compute producer's operand slices warnings and LRAM allocation dumps, but still exits 0. That's the tile-fit search backtracking to a smaller tile, not a fatal error — the vmfb is produced normally.
  • Inference fails with failed to writeXram(). With its default settings, the torq-runtime 2.2.0 Python runner fails on this model's first inference. The example scripts already pass --torq_device_allocator=cpu; do the same if you load the vmfb yourself: VMFBInferenceRunner(vmfb, function="main_graph", runtime_flags=["--torq_device_allocator=cpu"]).

Summary​

In this blog, we covered the end-to-end process of teaching a SO-101 robot arm a pick-and-place task by demonstration, then run the trained policy entirely on-device on a Synaptics Astra Machina SL2610 (or Synaptics Coralboard™ SL) — no cloud, no GPU, no host PC needed at inference time. Using the Torq NPU, we were able to achieve ~6× faster performance than the CPU and sub-1° joint fidelity. This just scratches the surface of the capabilities of the Synaptics Astra SL2610. The techniques covered in this blog can apply to other models and use-cases. We can't wait to see all the robotic projects you will build!

Additional Notes​

  1. The same fp32 ONNX model running on the Astra CPU with ONNX Runtime measures ~1.75 s/inference, against ~320 ms on the Torq NPU. There is no bf16 CPU number to compare against: native bf16 arithmetic exists only on recent server-class CPUs (Intel Cooper Lake and newer Xeons, AMD Zen 4 and newer, Armv8.6+ cores) — neither the Astra's Arm cores, mainstream laptop/desktop chips, nor a Raspberry Pi have it. Half-precision (fp16) isn't a shortcut either: CPU runtimes generally compute fp16 by converting to fp32 and back, which we measured as slower than running plain fp32. So fp32 is the natural CPU baseline.
Coralboard is a trademark of Google LLC.