Skip to main content

Running Real-time (stream) Whisper.cpp

This tutorial will guide you through the steps required to run whisper.cpp on an Synaptics Astra™ Machina™ dev kit using SL1680 to test real-time speech recognition using a microphone.

note

This tutorial is compatible with all SL16xx boards. While inference may vary, the steps remain the same across all processors.

Whisper.cpp is a C++ implementation of the Whisper speech-to-text model. View more details about Whisper.cpp at the Original GitHub Repo.

Whisper-Astra is a fork of whisper.cpp which has been specifically adapted and optimized for streaming on the Astra SL-Series processors.

info

The original whisper.cpp project continues to evolve beyond this point. Whisper-Astra ensures compatibility and stability as of June 26, 2024 from ggml + fix sync script commit of whisper.cpp.

Prerequisites​

Stream enables you to run Whisper in real-time and relies on the SDL library. For this, you need to do Cross-Compilation and two things: build your own image and add the SDL2 package to it. Then build your own toolchain and the SDL2 package will be included there.

warning

The Astra toolchain currently only supports x86 host machines due to closed-source third-party binaries. But we're working on it!

  1. Build your own image with the SDL2 package, which you will flash onto your Astra Board.

  2. Then, build your own toolchain and SDL2 package to create an environment where you can generate the stream binary.

Follow the steps from Build an Image until the step source meta-synaptics/setup/setup-environment command.

Once you run this command, it will create a directory build-sl1680, which will have local.conf:

source meta-synaptics/setup/setup-environment

Navigate to the location: build-sl1680/conf/local.conf, then add these two lines to the local.conf file so that you can build an image with SDL2 included:

IMAGE_INSTALL:append = "libsdl2"
TOOLCHAIN_TARGET_TASK:append = " libsdl2-staticdev"
bitbake astra-media

The resulting image can be found in build-sl1680/tmp/deploy/images/sl1680/SYNAIMG/. Copy this to a USB drive and flash this image onto your Astra Board using this tutorial.

Now, in the same terminal, run this command to build the SDK. This will create the toolchain in build-sl1680/tmp/deploy/sdk:

bitbake astra-media -c do_populate_sdk
info

The above command builds a toolchain, which could take hours depending on your development machine's specs.

Once built, open a terminal in Ubuntu and run the command:

bash poky-glibc-x86_64-astra-media-cortexa73-sl1680-toolchain-4.0.17.sh

Now run this command to activate the environment:

. /opt/poky/4.0.17/environment-setup-cortexa73-poky-linux
tip

To check if the environment is active, use the command in your Ubuntu terminal:

echo $CC

Step 1: Generate Binary for whisper.cpp​

You need to make sure the 'WHISPER_SDL2' option is enabled and the common-sdl library is correctly included in your build process.

Open a terminal in Ubuntu and install them using:

sudo apt install -y build-essential git cmake
sudo apt-get install libsdl2-dev

Clone the whisper.cpp repository for Astra from GitHub:

git clone https://github.com/astra-team-synaptics/whisper.cpp.git
cd whisper.cpp

Create a build directory inside whisper.cpp and navigate to it, then run the CMake command to build the project:

mkdir build && cd build
cmake -DWHISPER_SDL2=ON ..
cmake --build . --config Release

The binary for stream will be created in ~/build/bin/, and the shared library file libwhisper.so.1 in ~/build/src/. These two files will help you run the Whisper models, so you need to copy them to your Machina Board.

Step 2: Setting up Astra Machina Board​

Use ADB to access the Astra Machina Board from a host machine such as Ubuntu.

Follow these steps from the Access Machina Board tutorial to setup ADB.

Once in the ADB shell, create a new directory inside the home directory of the Machina Board:

mkdir whisper

Now, open a new Ubuntu terminal on your development system and use push to copy the binary files you generated from Step 1 to the Machina board:

adb push ~/whisper.cpp/build/bin/main /home/whisper
adb push ~/whisper.cpp/build/libwhisper.so.1 /home/whisper
adb push ~/ggml/build/src/libggml.so /home/whisper
adb push ~/ggml/build/src/libggml-base.so /home/whisper
adb push ~/ggml/build/src/libggml-cpu.so /home/whisper

Once you have copied the binary files to the Machina board, link the libwhisper.so.1 file. Follow these commands:

mv libwhisper.so.1 /usr/lib/
ln -s /usr/lib/libwhisper.so.1 /home/whisper/libwhisper.so.1
ls -l
note

You should see a linkage libwhisper.so.1 -> /usr/lib/libwhisper.so.1 after ls -l command like,

lrwxr-xr-x    1 root     root            24 Dec  5 22:03 libwhisper.so.1 -> /usr/bin/libwhisper.so.1

If running the binary gives an error:

./main: error while loading shared libraries: libwhisper.so.1: cannot open shared object file: No such file or directory

Remove libwhisper.so.1 using the command:

rm libwhisper.so.1

Copy it again, then push libwhisper.so.1 from the host development machine to the Machina Board's whisper directory and then give permissions.

Step 3: Download Whisper Models​

You need to download a Whisper model on your host development machine (Ubuntu in this case) from HuggingFace.

In this tutorial, you will use the Base quantized model ggml-small.en-q5_1.bin.

Once downloaded, you need to copy this model to your Astra Machina Board:

scp ~/Downloads/ggml-small.en-q5_1.bin root@10.3.10.132:/home/whisper/

Step 4: Running Whisper Stream on Machina Board​

You first need to set up the microphone for real-time inference.

To view the list of devices capable of recording (having a microphone), use the command and note down the device and card number, which will be used as -c 0 if the card number is 0:

arecord -l

With the model downloaded and the binary built, you can now run Whisper inside the whisper folder:

./stream -m ggml-small.en-q5_1 -c 0

The output from your Synaptics Astra board should look like:

..
[Start speaking]
1, 2, 3, 4, 5, 6, 7.
whisper_print_timings:     load time =   150.13 ms
whisper_print_timings:     fallbacks =   1 p /   0 h
whisper_print_timings:      mel time =    47.50 ms
whisper_print_timings:   sample time =    60.17 ms /     1 runs (   60.17 ms per run)
whisper_print_timings:   encode time = 17729.25 ms /     2 runs ( 8864.63 ms per run)
whisper_print_timings:   decode time =  4474.04 ms /    17 runs (  263.18 ms per run)
whisper_print_timings:   batchd time =   351.46 ms /    25 runs (   14.06 ms per run)
whisper_print_timings:   prompt time =     0.00 ms /     1 runs (    0.00 ms per run)
whisper_print_timings:    total time = 25711.63 ms
tip

Use Ctrl + C to stop Whisper and see the inference metrics.

Use the -sa parameter if you want to record the input going into the stream.

Congratulations!​

You have successfully installed and run real-time whisper.cpp on your Astra Machina Board. You can now explore further by trying different models or integrating Whisper into your projects. For more advanced usage and options, refer to the Whisper-Astra.cpp GitHub repo.