Overview

Beatmup is an extensible asynchronous image and signal processing framework.

Beatmup is for image processing, mainly.

• It enables low-level user-defined processing tasks execution
  • in parallel and using hardware acceleration for the maximum speed,
  • asynchronously, in an application-friendly fashion.
• It implements basic building blocks and predefined processing operations:
  • Image resampling and geometric operations
  • Basic color filtering
  • Flood fill and contour extraction
  • Realtime rendering

Beatmup runs quite everywhere.

• Used in linux, Windows and Android
• Written in C++11, it has Java and Python bindings
• Prebuilt Python packages are available for x64 desktop Windows and Ubuntu-based linux distributions
• Few to no dependencies: Beatmup is extremely easy to build for a different platform
  • git, CMake and a C++ compiler is all you need
  • Beatmup compiles with gcc, clang and msvc

Beatmup speeds up processing by using GPU regardless its vendor, price and release year.

• It runs on
  • desktop GPUs from mainstream vendors,
  • integrated desktop GPUs,
  • Android GPUs,
  • Raspberry Pi GPU (all models including Pi Zero W),
  • NVIDIA Jetson Nano,
  • ...
• It uses the GPU for
  • x2, neural network-based image upscaling technique,
  • running inference of small user-defined neural nets,
  • applying user-defined GLSL shaders to process images,
  • realtime camera image processing in Android,
  • scene rendering and predefined image processing tasks.

Highlights

x2: a neural net-based image upscaler

Beatmup is arguably the easiest way to get a fast hardware-accelerated neural network-based image superresolution running on any decent GPU. It does not provide a state-of-the-art PSNR score, but can run at a hundred of images per second on a desktop machine.

More details on the neural network design and the inference implementation: Inferring a super-resolution neural network on Raspberry Pi GPU

Update in 2.1 release: the fine-tuned version of the model (trained with multiple degradation kernels) achieves now 32.64 dB on DIV2K validation set compared to the initial score of 32.57 dB.

Available resources

Docs

• C++ documentation is available here. This is the most extensive documentation, it also contains additional explanations that do not relate to a specific programming language:
  • Programming model
  • NNets module overview
• Java package API documentation is here.
• Python documentation is here.

C++

There is a number of test apps showcasing the use of Beatmup with additional explanations in [apps](apps) folder. The code is in C++, but can be universally helpful. Few examples:

• apps/shaderer processes images with a custom GLSL fragment shader read from the standard input.
• apps/x2 is a test app for the x2 neural net.
• apps/classify is a dog image classifier, a variant of ResNeXt trained on a subset of ImageNet containing 120 classes of dogs and cats images. The inference is implemented with OpenGL shaders. Top-1 validation accuracy achieved on Raspberry Pi Zero W is 72.08%. Classifying a 385*385 image takes ~595 ms there.

Python

python/examples folder contains detailed examples of scripts using Beatmup in Python.

• cifar100.py explains how to train a neural net with TensorFlow/Keras, convert it into a Beatmup-compliant model and run its inference on CIFAR100 test set.
• x2_superresolution.py and x2_superresolution_video.py showcase applying x2 upsampling neural net to an image and a video.

Unit tests in Python are available in python/tests/test.py and python/tests/test_nnets.py.

Java (Android)

An Android test app is available in android/app folder. It consists of independent examples showing how to use Beatmup in Android.

• BasicCameraUse.java shows how to get the camera preview, apply a simple shader and display the result.
• UpsamplingConvnetOnCamera.java implements the application of x2 upsampling neural net to the camera preview.
• DogClassifier.java implements the dog classifier taking the camera preview image as input.

Quick start: compiling Beatmup

To get things running, make sure you have git and CMake installed, then pick one of the following copy-paste recipes.

Linux

Building X2 app upscaling an image using a neural net inferred with OpenGL:

git clone https://github.com/lnstadrum/beatmup.git
cd beatmup
git submodule update --init --recursive
mkdir -p build && cd build
cmake -DUSE_GLX=ON ..
make X2

• Try to use -DUSE_OPENGL=ON instead -DUSE_GLX=ON starting from a clean build folder if you run into trouble.

• On Raspberry Pi prior to series 4 use the EGL backend with following CMake command:

  cmake -DUSE_EGL=ON -DUSE_BRCM_LIBS=ON -DGLES_VERSION=20 ..

More details on Raspberry Pi setup here.

• On Raspberry Pi 4 rather use the following CMake command:

  cmake -DUSE_EGL_DRM=ON -DGLES_VERSION=20 ..

More details on Raspberry Pi 4 setup here.

Once the app is built, you can feed it with an image of your choice and get the upscaled result as follows:

./X2 <your-image>.bmp x2-result.bmp

Windows

To build the X2 app in Windows you may install Visual Studio and generate the VS solution as follows:

git clone https://github.com/lnstadrum/beatmup.git
cd beatmup
git submodule update --init --recursive
mkdir build && cd build
cmake -DUSE_OPENGL=ON ..

Then open the *.sln* solution file in Visual Studio and build the executable.

Android

There is Android project containing a library and a test app modules (lib and app). To build them install Android Studio and Android NDK bundle, clone the repository and open the project situated in android folder in Android studio.

You can build an apk in a docker container as well. Having docker installed, you can run the following script to build beatmupApp.apk installation package which you may then copy to your Android phone and open it there to install the test app:

git clone https://github.com/lnstadrum/beatmup.git
cd beatmup
git submodule update --init --recursive
docker build -f dockerfiles/android.Dockerfile -t beatmup-android .
docker run -v $(pwd):/opt/android-build beatmup-android

Quicker start with Python

Windows

Prebuilt python packages are available in 64-bit Windows for Python versions from 3.6 to 3.11:

python -m pip install --upgrade pip
python -m pip install beatmup

Ubuntu-based linux

Prebuilt packages for x64 desktop Ubuntu-based linux distributions are available for downloading on the releases page.

Compiling Python package

If there is no prebuilt package available for your platform/OS, you can easily build a Python wheel installable with pip on your own.

First, get the code and try to compile an app as explained above. Then in the repository root folder run

cd build
make beatmup
cd ../python
python3 setup.py bdist_wheel clean

The installation package is now available in python/dist folder. You can install it and make sure it works:

python3 -m pip install --no-index --find-links=dist beatmup
python3 -c "import beatmup; beatmup.say_hi()"

Licence and contributions

The project is licensed under the GNU GPL v3 licence.

The contributions are governed by the Contribution Licence Agreement (CLA). In short, under this Agreement you remain the copyright owner of your contribution, but you allow us to use and distribute your contribution under the terms of a possibly different licence.

If you want to contribute:

• make sure you own the copyright of your contribution,
• raise a pull request,
• send a signed CLA on beatm.nosp@m.up.i.nosp@m.magin.nosp@m.g@gm.nosp@m.ail.c.nosp@m.om.

Otherwise:

• raise an issue if you have a question or a feature request,
• contact us by email.