Image preprocessing operation. More...

#include <image_sampler.h>

Inheritance diagram for Beatmup::NNets::ImageSampler:

Public Member Functions
	ImageSampler (const std::string &name, const IntPoint &size, bool centerCrop=true, bool linearInterp=true)
	Creates an instance of image preprocessing operation. More...

void	setCenterCrop (bool enable)
	Enables / disables center crop. More...

bool	getCenterCrop () const

void	setLinearInterpolation (bool enable)
	Enables / disables linear interpolation. More...

bool	getLinearInterpolation () const

void	setRotation (int quarterTurns)
	Specifies a rotation to apply to the input image. More...

int	getRotation () const
	Returns rotation applied to the input image. More...

int	getInputCount () const
	Returns number of operation inputs. More...

int	getOutputCount () const
	Returns number of operation outputs. More...

bool	acceptsTextureInput (int index=0) const
	Returns `true` if the operation can take a GL::TextureHandler at a specific input. More...

bool	acceptsTextureOutput (int index=0) const
	Returns `true` if the operation can take a GL::TextureHandler at a specific output. More...

Size	getOutputSize (int outputIndex=0) const
	Returns full size of a specific operation output. More...

void	getOutput (GL::TextureHandler *&texture, int index=0)
	Returns a GL::TextureHandler bound to a specific operation output. More...

void	setInput (GL::TextureHandler &texture, int inputIndex=0)

void	setOutput (GL::TextureHandler &texture, int outputIndex=0)

std::map< std::string, std::string >	serialize () const
	Returns a serialized representation of th operation;. More...

void	disconnect ()
	Assigns empty inputs and outputs. More...

unsigned long	countTexelFetches () const
	Counts (approximate) number of texels fetches. More...

Public Member Functions inherited from Beatmup::NNets::AbstractOperation
virtual	~AbstractOperation ()

virtual bool	usesGpu () const
	Returns `true` if the operation is run on GPU. More...

virtual bool	acceptsStorageInput (int index=0) const
	Returns `true` if the operation can take a Storage::View at a specific input. More...

virtual bool	acceptsVectorInput (int index=0) const
	Returns `true` if the operation can take a GL::Vector at a specific input. More...

virtual bool	acceptsStorageOutput (int index=0) const
	Returns `true` if the operation can take a Storage::View at a specific output. More...

virtual bool	acceptsVectorOutput (int index=0) const
	Returns `true` if the operation can take a GL::Vector at a specific output. More...

virtual Storage::View	getOutput (int index=0)
	Returns a storage view bound to a specific operation output. More...

virtual void	getOutput (GL::Vector *&vector, int index=0)
	Returns a GL::Vector bound to a specific operation output. More...

virtual void	setInput (Storage::View &&storage, int index=0)

virtual void	setOutput (Storage::View &&storage, int index=0)

virtual void	setInput (GL::Vector &vector, int index=0)

virtual void	setOutput (GL::Vector &vector, int index=0)

virtual unsigned long	countMultiplyAdds () const
	Counts (approximate) number of multiply-adds used by this operation. More...

std::string	getName () const

Static Public Member Functions
static bool	initDeserializer ()
	Sets up deserialization of the operation. More...

Private Member Functions
void	prepare (GraphicPipeline &gpu, ChunkCollection &data, GL::ProgramBank &bank)
	Compiles GLSL shaders. More...

void	execute (TaskThread &thread, GraphicPipeline &gpu)
	Executes the operation. More...

int	getInputPadding (int index=0) const
	Retrieves minimum required size of zero padding for a given input. More...

void	getSampledChannels (int index, int &min, int &max) const
	Retrieves range of input features channels sampled at the same time for a specific input. More...

Private Attributes
const IntPoint	size

GL::TextureHandler *	input

GL::TextureHandler *	output

GL::RenderingProgram *	program

bool	linearInterpolation
	if `true`, the input image is linearly interpolated when possible More...

bool	centerCrop
	if `true`, a center crop is performed to sample the output image from the input; otherwise the input is stretched to match the output shape More...

int	rotation
	clockwise rotation to apply to the input image; 1 unit = 90 degrees More...

Additional Inherited Members
Protected Member Functions inherited from Beatmup::NNets::AbstractOperation
	AbstractOperation (const std::string &name)

virtual void	execute (TaskThread &thread)
	Executes the operation within a specific CPU thread. More...

Detailed Description

Image preprocessing operation.

Samples an image of a fixed size from an arbitrary size texture. Has three key missions. If enabled, performs a center crop keeping the output aspect ratio (otherwise the input is stretched to fit the output). If enabled, uses linear interpolation when possible to reduce aliasing (otherwise nearest neighbor sampling is used). Brings support of OES textures. This allows for example to read data directly from camera in Android.

Definition at line 32 of file image_sampler.h.

Constructor & Destructor Documentation

◆ ImageSampler()

ImageSampler::ImageSampler	(	const std::string &	name,
		const IntPoint &	size,
		bool	centerCrop = `true`,
		bool	linearInterp = `true`
	)

Creates an instance of image preprocessing operation.

Parameters

[in]	name	Layer name
[in]	size	Output image size in pixels
[in]	centerCrop	If `true`, the center crop is enabled
[in]	linearInterp	If `true`, the linear interpolation is enabled

Definition at line 28 of file image_sampler.cpp.

  :
     AbstractOperation(name),
     size(size),
     input(nullptr), output(nullptr), program(nullptr),
     linearInterpolation(linearInterp), centerCrop(centerCrop), rotation(0)
 {}

Member Function Documentation

◆ prepare()

void ImageSampler::prepare	(	GraphicPipeline &	gpu,
		ChunkCollection &	data,
		GL::ProgramBank &	bank
	)

privatevirtual

Compiles GLSL shaders.

Parameters

[in,out]	gpu	A graphic pipeline instance
[in,out]	data	Chunkfile containing operation data (e.g. weights and biases)
[in,out]	bank	A program bank with existing GLSL programs to be reused when possible. If a new program is built, it is added to the bank.

Implements Beatmup::NNets::AbstractOperation.

Definition at line 113 of file image_sampler.cpp.

                                                                                            {
     RuntimeError::check(input, "Input is not provided to a ImageSampler operation.");
  
     // delete the old program
     if (program)
         bank.release(gpu, program);
  
     // begin the shader code
     String code(GL::RenderingPrograms::DECLARE_TEXTURE_COORDINATES_IN_FRAG);
  
     // declare the sampler
     code.printf("uniform %s %s;", GL::FragmentShader::DIALECT_SAMPLER_DECL_TYPE, UNIFORM_INPUT);
  
     // add main code
     code.printf(R"glsl(
         void main() {
             gl_FragColor = %s(%s, %s);
         }
     )glsl", GL::FragmentShader::DIALECT_TEXTURE_SAMPLING_FUNC, UNIFORM_INPUT, GL::RenderingPrograms::TEXTURE_COORDINATES_ID);
  
     // init program
     bool enableExtTexExt = input->getTextureFormat() == GL::TextureHandler::TextureFormat::OES_Ext;
     program = bank(gpu, code, enableExtTexExt);
 }

◆ execute()

void ImageSampler::execute	(	TaskThread &	thread,
		GraphicPipeline &	gpu
	)

privatevirtual

Executes the operation.

The operation should be prepared.

Parameters

[in,out]	thread	Calling CPU thread descriptor
[in,out]	gpu	A graphic pipeline instance

Implements Beatmup::NNets::AbstractOperation.

Definition at line 139 of file image_sampler.cpp.

                                                                    {
     if (!program)
         throw NotReady(this);
     RuntimeError::check(input, "Input is not provided to a ImageSampler operation.");
     RuntimeError::check(output, "Output is not provided to a ImageSampler operation.");
  
     // enable program
     program->enable(gpu);
  
     // bind output
     gpu.bindOutput(*output);
  
     // bind input
     gpu.bind(*input, 0, linearInterpolation ? TextureParam::INTERP_LINEAR : TextureParam::INTERP_NEAREST);
  
     // setup texture coordinates
     const IntPoint inputSize(input->getWidth(), input->getHeight());
     Rectangle texCoords;
     if (centerCrop) {
         float hMargin = 0, vMargin = 0;
         if (input->getWidth() * output->getHeight() > input->getHeight() * output->getWidth()) {
             // input is cut vertically
             hMargin = 0.5f * (input->getWidth() - (float)input->getHeight() * output->getWidth() / output->getHeight());
         }
         else {
             // input is cut horizontally
             vMargin = 0.5f * (input->getHeight() - (float)input->getWidth() * output->getHeight() / output->getWidth());
         }
         texCoords = gpu.getTextureCoordinates(Rectangle(hMargin, vMargin, input->getWidth() - 1 - hMargin, input->getHeight() - 1 - vMargin), inputSize, size);
     }
     else {
         texCoords = gpu.getTextureCoordinates(Rectangle(0, 0, input->getWidth() - 1, input->getHeight() - 1), inputSize, size);
     }
  
     // apply rotation
     if (rotation % 4 != 0) {
         Point topLeft(texCoords.a), topRight(texCoords.b.x, texCoords.a.y), bottomLeft(texCoords.a.x, texCoords.b.y), bottomRight(texCoords.b), tmp;
         switch (rotation % 4) {
             case 1:
                 tmp = bottomLeft;
                 bottomLeft = bottomRight;
                 bottomRight = topRight;
                 topRight = topLeft;
                 topLeft = tmp;
                 break;
             case 2:
                 tmp = topLeft;
                 topLeft = bottomRight;
                 bottomRight = tmp;
                 tmp = topRight;
                 topRight = bottomLeft;
                 bottomLeft = tmp;
                 break;
             case 3:
                 tmp = topLeft;
                 topLeft = topRight;
                 topRight = bottomRight;
                 bottomRight = bottomLeft;
                 bottomLeft = tmp;
                 break;
         }
         gpu.setTextureCoordinates(topLeft, topRight, bottomLeft, bottomRight);
     }
     else {
         gpu.setTextureCoordinates(texCoords);
     }
  
     // blend
     program->blend();
 }

◆ getInputPadding()

int Beatmup::NNets::ImageSampler::getInputPadding ( int index = 0 ) const

inlineprivatevirtual

Retrieves minimum required size of zero padding for a given input.

Operations that sample a neighborhood of a pixel may need the input to be padded with zeros, if some of the neighboring samples fall out of the are containing data. In Beatmup the zero padding is handled by allocating a bigger input and putting zeros around the area that is actually filled with data.

Returns: number of zero columns and rows to be added to the input area.

Reimplemented from Beatmup::NNets::AbstractOperation.

Definition at line 45 of file image_sampler.h.

45 { return 0; }

◆ getSampledChannels()

void Beatmup::NNets::ImageSampler::getSampledChannels	(	int	index,
		int &	min,
		int &	max
	)		const

inlineprivatevirtual

Retrieves range of input features channels sampled at the same time for a specific input.

The operation would typically take the entire storage and sample it at once, if needed. If the number of textures in a storage exceeds the number of texture samplers that the GPU may use simultaneously, an exception occurs. This function provides the necessary information to limit the number of textures in the storage when allocating it. When the limit is reached, multiple channels are packed into a single texture in the storage.

Parameters

[in]	index	The input index. Expected to fall in the valid range, i.e. from zero to getInputCount() - 1 inclusive.
[out]	min	The minimum number of channels that can be sampled at once
[out]	max	The maximum number of channels that can be sampled at once

Implements Beatmup::NNets::AbstractOperation.

Definition at line 46 of file image_sampler.h.

46 { min = max = 0; }

std::min

CustomPoint< numeric > min(const CustomPoint< numeric > &a, const CustomPoint< numeric > &b)

Definition: geometry.h:724

std::max

CustomPoint< numeric > max(const CustomPoint< numeric > &a, const CustomPoint< numeric > &b)

Definition: geometry.h:728

◆ setCenterCrop()

void Beatmup::NNets::ImageSampler::setCenterCrop ( bool enable )

inline

Enables / disables center crop.

Definition at line 66 of file image_sampler.h.

66 { this->centerCrop = enable; }

◆ getCenterCrop()

bool Beatmup::NNets::ImageSampler::getCenterCrop ( ) const

inline

Definition at line 67 of file image_sampler.h.

67 { return this->centerCrop; }

◆ setLinearInterpolation()

void Beatmup::NNets::ImageSampler::setLinearInterpolation ( bool enable )

inline

Enables / disables linear interpolation.

Definition at line 72 of file image_sampler.h.

72 { this->linearInterpolation = enable; }

◆ getLinearInterpolation()

bool Beatmup::NNets::ImageSampler::getLinearInterpolation ( ) const

inline

Definition at line 73 of file image_sampler.h.

73 { return this->linearInterpolation; }

◆ setRotation()

void Beatmup::NNets::ImageSampler::setRotation ( int quarterTurns )

inline

Specifies a rotation to apply to the input image.

Parameters

quarterTurns Number of times a clockwise rotation by 90 degree is applied to the input image.

Definition at line 79 of file image_sampler.h.

79 { this->rotation = quarterTurns; }

◆ getRotation()

int Beatmup::NNets::ImageSampler::getRotation ( ) const

inline

Returns rotation applied to the input image.

Returns: number of times a clockwise rotation by 90 degree is applied to the input image.

Definition at line 85 of file image_sampler.h.

85 { return this->rotation; }

◆ getInputCount()

int Beatmup::NNets::ImageSampler::getInputCount ( ) const

inlinevirtual

Returns number of operation inputs.

Inputs are then indexed from zero to the returned value minus one inclusive.

Reimplemented from Beatmup::NNets::AbstractOperation.

Definition at line 87 of file image_sampler.h.

87 { return 1; }

◆ getOutputCount()

int Beatmup::NNets::ImageSampler::getOutputCount ( ) const

inlinevirtual

Returns number of operation outputs.

Outputs are then indexed from zero to the returned value minus one inclusive.

Reimplemented from Beatmup::NNets::AbstractOperation.

Definition at line 88 of file image_sampler.h.

88 { return 1; }

◆ acceptsTextureInput()

bool Beatmup::NNets::ImageSampler::acceptsTextureInput ( int index = 0 ) const

inlinevirtual

Returns true if the operation can take a GL::TextureHandler at a specific input.

Neural network operations may accept different kinds of data containers on inputs and outputs, namely Storage::View, GL::Vector and textures. This function is used to check whether a given operation accepts a texture on input.

Parameters

[in] index The input index. Expected to fall in the valid range, i.e. from zero to getInputCount() - 1 inclusive.

Reimplemented from Beatmup::NNets::AbstractOperation.

Definition at line 90 of file image_sampler.h.

90 { return index == 0; }

index

jlong jint index

Definition: wrapper_core.cpp:434

◆ acceptsTextureOutput()

bool Beatmup::NNets::ImageSampler::acceptsTextureOutput ( int index = 0 ) const

inlinevirtual

Returns true if the operation can take a GL::TextureHandler at a specific output.

Neural network operations may accept different kinds of data containers on outputs and outputs, namely Storage::View, GL::Vector and textures. This function is used to check whether a given operation accepts a texture on output.

Parameters

[in] index The output index. Expected to fall in the valid range, i.e. from zero to getOutputCount() - 1 inclusive.

Reimplemented from Beatmup::NNets::AbstractOperation.

Definition at line 91 of file image_sampler.h.

91 { return index == 0; }

◆ getOutputSize()

Size Beatmup::NNets::ImageSampler::getOutputSize ( int outputIndex = 0 ) const

inlinevirtual

Returns full size of a specific operation output.

Reimplemented from Beatmup::NNets::AbstractOperation.

Definition at line 93 of file image_sampler.h.

93 { return Size(size.x, size.y, 3); }

◆ getOutput()

void ImageSampler::getOutput	(	GL::TextureHandler *&	vector,
		int	index = `0`
	)

virtual

Returns a GL::TextureHandler bound to a specific operation output.

Parameters

[out]	vector	Pointer to the GL::TextureHandler. If no texture is bound, becomes null.
[in]	index	The output index. Expected to fall in the valid range, i.e. from zero to getOutputCount() - 1 inclusive.

Reimplemented from Beatmup::NNets::AbstractOperation.

Definition at line 41 of file image_sampler.cpp.

                                                                   {
     InvalidArgument::check(index == 0, "Invalid output index of ImageSampler operation: " + std::to_string(index));
     texture = this->output;
 }

◆ setInput()

void ImageSampler::setInput	(	GL::TextureHandler &	texture,
		int	inputIndex = `0`
	)

virtual

Reimplemented from Beatmup::NNets::AbstractOperation.

Definition at line 47 of file image_sampler.cpp.

                                                                      {
     InvalidArgument::check(inputIndex == 0, "Invalid input index of ImageSampler operation: " + std::to_string(inputIndex));
     this->input = &texture;
 }

◆ setOutput()

void ImageSampler::setOutput	(	GL::TextureHandler &	texture,
		int	outputIndex = `0`
	)

virtual

Reimplemented from Beatmup::NNets::AbstractOperation.

Definition at line 53 of file image_sampler.cpp.

                                                                        {
     InvalidArgument::check(outputIndex == 0, "Invalid output index of ImageSampler operation: " + std::to_string(outputIndex));
     InvalidArgument::check(texture.getWidth() == size.x && texture.getHeight() == size.y,
         "ImageSampler output size mismatch");
     this->output = &texture;
 }

◆ serialize()

std::map< std::string, std::string > ImageSampler::serialize ( ) const

virtual

Returns a serialized representation of th operation;.

Implements Beatmup::NNets::AbstractOperation.

Definition at line 90 of file image_sampler.cpp.

                                                            {
     return {
         { "_name",          getName() },
         { "_type",          "image_sampler" },
         { "output_width",   std::to_string(size.x) },
         { "output_height",  std::to_string(size.y) },
         { "linear_interp",  linearInterpolation ? "true" : "false" },
         { "center_crop",    centerCrop ? "true" : "false" }
     };
 }

◆ disconnect()

void ImageSampler::disconnect ( )

virtual

Assigns empty inputs and outputs.

Implements Beatmup::NNets::AbstractOperation.

Definition at line 102 of file image_sampler.cpp.

                               {
     input = output = nullptr;
 }

◆ countTexelFetches()

unsigned long ImageSampler::countTexelFetches ( ) const

virtual

Counts (approximate) number of texels fetches.

Reimplemented from Beatmup::NNets::AbstractOperation.

Definition at line 108 of file image_sampler.cpp.

                                                     {
     return size.x * size.y;
 }

◆ initDeserializer()

static bool Beatmup::NNets::ImageSampler::initDeserializer ( )

static

Sets up deserialization of the operation.

Member Data Documentation

◆ size

const IntPoint Beatmup::NNets::ImageSampler::size

private

Definition at line 34 of file image_sampler.h.

◆ input

GL::TextureHandler* Beatmup::NNets::ImageSampler::input

private

Definition at line 35 of file image_sampler.h.

◆ output

GL::TextureHandler * Beatmup::NNets::ImageSampler::output

private

Definition at line 35 of file image_sampler.h.

◆ program

GL::RenderingProgram* Beatmup::NNets::ImageSampler::program

private

Definition at line 36 of file image_sampler.h.

◆ linearInterpolation

bool Beatmup::NNets::ImageSampler::linearInterpolation

private

if true, the input image is linearly interpolated when possible

Definition at line 37 of file image_sampler.h.

◆ centerCrop

bool Beatmup::NNets::ImageSampler::centerCrop

private

if true, a center crop is performed to sample the output image from the input; otherwise the input is stretched to match the output shape

Definition at line 38 of file image_sampler.h.

◆ rotation

int Beatmup::NNets::ImageSampler::rotation

private

clockwise rotation to apply to the input image; 1 unit = 90 degrees

Definition at line 41 of file image_sampler.h.

The documentation for this class was generated from the following files:

core/nnets/image_sampler.h
core/nnets/image_sampler.cpp

Public Member Functions

Static Public Member Functions

Private Member Functions

Private Attributes

Additional Inherited Members

Detailed Description

Constructor & Destructor Documentation

◆ ImageSampler()

Member Function Documentation

◆ prepare()

◆ execute()

◆ getInputPadding()

◆ getSampledChannels()

◆ setCenterCrop()

◆ getCenterCrop()

◆ setLinearInterpolation()

◆ getLinearInterpolation()

◆ setRotation()

◆ getRotation()

◆ getInputCount()

◆ getOutputCount()

◆ acceptsTextureInput()

◆ acceptsTextureOutput()

◆ getOutputSize()

◆ getOutput()

◆ setInput()

◆ setOutput()

◆ serialize()

◆ disconnect()

◆ countTexelFetches()

◆ initDeserializer()

Member Data Documentation

◆ size

◆ input

◆ output

◆ program

◆ linearInterpolation

◆ centerCrop

◆ rotation