Shaders in Flutter

What is a shader?

A shader is a computer program embedded into a rendering pipeline. It is a brief program that runs on the GPU and can be effectively parallelized. When executed, the shader can define the color of each pixel in a rendered image or transform the coordinates of a vertex.

Why consider using them?

Ever wondered how to incorporate eye-catching visual effects in your app or game? Like:

• Particle systems (for instance, smoke);
• Ocean waves;
• Ambient occlusion;
• Global illumination.

Utilize the full potential of your GPU by employing shaders for these purposes.

Clockwise, starting from top left: particle systems, per-pixel lighting, global illumination, ambient occlusion.

Types of shaders

Let’s begin with basic definitions. A primitive refers to a group of vertices and the methods of linking them.

Shaders can be:

• Fragment/pixel shaders are programs that execute during the rasterization stage. They operate on each pixel to determine its final color and can be used for tasks such as applying textures or calculating lighting. There are two fundamental concepts:
  ‣ Fragment. A renderer creates fragments, which are the smallest unit of data produced by rasterization. A fragment does not always correspond to a physical pixel. In cases of multi-sampling, several fragments combine to produce the color of a single pixel. The area of a fragment is related to the area of a pixel.
  ‣ Pixel is a singular point on the physical screen.
• Vertex shader takes only a single primitive as an argument and returns only a single primitive as a result.
• Geometry shaders take one primitive as input and can output zero or more primitives.
• Compute shaders are a type of shader that does not deal with graphics and instead utilizes GPU computational power for tasks such as physics computations.

Let’s explain our rendering process in simple words. Imagine a scene made up of basic primitives.

1. The user’s program defines primitives, colors, textures, camera position, and orientation through method calls of graphics APIs such as Direct3D and Vulkan.
2. Vertex shaders are supplied with primitive vertex data to undergo additional transformations, such as animations.
3. The geometry shader receives primitives with transformed vertices and has the option to either generate new primitives or output zero.
4. After applying viewport and perspective transformations as well as textures, the geometry generated by the geometry shader is rasterized to obtain the colors of pixels.
5. Fragment shaders alter the colors of pixels to simulate lighting and other per-pixel effects.

Simple example

In its simplest form, a shader consists of the following parts:

1. Includes. The part describes dependencies of the shader. In this example, we include flutter/runtime_effect.glsl containing a convenient function to obtain coordinates of a fragment being processed.
2. Uniforms. These are the parameters sent from a client to a shader program. They are called uniforms because their values remain constant across multiple calls to the program.
3. Output fragment values. It’s the result of program execution. It can be a color or a depth value.
4. The code of the program itself. The entry function (called main) contains code in a C-like language called GLSL.

Here’s a simple example of a fragment shader:

// 1
#include <flutter/runtime_effect.glsl>

// 2
uniform vec2 uSize;
uniform vec4 uColor;

// 3
out vec4 fragColor;

// 4
void main() {
    vec2 currentPos = FlutterFragCoord().xy / uSize;
    float multiplier = currentPos.x * currentPos.y;
    fragColor = vec4(uColor.r * multiplier, uColor.g * multiplier, uColor.b * multiplier, uColor.a);
}

Let’s review it section by section:

1. Section 1 includes handy helpers as we mentioned above.
2. Section 2 describes input data represented as vectors. The size vector contains 2 values denoting the X and Y coordinates, while the color vector consists of 4 values representing the red, green, blue, and alpha components.
3. In section 3, fragColor is a reserved word. The shader uses it to write a calculated color of the current fragment.
4. In the main function (section 4), we calculate the position within the range of [0..1]. Then we multiply it by each component of the color provided in the uniform. We create a new vec4 out of these values and set it as the output value.

As a result we get a nice gradient from black to uColor. And this program is evaluated for every fragment of the rendered image.

How to load a shader in Flutter code

So, we devised a shader program. The following step is to use it within the application.

To access the program, it must be included in the project. The shader program file should be placed in the shaders directory located at the root of your project. To include the file in the project binary, reference it in the flutter/shaders section of the Pubspec.yaml file:

flutter:
  shaders:
    - shaders/shader.frag

Next we load the fragment program from assets:

class ShaderWidget extends StatefulWidget {
  const ShaderWidget({Key? key}) : super(key: key);

  @override
  State<ShaderWidget> createState() => _ShaderWidgetState();
}

class _ShaderWidgetState extends State<ShaderWidget> {
  FragmentShader? shader;

  @override
  void initState() {
    super.initState();

    _loadMyShader();
  }

  Future<void> _loadMyShader() async {
    var program = await FragmentProgram.fromAsset('shaders/shader.frag');
    setState(() {
      shader = program.fragmentShader();
    });
  }

  // ... 
}

That’s it! Once the shader is loaded, we can paint it on Canvas:

class MyPainter extends CustomPainter {
  MyPainter({
    required this.shader,
  });

  final FragmentShader shader;

  @override
  void paint(Canvas canvas, Size size) {
    final random = Random();

    shader
      ..setFloat(0, size.width) // width
      ..setFloat(1, size.height) // height
      ..setFloat(2, random.nextDouble()) // red
      ..setFloat(3, random.nextDouble()) // green
      ..setFloat(4, random.nextDouble()) // blue
      ..setFloat(5, 1); // alpha

    final paint = Paint()..shader = shader;
    canvas.drawRect(Rect.fromLTWH(0, 0, size.width, size.height), paint);
  }

  @override
  bool shouldRepaint(covariant CustomPainter oldDelegate) => true;
}

In the paint method, we pass uniform values to the shader. For instance, at the first position in the shader program, we have uniform vec2 uSize, which is a vector of two values. This implies that we should pass values to parameters with indices 0 and 1. These values are stored between frames and can be updated anytime.

Lastly, use this Painter.

class _ShaderWidgetState extends State<ShaderWidget> {
  FragmentShader? shader;

  // ...

  @override
  Widget build(BuildContext context) {
    final shader = this.shader;
    return shader == null // As the shader is loaded ascynchronously, we need to render something meanwhile
        ? const Center(
            child: CircularProgressIndicator(),
          )
        : CustomPaint(
            painter: MyPainter(shader: shader),
          );
  }
}

Et voilá! We’ve just successfully painted a gradient with a fragment shader!

Flutter limitations

When working with shaders in Flutter, it’s important to be aware of certain limitations and constraints that can impact your shader development.

• Only fragment shaders are supported.
• Supported GLSL versions are 100–460.
• Buffer Objects are not supported. Buffer Objects are essential for efficient data storage and transfer in OpenGL, and their absence in Flutter can impact certain advanced rendering techniques. Be prepared to work within this constraint when designing your shader-based applications.
• Only 2D samples (textures) are supported. If your application relies on 3D textures or other types, you may need to find alternative solutions or work within the confines of 2D textures.
• Unsigned types are not supported as well.

Understanding these limitations is crucial for effectively using shaders in Flutter. By being mindful of these constraints and working within them, you can harness the power of shaders to enhance the visual quality and interactivity of your Flutter applications.