If the title caught your eye, you’re probably wondering what this is all about. I’ve been fascinated by the latest trend from Apple and Samsung — integrating shaders directly into apps to create a fluid, dynamic user experience (See Video 1). These shaders often sync with real-time audio and, when combined with AI, give the interface a natural, organic feel. The result? AI feels like a seamless part of the entire system rather than an add-on. I wanted to bring that same experience to my own project — so here’s how I made it happen.

I set out to build a real-time, sound-reactive shader in Flutter, aiming to have different frequency ranges in a voice independently alter specific parts of the shader. For example, the bass frequencies would influence one part of the shader, while the mid and high frequencies would affect other parts. The final result looks like this:

Let’s examine the code that enables frequency extraction from a real-time audio signal:

class VoiceApi {
  void getFrequencies(void Function(List<({
    FrequencySpectrum spectrum, 
    double value,
    })> data,
  ) onData,
  ) {
    final _flutterAudioCapture = FlutterAudioCapture();    
    _flutterAudioCapture.start(
      (data) {
        final buffer = data;
        final fft = FFT(buffer.length);

        final freq = fft.realFft(buffer);
        final freqList = freq.discardConjugates().magnitudes().toList();
        final frequencies = [
          FrequencySpectrum(0, 20),
          FrequencySpectrum(20, 25),
          FrequencySpectrum(25, 31),
          FrequencySpectrum(31, 40),
          FrequencySpectrum(40, 50),
          FrequencySpectrum(50, 63),
          FrequencySpectrum(63, 80),
          FrequencySpectrum(80, 100),
          FrequencySpectrum(100, 125),
          FrequencySpectrum(125, 160),
          FrequencySpectrum(160, 200),
          FrequencySpectrum(200, 250),
          FrequencySpectrum(250, 315),
          FrequencySpectrum(315, 400),
          FrequencySpectrum(400, 500),
          FrequencySpectrum(500, 630),
          FrequencySpectrum(630, 800),
          FrequencySpectrum(800, 1000),
          FrequencySpectrum(1000, 1250),
          FrequencySpectrum(1250, 1600),
          FrequencySpectrum(1600, 2000),
          FrequencySpectrum(2000, 2500),
          FrequencySpectrum(2500, 3150),
          FrequencySpectrum(3150, 4000),
          FrequencySpectrum(4000, 5000),
          FrequencySpectrum(5000, 6300),
          FrequencySpectrum(6300, 8000),
          FrequencySpectrum(8000, 10000),
          FrequencySpectrum(10000, 12500),
          FrequencySpectrum(12500, 16000),
          FrequencySpectrum(16000, 22000),
        ];
        List<({FrequencySpectrum spectrum, double value})> frequencyValues =
            frequencies.map((e) {
          final min = fft.indexOfFrequency(e.min.toDouble(), 44000);
          final max = fft.indexOfFrequency(e.max.toDouble(), 44000);

          return (
            spectrum: e,
            value: freqList
                .sublist(min.floor(), max.ceil())
                .reduce((a, b) => a + b),
          );
        }).toList();
        onData(frequencyValues);
      },
      (e) {
        debugPrint(e);
      },
      sampleRate: 44000,
      bufferSize: 256,
    );
  }
}

// A frequency spectrum
class FrequencySpectrum {
  FrequencySpectrum(this.min, this.max);

  final int min;
  final int max;
}

This might not look like a lot of code, but it brings a high level of complexity with it. To really understand how this works, we need to dig (somewhat) deep into audio processing. This is what I will do in this article.

Implementation Details and Methodology

Before delving into the code, it’s essential to first understand what an audio frequency is, in order to effectively extract it from audio signals.

Audio Frequencies

What exactly are audio frequencies? A frequency is a periodic vibration that is audible to the human ear¹. The unit of a frequency is called Hertz (Hz) and refers to how often the wave of the audio signal repeats complete cycles within a second, i.e. 5 repetitions equal 5 Hz (See Figure 2).

We, as humans, link a small Hz number with a low tone, e.g. bass or the lowest “a” key on the piano, while a larger Hz number is linked to a high tone, such as whistling or the highest “a” key on the piano. Generally, the human ear can recognize frequencies between 20–20000Hz (20kHz). Therefore, every audible sound that a human can recognize, is a combination of the different frequencies with different levels in this frequency range (See Figure 4).

Now that we understand what frequencies are, how are these extracted from an audio signal?

Microphone Signal

Before we perform any extraction, the captured audio signal from the microphone is required first. There are packages on pub.dev that take care of that. In this case, the flutter_audio_capture is used, but feel free to change to any other package.

To begin using this package, we must first initialize the flutter_audio_capture before listening to the audio signal.

final  _flutterAudioCapture = FlutterAudioCapture().initialize();

Once initialized, we can begin listening to the audio signal directly from the microphone:

_flutterAudioCapture.start(
  (data) { ... },
  (e) { ... },
  sampleRate: <sample_rate>,
  bufferSize: <buffer_size>,
);

Before applying any logic on the microphone data, let’s first examine the following two parameters sampleRate and bufferSize :

Sample Rate

The sample rate refers to the number of sample that are captured per second (See Figure 4). To capture high frequencies from an audio signal, a higher sampling rate is required. For instance, a typical phone call has a sampling rate of only 8000 Hz (8 kHz), which limits the capture of higher frequencies, resulting in a muffled sound quality and a loss in information.

Therefore, it would be logical to use a sampling rate of approximately 20 kHz; however, the actual sampling rate must be at least 40 kHz. This requirement is based on the Nyquist–Shannon sampling theorem.

Nyquist–Shannon sampling theorem

Audio consists of pressure waves, which, when visualized, exhibit an oscillating pattern, i.e. they go up and down. To accurately capture the full range of an audio wave, it is essential to record the points where the wave reaches its lowest and highest levels, ensuring that the wave has completed a full cycle. Otherwise, this can lead to incorrect signal capture, as illustrated in Figure 5 A and C. This phenomenon is called aliasing.

The wagon wheel problem describes a similar problem, where the wheel of a car spinning 24 times a second appears stationary when filmed with a 24 frames per second (FPS) camera. With more than 24 rotations, the wheel can even look like it is going reverse.

That is the reason why, according to the Nyquist–Shannon sampling theorem, the Hz of the sample rate must be twice the Hz of the actual frequency that should be measured/reconstructed. This requirement ensures that the original waveform can be accurately and precisely reconstructed.

So for 20000 Hz, the upper frequency bound the human ear can hear, this equals to 40000 Hz. So this means that audio should be recorded at a sampling rate of 40000 Hz to capture the full spectrum from 20 to 20000 Hz.

However, the most common sampling rates are 44.1 kHz and 48 kHz and not 40kHz. The reason for that is, that we do not want to record any audio above the 40kHz, but it is not directly possible to cut off any signals above a certain frequency. Instead the so called anti-aliasing filters, which are used for cutting off frequencies above or below a certain threshold, feature a transition band, i.e. it takes from frequency 20000Hz to frequency 22000Hz to have all frequencies above the threshold of 20000Hz removed (See Figure 7). Within this range (e.g. 20000Hz–22000Hz), the frequencies gradually decrease but are still present until they are completely absent.

This is the reason, why the sample rate should not be 40 kHz, but should have a buffer to allow for the transition band. Therefore, going back to our code example, we can choose a sample rate of 48000.

Buffer Size

The buffer size defines the number of samples that should be buffered before processing. The higher the number of data points we buffer, the more delay will be in the signal. The lower the number is, the less time we have to apply any calculations to prevent any stuttering, e.g. the buffer size has 53ms of data (buffer size/sample rate) and our calculation takes 59ms, there will be a stuttering/no sound of 6ms when playing back the data in real-time.

We will go with a buffer size of 256, i.e. 256 data points are buffered, but feel free to change to a lower or higher number if necessary.

With both the sample rate and buffer size in place, we can finally take a look at the code for extracting the frequencies from our audio signal:

_flutterAudioCapture.start(
(data) {
  final buffer = data;  
  <...>
},

So what is the data that we get from the package?

PCM Data

The data the package returns is called PCM data. Basically, this is the raw audio data representing sound wave amplitudes over time. In more detail, PCM stands for Pulse Code Modulation and is the method for transforming analog signals to digital ones. Only with the raw/uncompressed audio data, it is possible to extract the frequencies as desired.

How can the frequencies be extracted? Are they directly accessible in the data? No, the data must first be transformed. This transformation is performed in the following code lines using the fftea package:

final fft = FFT(buffer.length);
final freq = fft.realFft(buffer);
final freqList = freq.discardConjugates().magnitudes().toList();

What is happening here? To fully understand this, we must first grasp the underlying concept.

Time vs Frequency Domain

Why can we not extract the frequency information directly? Because the PCM Data is in the time domain, i.e. the data represents audio over time and not per frequency. But luckily, having the data in time domain also means that we have the data in the frequency domain but just can not access it directly (See Figure 1). Instead, it’s required to apply an Fast-Fourier-Transform (FFT). The FFT is used to transform the data from time domain to frequency domain.

Fast-Fourier-Transform: Transforming Time to Frequency Domain

How does the Fast Fourier Transform (FFT) work? In the ‘Audio Frequencies’ section, it was explained that every sound is composed of various frequencies at different levels. Therefore, it should be possible to reconstruct any sound by combining and adding these different frequencies. That’s correct, and the fascinating part is, that when reconstructing a sound using different frequencies, the level of each frequency needed to reproduce the waveform is determined (See Figure 4). This is exactly what is required: The level of each frequency aka. the frequency domain.

That is what the FFT is all about: It reengineers the audio signal to extract the frequency domain by determining the level of each frequency required. This is the process that occurs in the FFT formula:

Basically, this is what is happening behind the scenes when executing the following code:

final freq = fft.realFft(buffer);

Symmetrical Result

The output of the FFT consists of pairs of complex numbers, which can be used to calculate values such as the magnitude. We then apply two functions to this output. The first is fftOutput.discardConjugates(), which removes half of the values from the FFT. This is possible because the FFT produces a symmetrical, mirrored result for real signals (See Figure 9), with one half representing negative frequencies and the other representing positive frequencies.

Both the positive and negative frequency components are required to fully reconstruct a signal. However, since the use case is not about full reconstruction but about extracting frequencies, the negative frequencies are not needed and can be discarded, effectively reducing the output size by half.

Magnitude

The second function we call is fftOutput.magnitudes() .

The FFT outputs us pairs of complex numbers. These are not in a useable format. What we want, is the maximum absolute number for a frequency, known as the magnitude. To calculate the magnitude, the Pythagorean theorem (a² + b² = c²) is used, where a and b equal the complex numbers from the FFT output and c equals the magnitude.

This calculation does not come out of nowhere. Instead the values for a and b represent sine and cosine values within a circle and together with the third value, i.e. the magnitude, form a triangle within a circle (See Figure 8).

Therefore, we can calculate the magnitude by calculating the missing side of a triangle using the Pythagorean theorem.

Frequency Spectrums

Finally, we will group various frequencies into broader frequency spectrums to reduce the number of data points for visualization. This step is optional.

For this we will use a FrequencySpectrum model, which includes a minimum and maximum frequency value:

class FrequencySpectrum {
  FrequencySpectrum(this.min, this.max);

  final int min;
  final int max;
}

The frequencies will be grouped by octaves. Within an octave the frequency doubles, e.g. 40–80 Hz is one octave and the following octave would go from 80–160 Hz. Interestingly, humans perceive these two areas as qualitatively identical, with the latter merely being at a higher pitch, still the distance between the start and end frequency being twice as far. All octaves, starting with 0 — 20 Hz and ending with 16kHz-20kHz, will be put into a list:

final frequencies = [
  FrequencySpectrum(0, 20),
  FrequencySpectrum(20, 25),
  FrequencySpectrum(25, 31),
  FrequencySpectrum(31, 40),
  FrequencySpectrum(40, 50),
  FrequencySpectrum(50, 63),
  FrequencySpectrum(63, 80),
  FrequencySpectrum(80, 100),
  FrequencySpectrum(100, 125),
  FrequencySpectrum(125, 160),
  FrequencySpectrum(160, 200),
  FrequencySpectrum(200, 250),
  FrequencySpectrum(250, 315),
  FrequencySpectrum(315, 400),
  FrequencySpectrum(400, 500),
  FrequencySpectrum(500, 630),
  FrequencySpectrum(630, 800),
  FrequencySpectrum(800, 1000),
  FrequencySpectrum(1000, 1250),
  FrequencySpectrum(1250, 1600),
  FrequencySpectrum(1600, 2000),
  FrequencySpectrum(2000, 2500),
  FrequencySpectrum(2500, 3150),
  FrequencySpectrum(3150, 4000),
  FrequencySpectrum(4000, 5000),
  FrequencySpectrum(5000, 6300),
  FrequencySpectrum(6300, 8000),
  FrequencySpectrum(8000, 10000),
  FrequencySpectrum(10000, 12500),
  FrequencySpectrum(12500, 16000),
  FrequencySpectrum(16000, 20000),
];
       

For each frequency spectrum, the combined magnitude is calculated afterwards:

List<({FrequencySpectrum spectrum, double value})> frequencyValues =
  frequencies.map((e) {
    final min = fft.indexOfFrequency(e.min.toDouble(), 44000);
    final max = fft.indexOfFrequency(e.max.toDouble(), 44000);
    
    return (
      spectrum: e,
      value: freqList
          .sublist(min.floor(), max.ceil())
          .reduce((a, b) => a + b),
    );
}

Finally, these frequencyValues are passed to theonData function, which can then be used for visualizing the data in the UI.

We can now extract frequencies from a real-time audio signal, allowing us to visualize, animate, and implement various creative applications, e.g. integrating different frequencies into an apple intelligence clone like I did here:

Summary

In this article, we explored how frequency data can be extracted from real-time audio signals using Dart/Flutter to create a responsive visualization. To achieve this, raw PCM data is captured first, then a Fast Fourier Transform (FFT) is applied to convert the data from time domain to frequency domain. By analyzing this transformed data, we can group various frequencies to drive different parts of a shader, allowing specific audio frequency spectrums to influence separate parts of an animation.

We also covered essential concepts such as sample rate and buffer size, explaining how they affect real-time audio capture and processing. The sample rate should be high enough to prevent aliasing as suggested by the Nyquist–Shannon sampling theorem, which requires a rate at least double the highest frequency in the signal. Buffer size, meanwhile, controls the latency and processing delay, balancing responsiveness with smooth performance.

By following these steps, you can successfully integrate audio-responsive visualizations in Flutter, aligning with the latest trends set by companies like Apple and Google.

The full source code, including the visualization with the shader and Equalizer, is available under:

GitHub - vgtle/shader_studio at intelligence

If you enjoyed this guide and would like to see a Part 2, let me know! Clap for this article, leave a comment, or share it with others who might find it useful. Your feedback will help me create even more in-depth guides on different Flutter topics!

You can connect with me on X (formerly Twitter) or LinkedIn if you’re interested in Flutter and want to explore more in-depth topics.

Sources

[1] Pilhofer, Michael (2007). Music Theory for Dummies. For Dummies. p. 97. ISBN 9780470167946.

[2] https://vru.vibrationresearch.com/lesson/introduction-sine/ (15.10.24)

[3] Chowdhury, Mehdi & Cheung, Ray C.C.. (2019). Reconfigurable Architecture for Multi-lead ECG Signal Compression with High-frequency Noise Reduction. Scientific Reports. 9. 10.1038/s41598–019–53460–3.

[4] https://www.youtube.com/watch?v=rUtz-471LkQ (06.11.24)

[5] Kapić, Aladin & Sarić, Rijad & Lubura, Slobodan & Jokic, Dejan. (2021). FPGA-based Implementation of IIR Filter for Real-Time Noise Reduction in Signal. Journal of Engineering and Natural Sciences. 3. 10.14706/JONSAE2021316.

[6] https://www.youtube.com/watch?v=VNftf5qLpiA (06.11.2024)

[7] https://www.ni.com/de/shop/data-acquisition/measurement-fundamentals/analog-fundamentals/acquiring-an-analog-signal--bandwidth--nyquist-sampling-theorem-.html (06.11.2024)

[8] Mastriani, Mario. (2018). Quantum-Classical Algorithm for an Instantaneous Spectral Analysis of Signals: A Complement to Fourier Theory. Journal of Quantum Information Science. 08. 52–77. 10.4236/jqis.2018.82005.

[9] https://www.jezzamon.com/fourier/ (06.11.2024)

[10] https://www.hollyland.com/blog/tips/what-is-sample-rate-in-audio (12.11.2024)

Master Real-Time Frequency Extraction in Flutter to Elevate Your App Experience was originally published in neusta mobile solutions on Medium, where people are continuing the conversation by highlighting and responding to this story.

The short answer to this is: Yes, Flutter is absolutely capable of doing this:

So how does it work?

Well the answer to this is not that easy. For the purpose of this post, this is split up into 4 Chapters:

• The shader
• Integration of the shader into Flutter
• Animation Behavior
• Ticker

So let’s get started!

The Shader

Introduction

Shaders are an incredible tool to manipulate the pixels drawn on the screen. The benefit? Each pixel is manipulated independently, which allows parallel processing on the GPU for great performance. Sadly, this can’t be done within Flutter for now, but requires the shader to be written in the OpenGL Shading Language. BUT, it is not that complicated. Trust me on that.

The RGB Distortion Effect

As I said, the language itself is not difficult and the code is not a lot (See below), but it is all about the details and a little math. So let’s break it down!

#include <flutter/runtime_effect.glsl>

uniform vec2 u_size;
uniform vec2 u_location;
uniform vec2 u_velocity;
uniform sampler2D u_texture;

out vec4 frag_color;

void main() {
  vec2 l = u_location;
  vec2 v = u_velocity;
  vec2 p = FlutterFragCoord().xy;


  vec2 m = -v * pow(clamp(1.0 - length(l - p) / 190, 0.0, 1.0), 2) * 1.5 ;
  
  vec3 c = vec3(0.0);

  for (int i = 0; i < 10; i++) {
    float s = 0.175 + 0.005 * i;
      c += vec3(
          texture(u_texture, (p + s * m) / u_size).r, 
         texture(u_texture, (p + (s + 0.035) * m) / u_size).g, 
          texture(u_texture, (p + (s + 0.06) * m) / u_size).b
      );
    }
    
    frag_color = vec4(c / 10.0, To apply the RGB Split, we first need to receive the pixel according pixel of the rendered image from Flutter:);
}

We start very simple. Since we are adjusting single independent pixels, the script needs to output a pixel, more precisely a pixel in the RGBA format, i.e it consists of the red, green, blue and alpha channel values. We can declare that the following way:

out vec4 frag_color;

Not that hard, right? So, let us continue! Next, input variables are required, allowing control of the behavior of the shader from outside. These are called uniforms and work as the bridge between Flutter and the shader for passing information. They will allow us to animate the shader later on. Again these uniforms are not hard to declare:

uniform vec2 u_size;
uniform vec2 u_location;
uniform vec2 u_velocity;
uniform sampler2D u_texture;

• u_size: The screen size (width x height) that we need to keep the correct aspect ratio.
• u_location: The position on the screen that is touched by the user, used to trigger the effect.
• u_velocity: The speed of the pan gesture of the user on the device, used to control the strength of the effect.
• u_texture: This is the rendered image from Flutter, allowing for manipulation of the pixels rendered by Flutter.

Now, there is shader input, i.e. multiple uniform variables, and a shader output, i.e. an out variable. So, only the logic for transforming the image is missing.

This is done in the main() function:

void main() {
  vec2 l = u_location;
  vec2 v = u_velocity;
  vec2 p = FlutterFragCoord().xy;

  vec2 m = -v * pow(clamp(1.0 - length(l - p) / 190, 0.0, 1.0), 2) * 1.5 ;
  
  vec3 c = vec3(0.0);

  for (int i = 0; i < 10; i++) {
    float s = 0.175 + 0.005 * i;
      c += vec3(
          texture(u_texture, (p + s * m) / u_size).r, 
         texture(u_texture, (p + (s + 0.035) * m) / u_size).g, 
          texture(u_texture, (p + (s + 0.06) * m) / u_size).b
      );
    }
    
    frag_color = vec4(c / 10.0, 1.0);
}

Still, these are only 18 lines of code: A lot is happening. So, let’s go through this step by step:

1. Assigning Variables

vec2 l = u_location;
vec2 v = u_velocity;
vec2 p = FlutterFragCoord().xy;

For shorter code, u_location and u_velocity are reassigned. In addition, FlutterFragCoord().xy, i.e. the actual current pixel that should be manipulated, is also reassigned.

2. Calculating the amplitude of the effect

The next step is to calculate how strongly the effect should be applied to the individual pixels. This is based on the following conceptual idea: the further away a pixel (p) is from the touched pixel on the screen (l), the smaller the effect should be.

The actual implementation also takes into account the velocity (v), i.e. the dragging speed and the direction of the user’s gesture, and applies some modifiers to make the function non-linear:

This results in the following line of code:

vec2 m = -v * pow(clamp(1.0 - length(l - p) / 190, 0.0, 1.0), 2) * 1.5 ;

Playing around with different parameters for this function will lead to interestingly different results. So feel free to play around with it.

3. The RGB Split Effect

Finally, something more interesting: The RGB Split Effect!

To calculate the RGB split, we must first obtain the corresponding pixel of the rendered image from Flutter, i.e. the pixel that belongs to the position in the variable p. This can be done with the following code:

texture(u_texture, p)

The idea behind the RGB split is that each color channel, i.e. the red, green and blue channels, is translated in different directions:

vec3 pixel_red = texture(u_texture, p - 0.02);
vec3 pixel_green = texture(u_texture, p + 0.035);
vec3 pixel_blue = texture(u_texture, p + 0.06);
vec4 new_pixel = vec4(pixel_red, pixel_green, pixel_blue, 1)

Playing around with these translation numbers will again turn into different results:

Obviously, this is not the final result we expect to get. There is no real distortion effect visible. So let’s add this now.

4. The Distortion Effect

The distortion effect contains of two different effects:

• Translation of Pixels
• Blur Effect

The translation of pixels is done by applying the same translation value to each pixel:

float distortion = 10;
vec3 pixel_red = texture(u_texture, p - 0.02 + distortion);

This results in the following:

This again does not look even close to what we want to achieve. To ensure the correct behavior, the previously calculated amplitude (m) is integrated into the current logic by multiplying the RGB split value and the distortion ratio by the amplitude:

vec3 pixel_red = texture(u_texture, p - (0.02 + distortion) * m);

This results in the following, which looks way closer to the expected behavior:

Still missing is the blur effect. This effect is again generated by iteratively translating the image in different directions and layering these on top of each other. The following shows the blur effect with just 2 iterations:

For the actual effect, an iteration of 10 times is required to gain the expected result:

This results in the following code:

vec4 new_pixel = vec4(0);

 for (int i = 0; i < 10; i++) {
   float distortion = 0.175 + 0.005 * i;
   vec3 pixel_red = texture(u_texture, p - (0.02 + distortion) * m);
   vec3 pixel_green = texture(u_texture, p + (0.035 + distortion) * m);
   vec3 pixel_blue = texture(u_texture, p + (0.06 + distortion) * m);

   new_pixel += vec4(pixel_red, pixel_green, pixel_blue, 1)
}
new_pixel = new_pixel / 10;

and the final result of the effect:

With this result, it is finally time to jump into Flutter to integrate this effect.

Integration of the shader into Flutter

The integration of shaders is pretty straight forward and generally explained in many posts (For more information take a look at https://docs.flutter.dev/ui/design/graphics/fragment-shaders).

The first thing to add is the path to the shader in your pubspec.yaml:

flutter:
  shaders:
    - <path>/<name>.frag

To use the shader in the application, just add this single line of code in your main() function to load that specific shader:

FragmentProgram.fromAsset('<path>/<name>.frag'); 

Now, the shader can be used within every CustomPainter by assigning the shader to the Paint object:

class ShaderPainter extends CustomPainter {
  @override
  void paint(Canvas canvas, Size size) {
    var shader = _program.fragmentShader();
    final paint = Paint()..shader = shader;
    canvas.drawRect(
      Rect.fromLTWH(0, 0, size.width, size.height),
      paint,
    );
  }

  @override
  bool shouldRepaint(covariant CustomPainter oldDelegate) {
    return false;
  }
}

So now we can render any shader, but how do we adjust the variables we defined for the shader in the last section? Well, it’s a bit odd, but once understood, it just works. The idea behind this is to use indices to fill the values of our variables, e.g. 0, 1 and 2. But, there is one more clue, that is essential to understand. When working with vectors, each value within the vector has its own index. For example:

uniform vec4 pixel
uniform vec2 position

will result in the following:

shader.setFloat(0, pixel.r);
shader.setFloat(1, pixel.g);
shader.setFloat(2, pixel.b);
shader.setFloat(3, pixel.a);
shader.setFloat(4, position.x);
shader.setFloat(5, position.y);

The last parameter that must be passed to the shader is the image of the rendered widget by calling shader.setImageSampler. For this purpose, another package is used called flutter_shaders (https://github.com/jonahwilliams/flutter_shaders). While it is possible to receive the image without any additional packages, the package brings the AnimatedSampler. It allows for quick access to the image of the rendered Flutter Element the shader should be applied to and for easier animation support.

So finally, we can put together our shader integration:

AnimatedSampler(
              (image, size, canvas) {
                final shader = _program.fragmentShader();
                shader.setFloat(0, size.width);
                shader.setFloat(1, size.height);
                shader.setFloat(2, position.dx);
                shader.setFloat(3, position.dy);
                shader.setFloat(4, velocity.dx);
                shader.setFloat(5, velocity.dy);
                shader.setImageSampler(0, image);
                canvas.drawRect(
                  Rect.fromLTWH(0, 0, size.width, size.height),
                  Paint()..shader = shader,
                );
              },
child: <your Flutter Widget that the shader should be applied to>
 ),

Maybe you are wondering where the position and velocity value are coming from? This will be answered in the next section, where the animation is added.

Animation Behavior

The position and velocity are the two values still missing. The position can easily be obtained by the GestureDetector provided by Flutter:

GestureDetector(
            onPanStart: (details) {
              setState(() {
                position = details.globalPosition;
              });
            },
            onPanUpdate: (event) {
              setState(() {
                position = details.globalPosition;
              });
            },
            onPanEnd: (details) => setState(() {
position = null; }),
            onPanCancel: () => setState(() {

position = null; 
}),

This results in the following effect with a static velocity:

The velocity is the most tricky part of the animation. For this. the following behavior needs to be implemented:

With this behavior, the green point is the actual position of the user’s finger on the screen. The red point follows the green point but in a lazy way by slowly getting closer to the green point. With the help of these two points the velocity is calculated. The idea is that the distance between both points describes the velocity. This results in the following math function to calculate the velocity v, with p being the position of the green point, l being the position of the red point and dmax being the distance between p and l where the pull effect should be the strongest:

This looks very complicated, but lets break it down and take a look at the code. It is not as difficult to understand as it seems. First, the behavior can be simplified to show the general concept, which drastically reduces the complexity of the function while retaining the core concept:

The idea here is to take the vector between p and l and reduce the length to 20%. This is our velocity v. And obviously, the corresponding code is not that complicated too:

final distance = desiredPosition - position;
      velocity = distance * 0.2;

This results in the following behavior:

This is close to the wanted behavior, but the actual animation should have a more non linear behavior, i.e. the further the red point is away from the green point, the faster the red point should move.

For those who know about animations, you probably know about the Curves class and the different transformer functions it provides for transforming linear animations from 0–1 to a more curved animation:

This is basically just an implementation of the following math function:

This is exactly what we need to make the animation a little more dynamic. So let’s put that in some code:

final distance = desiredPosition - position;
final amplitude = 1 - max(0, 1000 - distance.distance) / 1000;
velocity = distance * (0.02 + 0.2 * Curves.easeOutQuart.transform(amplitude));

In the code, there is a new variable amplitude, which maps the distance of the red point (position) and the green point (desiredPosition) into a value between 0 and 1. The variable is then used to calculate the new velocity, with a minimum speed of 0.02, i.e. 2% of the distance, which goes up to 0.22, i.e. 22% of the distance between both points. To get the dynamic animation behavior the ease out curve transformation is applied. Finally, the velocity can be added to the position of the red point:

position += velocity;

Now, the velocity is calculated and the whole logic just needs to be added somewhere in the Flutter code to actually trigger the animation. So how do we do that?

Ticker

The answer is the Ticker class. While there is the AnimationController, which is ususally used for any kind of animation, we do not want any kind of duration or animated value. Instead, we want to have a continuous loop that is called over and over again, allowing for modification of values frame by frame. This is exactly what the Ticker class provides us with.

Creating a Ticker is easy. First, we need to add the SingleTickerProviderMixin to our state class. The Mixin now offers the createTicker function.

ticker = createTicker(onUpdate)..start();

Now all you have to do is call the createTicker function with a callback that will be called for each frame and start the ticker.

The callback should contain our velocity update logic:

void onUpdate(Duration elapsed) {
    final delta = ((elapsed.inMicroseconds - lastTime.inMicroseconds) / Duration.microsecondsPerSecond) * 60;
    lastTime = elapsed;
      final distance = desiredPosition - position;
      final amplitude = 1 - max(0, 1000 - distance.distance) / 1000;
    setState(() {
      velocity = distance * (0.02 + 0.2 * Curves.easeOutQuart.transform(amplitude));
      position += velocity * delta;
    });
  }

One thing that is added here is the delta value, i.e. the time difference between the frames, that takes care that our animation always runs with the same speed regardless of the FPS of the running device, e.g. 30 FPS, 60 FPS or even 120 FPS.

Left: With the delta value , Right: Without the delta value

With this final step, the effect is finally implemented:

Contact

Whether you have any questions left or you are just interested in more Flutter related topics, you can find me on X:

x.com

Source Code

The source code of this project can be found here:

GitHub - vgtle/rgb_split_distortion_shader_example: This is the source code for the rgb split distortion effect

The Magical RGB Split Distortion Effect in Flutter was originally published in neusta mobile solutions on Medium, where people are continuing the conversation by highlighting and responding to this story.