Adelta Tutorial — Part 4: Animating the Celtic Knot
[Adelta: Automatic Differentiation for Discontinuous Programs — Part 1: Math Basics]
[Adelta: Automatic Differentiation for Discontinuous Programs — Part 2: Introduction to the DSL]
[Adelta Tutorial — Part 1: Differentiating a Simple Shader Program]
[Adelta Tutorial — Part 2: Raymarching Primitive]
[Adelta Tutorial — Part 3: Animating the SIGGRAPH logo]
In the last tutorial, we use the SIGGRAPH logo to demonstrate how to use the Aδ framework to author a shader program, optimize its parameters to match a target image and animate the shader in GLSL. One caveat is that because the GLSL program is compiler-generated, it is less readable and could cause difficulty modifying the program. Last time we introduces several interfaces provided by the compiler to allow easier code modification. In this tutorial, we use another shader example to demonstrate more compiler interfaces that add extra freedom when making changes to the GLSL backend.
Target Image and Shader Program
In this example, we will write a ring shader to express the Celtic Knot icon by
Alexander Panasovsky from thenounproject.com.
The icon can be expressed with several rings. However, if we want to manually specify the parameters, there’re several challenges. First, while it may not be too difficult to manually figure out the x, y position, and radius for each ring, it can be very tedious to achieve pixel-wise perfect parameterization. Additionally, the rings are interlocking with each other, so their Z-order cannot be trivially expressed as some constant value. In fact, we will encode the Z-order for each ring as a linear function with respect to the x coordinate relative to the ring’s center. However, one might still need multiple trial and error iterations to manually figure out the scale of the linear modeling. Therefore, we’re going to author a shader program in our DSL and automatically get the best parameterization through optimization.
We will model each ring as an individual primitive. Because primitives can easily get stuck at local minimums during optimization, we will render 10 rings for optimization instead of 6 which are shown in the target image. The extra ones will either be pushed outside of the image boundary, or their radius will be shrunk into a small value to allow them to hide behind other visible primitives.
Because each ring can be rendered as an individual primitive, we will wrap the parameters related to each ring into a separate Object instance, this allows the compiler to generate semantically meaningful names to these parameters, so that it is easier to modify the GLSL program after the optimization.
rings = []
for i in range(nrings):
ring_params = [X[i + k * nrings + 2] for k in range(4)]
ring = Object('ring',
pos = ring_params[:2],
radius = ring_params[2],
tilt = ring_params[3])
rings.append(ring)With the grouped Object instances, we could easily define a function that draws one ring, and calls it 10 times on each ring. The entire program can be found here. The primitives it uses are already introduced in previous tutorials.
Finding the optimum parameters using Aδ
Similarly, we will use a multi-scale L2 to find the optimum parameter configurations. This task is more challenging because the black and white pattern provides no color hue. As a result, the optimizations converge at a lower rate. If you’re interested, this is also described in our paper Section 8.2.2. We optimize with 5 random initialization using the following command:
python approx_gradient.py --dir <path_to_store_result> --shader celtic_knot --backend hl --init_values_pool apps/example_init_values/test_finite_diffring_contour_init_values_pool.npy --modes optimization --metrics 5_scale_L2 --smoothing_sigmas 0.5,1,2,5 --learning_rate 0.01 --render_size 640,640 --gt_file celtic_knot.png --gt_transposed --multi_scale_optimization --alternating_times 5 --tunable_param_random_var --tunable_param_random_var_opt --tunable_param_random_var_seperate_opt --tunable_param_random_var_std 1 --no_reset_sigma --no_reset_opt --save_best_par --quietAlternatively, one could also execute the optimization using the Jupyter notebook here.
We show a pair of initialization and converged result with the lowest optimization error. It nicely reconstructs the original image with correct interlocking patterns.
Pruning Redundant Computation
Similar to the SIGGRAPH shader, we could create interesting animations using the compiler-generated GLSL code stored in path_to_store_result/compiler_problem.frag. An example program can be found here.
One caveat is the GLSL code includes parameters for all 10 rings, but only 6 of them are visible in the final rendering. It would be tedious to look at parameter values to figure out which ones are visible, and which ones are not. For example, at the beginning of the GLSL program, the compile defines variables for the x position on all 10 rings:
#define ring_0_pos_0_idx 2
float ring_0_pos_0 = X[ring_0_pos_0_idx];
#define ring_1_pos_0_idx 3
float ring_1_pos_0 = X[ring_1_pos_0_idx];
#define ring_2_pos_0_idx 4
float ring_2_pos_0 = X[ring_2_pos_0_idx];
#define ring_3_pos_0_idx 5
float ring_3_pos_0 = X[ring_3_pos_0_idx];
#define ring_4_pos_0_idx 6
float ring_4_pos_0 = X[ring_4_pos_0_idx];
#define ring_5_pos_0_idx 7
float ring_5_pos_0 = X[ring_5_pos_0_idx];
#define ring_6_pos_0_idx 8
float ring_6_pos_0 = X[ring_6_pos_0_idx];
#define ring_7_pos_0_idx 9
float ring_7_pos_0 = X[ring_7_pos_0_idx];
#define ring_8_pos_0_idx 10
float ring_8_pos_0 = X[ring_8_pos_0_idx];
#define ring_9_pos_0_idx 11
float ring_9_pos_0 = X[ring_9_pos_0_idx];To address this issue, our framework provides an interface called optional_update. The purpose of the interface is to identify computation that updates values, and allow the compiler to determine whether these updates are redundant and can be safely removed.
For example, in the rings shader, we update the color and Z order using an identical function call to each ring:
vals = [col, default_phase]
for i in range(nrings):
vals = update_ring(vals, rings[i], i)
return vals[0]To apply the optional_update interface, we need to wrap the update computation into a side-effect-free function call whose first input argument includes values to be updated, and returns the updated values. In our rings shader example, the update_rings() happens to be one such function. We therefore rewrite the above code block into the following, the input arguments to the optional_update interface are the subroutine function, values to be updated, and extra arguments feed into the subroutine function. The entire modified program can be found here.
vals = [col, default_phase]
for i in range(nrings):
optional_update(update_ring, vals, rings[i], i)
return vals[0]During optimization, using optional_update or directly calling the subroutine update_ring are equivalent. However, when generating the GLSL code after convergence, the compiler tries to greedily skip computations wrapped in each subroutine, and if skipping the computation has no difference to the final rendering result, this computation will be pruned.
We could use the new shader to generate GLSL code from previously optimized parameter configurations, so that we do not need to run the optimization once again. This can be achieved using the following command:
python approx_gradient.py --dir <path2_to_store_result> --shader celtic_knot2 --backend hl --init_values_pool <path_to_store_result>/best_par.npy --modes render --render_size 640,640 --gt_file celtic_knot.png --gt_transposedNote in the above command, <path_to_store_result> should be the path used for the optimization, but <path2_to_store_result> should be a different path to make sure each shader is compiled to a different directory. One could also make the following change to the same Jupyter notebook we used for optimization and run the cells again. The notebook should be restarted before running on a different shader, to make sure all pybind modules are correctly loaded.
run_optimization = False
shader_suffix = '2'The compiler-generated GLSL code is stored in path_to_store_result2/compiler_problem.frag. An example program can be found here. If we look at the shader parameters again, they only contain the ones corresponding to the 6 visible rings now.
#define ring_1_pos_0_idx 2
float ring_1_pos_0 = X[ring_1_pos_0_idx];
#define ring_2_pos_0_idx 3
float ring_2_pos_0 = X[ring_2_pos_0_idx];
#define ring_3_pos_0_idx 4
float ring_3_pos_0 = X[ring_3_pos_0_idx];
#define ring_5_pos_0_idx 5
float ring_5_pos_0 = X[ring_5_pos_0_idx];
#define ring_6_pos_0_idx 6
float ring_6_pos_0 = X[ring_6_pos_0_idx];
#define ring_8_pos_0_idx 7
float ring_8_pos_0 = X[ring_8_pos_0_idx];Animation that Modifies Intermediate Variables
Similar to the SIGGRAPH example, the compiler already inserts empty functions in the GLSL code to allow easy modification to shader parameter values. But what if we need more freedom in code modification? For example, we may want to color the rings with some rainbow palette. But because the original shader simply renders white to ring interiors, no parameter can be changed to achieve such effect. We have to look into the compiler-generated main shader function and look for places to modify the intermediate variable values.
To avoid this tedious extra work, our compiler provides another interface called Animate, that allows programmers to insert dummy functions to update intermediate variables that they wish to animate later.
For example, the color of each ring is set using two constant color vectors: the black edge_col and the white fill_col :
ring.fill_col = fill_col
col_current = Var('col_current_%s' % ring.name, select(cond1, edge_col, ring.fill_col))To color each ring, we need to update fill_col into a different value for each ring. This can be achieved by using the Animate interface to insert a dummy function in the compute graph where the update needs to be applied. The interface specifies the name and input signature of the dummy function. In analogy to the in and inout keywords in the GLSL function, the interface also distinguishes the list of variables that should be unchanged, and the list of variables that needs to be modified using in_ls and inout_ls keyword arguments separately. The above two lines of code can be rewritten as follows. Because we plan to build a rainbow-like palette around the ring, we also input the relative pixel position with respect to the ring center as the function signature.
ring.fill_col, = Animate("animate_ring_col_%d" % idx, inout_ls=[fill_col], in_ls=[rel_pos]).update()
col_current = Var('col_current_%s' % ring.name, select(cond1, edge_col, ring.fill_col))Similarly, we could apply Animate to the u, v coordinates to achieve a global rotation effect. The entire program can be found here.
u, v = Animate("animate_uv", inout_ls=[Var("u", u), Var("v", v)]).update()Again, we could generate the GLSL program to the new shader using previously optimized parameterizations:
python approx_gradient.py --dir <path3_to_store_result> --shader celtic_knot3 --backend hl --init_values_pool <path_to_store_result>/best_par.npy --modes render --render_size 640,640 --gt_file celtic_knot.png --gt_transposedAlternatively, one could make the following modification to the Jupyter notebook and restart the kernel to generate the GLSL program using the new shader. An example GLSL program can be found here.
run_optimization = False
shader_suffix = '3'We could find empty functions are inserted before the main function. For example, we could color each ring based on its relative angular position with respect to the ring center. The color palette function can be copied from this Palettes shader by Shadertoy author Inigo Quilez.
float PI = 3.141592653589793;float get_t(float x, float y, float speed) {
return atan(x, y) / 2. / PI + iTime * speed;
}void animate_ring_col_8(
inout vec3 fill_col,
in float rel_pos_8_x,
in float rel_pos_8_y){ float t = get_t(rel_pos_8_x, rel_pos_8_y, 0.5);
fill_col = pal(t, vec3(0.5,0.5,0.5),vec3(0.5,0.5,0.5),vec3(1.0,1.0,1.0),vec3(0.0,0.33,0.67) );}
The rest of the rings can also be colored similarly, but with slightly different palette settings or rotation rates.
void animate_ring_col_6(
inout vec3 fill_col,
in float rel_pos_6_x,
in float rel_pos_6_y){ float t = get_t(rel_pos_6_x, rel_pos_6_y, 1.);
fill_col = pal(t, vec3(0.5,0.5,0.5),vec3(0.5,0.5,0.5),vec3(1.0,1.0,1.0),vec3(0.0,0.10,0.20) );}void animate_ring_col_5(
inout vec3 fill_col,
in float rel_pos_5_x,
in float rel_pos_5_y){ float t = get_t(rel_pos_5_x, rel_pos_5_y, 0.25);
fill_col = pal(t, vec3(0.5,0.5,0.5),vec3(0.5,0.5,0.5),vec3(1.0,1.0,1.0),vec3(0.3,0.20,0.20) );}void animate_ring_col_3(
inout vec3 fill_col,
in float rel_pos_3_x,
in float rel_pos_3_y){
float t = get_t(rel_pos_3_x, rel_pos_3_y, 2.);
fill_col = pal(t, vec3(0.5,0.5,0.5),vec3(0.5,0.5,0.5),vec3(1.0,1.0,2.0),vec3(0.8,0.90,0.30) );}void animate_ring_col_2(
inout vec3 fill_col,
in float rel_pos_2_x,
in float rel_pos_2_y){ float t = get_t(rel_pos_2_x, rel_pos_2_y, 4. / 3.);
fill_col = pal(t, vec3(0.5,0.5,0.5),vec3(0.5,0.5,0.5),vec3(2.0,1.0,0.0),vec3(0.5,0.20,0.25) );}void animate_ring_col_1(
inout vec3 fill_col,
in float rel_pos_1_x,
in float rel_pos_1_y){
float t = get_t(rel_pos_1_x, rel_pos_1_y, 2. / 3.);
fill_col = pal(t, vec3(0.8,0.5,0.4),vec3(0.2,0.4,0.2),vec3(2.0,1.0,1.0),vec3(0.0,0.25,0.25) );}
We could further give a global rotation to the entire image. The final GLSL program can be found in this Shadertoy link that allows interactive animation.
void animate_uv(
inout float u,
inout float v){ vec2 rel_pos = vec2(u, v) - vec2(width, height) / 2.;
float r = length(rel_pos);
float theta = atan(rel_pos.x, rel_pos.y) + iGlobalTime * PI / 2.; u = r * sin(theta) + width / 2.;
v = r * cos(theta) + height / 2.;}
Summary
This post uses the Celtic knot icon as an example to demonstrate additional compiler interfaces for animating the machine-generated GLSL code. If the shader program used for optimization renders redundant primitives to avoid local minimum, the optional_update interface helps automatically pruning redundant computation after the optimization has converged. The compiler further provides the Animate interface to help modifying the intermediate variables of the machine-generated GLSL program. This will insert empty functions with input signatures containing user-variables at program locations where the user imagined the code modification to take place, and avoids exposing the less readable machine-generated main function to the user.
Reference
[1] Aδ: Autodiff for Discontinuous Program — Applied to Shaders: [project page] [Github]
[2] Celtic Icon by Alexander Panasovsky: [the Noun Project]
[2] Palettes by Inigo Quilez: [Shadertoy]