TLDR

[Source code] — [Unwalled article]

Introduction

Metaballs! Yeah, like meatballs, but no meat.

Before jumping into that, I recall, years ago, when a colleague at the office gave a small talk about Fragment Shaders.
At that time, I thought I’d be better off deciphering the Necronomicon (including searching for it in the local library).

Today, my almost absurd and preferred simplification to explain the concept is that it’s just a function that returns 4 values, which turn out to be the color of the pixel to be drawn (Red, Green, Blue, and Alpha).

Of course, this is a partial simplification of something more complex than that, but why scare innocent people too early?
All this fits within the world of ‘how to draw beautiful things on the screen’.
And this post is precisely about that.

Well, maybe not so beautiful things, but some basic attempts.

Metaballs: What are they?

This brings us back to Metaballs: What are they? The simplest and most direct way is: ‘Do you remember those lava lamps?’

Those viscous balls that would join and then separate in a somewhat animal-like way?

Well, Metaballs are about that effect, and this article, like many others, is about how to achieve it.

How to ‘achieve’ the effect

Simplifying, I’ve seen two approaches out there on how to do it.
The mathematical way, and the slightly trickier way.

The first approach, more elegant, follows a technique called “Marching Cubes”.
It was published in 1987 in a paper called “Marching Cubes: A High Resolution 3D Surface Construction Algorithm”.

In a simplified view, it uses information from circumferences and the vertices of a grid of cubes in space to draw.
Draw what? Well, whatever needs to be drawn.

I know that it sounds strange, but let’s say on one side we have a grid with sensors at each vertex, and on the other side our balls.
(Plus, a ball can be over multiple sensors).

Whenever a ball is over a sensor, the sensor will be active, then, taking a set of sensors, we will draw lines according to which sensors are active in that set. And so on until we complete our entire grid.

Easier to say than done?
This is an excellent resource to get a grasp of it
(this is the 2D version, squares instead of cubes)

In the second approach, slightly more ingenious tricks are used to calculate or compose the effect of ‘closeness’ or ‘influence’, and according to this, we draw!

Specifically, we can use elements that have an opaque color with a semi-transparent gradient border.
Now we can use the alpha values (which indicate how opaque or transparent the color is) to achieve the the final effect.
So, in a post-drawn process, we define some alpha thresholds to draw our Metaballs.
We can define more than one level, so we can have one color for the body and another for the border.

In the case of a single element, there’s nothing very interesting to see.
But when two bodies get close, their respective semi-transparent borders will join forces to ‘saturate’ the alpha channel until they reach the threshold that indicates that we have to draw something on the screen.

In summary;

• Marching cubes: 1987 paper called ‘Marching Cubes: A High Resolution 3D Surface Construction Algorithm’
• Calculate or draw closeness/influence, paint body and contours according to it

Some implementations / Tutorials

In the first approach:

• 1.a. This tutorial of Jamie Wong is quite clear and illustrative
  Note that this technique works for 3D and also for 2D, hence the name “Marching Cubes” or “Marching Squares”.

In the second category, I share with you these four implementations;

• 2.a. A cool tutorial from Daniel Lilett where he uses the metaball positions and radius to calculate the ‘closeness’ and to draw the ‘area of influence’; then ‘apply the colors according to it’ on a render pass
• 2.b. A smart and tricky approach where Artjoms Neimanis uses the Blur effect to obtain the ‘alpha’ channel, overlapping balls will saturate the ‘area of influence’ and an extra camera applies the colors according to the rules on a shader in a dedicated render texture
• 2.c. An evolved version of the previous one by HuvaaKoodia here we avoid the use of an extra camera and the render texture, applying the effect on a render pass
• 2.d. And another tutorial from Bronson Zgeb similar to the previous one, Blur + URP render pass!

So, why another one?

With Marching, there are some limitations, you can’t just put different colors on the edges, just ‘contours’ (Although with some work I think it could be done, for instance; having a second system that would draw a second pass with a more adjusted ‘contour’ level?).

On the others, I would like to be able to avoid blur or to have to pass information about the location of each metaball on each frame.

And that’s the reason for this post; here I’m not coming to sell but to give away.

My idea is to remove the blur, using the alpha trick (which we will achieve with a shader) and then use a render pass with the coloring rules using another shader, similar to the last 2 posts, but this time we will use RenderGraph, which according to the Unity people, is what’s in vogue these days…

Why does the blur bother me?

The blur effect is calculated by copying, moving, and overlaying a semi-transparent version of the texture, giving the desired effect.
And this is done over multiple ‘passes’.
On low-end devices, this can negatively affect your frame rate (especially if you’re doing a blur on every frame!), so it’s sometimes good to look for alternatives.

So, what do we need?

• Unity 6
• A Universal Render Project (Since we’re in Unity6, the URP version is 17.03)

First step: Shader for Circles

Instead of blur, we are going to use a shader to draw our Metaballs with an opaque body and a gradual semi-transparent border (more opaque to the center, more transparent to the ‘outside’).

The idea is to define which part of our circle will be solid and which part will be our ‘area of influence’, that is, when two balls are close, their areas of influence will be added to make the linking effect appear.

For this, we have the following code. Pretty simple, but it works.

Shader "Custom/GradientCircle"
{
    Properties
    {
        _Color ("Color", Color) = (1.0, 1.0, 1.0, 1)
        _Radius ("Radius", Range(0, 1)) = 1.0
        _Smoothness ("Smoothness", Range(0, 1)) = 0.8
    }
    
    SubShader
    {
        Tags {
            "RenderType"="Transparent" 
            "RenderPipeline"="UniversalPipeline" 
            "Queue"="Transparent"
        }
        LOD 100
        Blend SrcAlpha OneMinusSrcAlpha
        ZWrite Off
        
        Pass
        {
            HLSLPROGRAM
            #pragma vertex vert
            #pragma fragment frag
            #include "Packages/com.unity.render-pipelines.universal/ShaderLibrary/Core.hlsl"
            
            struct Attributes
            {
                float4 positionOS : POSITION;
                float2 uv : TEXCOORD0;
            };
            struct Varyings
            {
                float2 uv : TEXCOORD0;
                float4 positionCS : SV_POSITION;
            };

            CBUFFER_START(UnityPerMaterial)
                float4 _MainTex_ST;
                float4 _Color;
                float _Radius;
                float _Smoothness;
            CBUFFER_END

            Varyings vert(Attributes IN)
            {
                Varyings OUT;
                OUT.positionCS = TransformObjectToHClip(IN.positionOS.xyz);
                OUT.uv = IN.uv;
                return OUT;
            }

            half4 frag(Varyings IN) : SV_Target
            {
                // Normalize UV coordinates to range from -1 to 1
                const float2 uv = IN.uv * 2.0 - 1.0;
                // Circle center (in normalized coordinates)
                const float2 center = float2(0.0, 0.0);
                // Distance from current point (uv) to circle center
                const float dist = length(uv - center);
                // Smooth gradient: 1.0 at center, 0.0 at edge
                // The smoothstep function produces a smooth transition between 0.0 and 1.0
                // based on the distance from the circle center and the defined radius and smoothness
                const float smooth_circle = smoothstep(_Radius, _Radius - _Smoothness, dist);
                // The final alpha value is the smooth gradient value
                half4 color = _Color;
                color.a *= smooth_circle;
                
                return color;
            }
            ENDHLSL
        }
    }
}

We create a new material with this shader and, voilà:

Second step: URP Custom Pass | Render Feature

Once we have all the spheres on the screen, we are ready to draw the effect.

To do this, we will create a second shader and a custom render pass. Well, strictly speaking, two passes.

This second shader will draw depending on the amount of alpha present, in this way, we can paint the body of the Metaballs and the border.

This time, we will use Shader Graph, and we are going to call it MetaEffect:

With these settings (The important part: Material = Fullscreen):

And with the following properties:

The Border Color and Body Color parameters represent the respective colors of the border and the body.
The Min Alpha Threshold defines the minimum alpha value that will be considered as part of the border, while the Body Threshold specifies the minimum alpha value that will be classified as part of the body.
Any alpha value that falls between the Min Alpha Threshold and the Body Threshold will be regarded as part of the border.

Now, for our custom render pass!

In the first pass, we draw our semi-transparent balls on a temporary texture.

There are some caveats though.
On one hand, we don’t want to show our original balls, just the effect of it.
And we don’t want to apply this to everything that is on the screen, only to our balls.

To do this, in our render pass we will work only with a specific layer, so we will create a FilteringSettings to only select a specific layer during the process:

// Settings to filter which renderers should be drawn
private readonly FilteringSettings _filterSettings = default;

...
_filterSettings = new FilteringSettings(renderQueueRange, layerMask);
...

private void InitRendererLists(ContextContainer frameData, ref LayerRenderPassData layerRenderPassData, RenderGraph renderGraph)
{
    // Access the relevant frame data from the Universal Render Pipeline
    var universalRenderingData = frameData.Get<UniversalRenderingData>();
    var cameraData = frameData.Get<UniversalCameraData>();
    var lightData = frameData.Get<UniversalLightData>();
    var sortingCriteria = cameraData.defaultOpaqueSortFlags;
    
    // Create drawing settings based on the shader tags and frame data
    var drawSettings = RenderingUtils.CreateDrawingSettings(_shaderTagIdList, universalRenderingData, cameraData, lightData, sortingCriteria);
    
    // Create renderer list parameters,
    // here we are using _filterSettings, and that is where we specified the layerMask to use
    var param = new RendererListParams(universalRenderingData.cullResults, drawSettings, _filterSettings);
    
    // Finally create a RenderListHandle 
    layerRenderPassData.RendererListHandle = renderGraph.CreateRendererList(param);
}

Then our first pass would look like this:

private static void ExecuteLayerRenderPass(LayerRenderPassData data, RasterGraphContext context)
{
    // Draw all renderers in the list
    context.cmd.DrawRendererList(data.RendererListHandle);
}

...

// Set up the layer render pass
const string layerRenderPassName = "Mat2Layer: Layer Render 1/2";
using (var builder = renderGraph.AddRasterRenderPass<LayerRenderPassData>(layerRenderPassName, out var passData))
{
    InitRendererLists(frameData, ref passData, renderGraph);
    builder.UseRendererList(passData.RendererListHandle);
    
    // Set up texture dependencies
    // We are not really using 'srcCamColor' on this pass,
    // but we are going to keep the next line for clarity and documentation... 
    builder.UseTexture(srcCamColor); 
    builder.SetRenderAttachment(temporaryHandle, 0);
    builder.SetRenderAttachmentDepth(srcCamDepth);
    
    builder.SetRenderFunc((LayerRenderPassData data, RasterGraphContext context) => ExecuteLayerRenderPass(data, context));
}

Here is worth note that we follow the structure of:
a.- setting a source (with builder.UseTexture(srcCamColor)),
b.- setting the destination (with SetRenderAttachment(temporaryHandle, 0) and SetRenderAttachmentDepth(srcCamDepth)),
c.- doing some work (with builder.SetRenderFunc)

But, as you can see in the comments, we are not really using srcCamColor (ie; ‘what is rendered in the camera so far’), instead we are instructing our pass to use our FilteringSettings defined onto RendererListHandle, in this line: builder.UseRendererList(passData.RendererListHandle);

In the second pass, we apply the second shader to what was drawn in the previous pass (stored in our temporary texture handle called temporaryHandle) with BlitTexture.
And we draw the result on the screen.

Something like this:

private static void ExecuteBlitPass(BlitPassData data, RasterGraphContext context)
{
    // Blit the source texture to the current render target using the specified material
    Blitter.BlitTexture(context.cmd, data.Source, ScaleBias, data.Material, 0);
}
...

// Set up the blit pass
const string blitPassName = "Mat2Layer: Blit Pass 2/2";
using (var builder = renderGraph.AddRasterRenderPass<BlitPassData>(blitPassName, out var passData))
{
    // Configure pass data
    passData.Material = _material;
    // Use the output of the previous pass as the input
    passData.Source = temporaryHandle;
    builder.UseTexture(passData.Source);
    
    // Set the render target to the original color buffer
    builder.SetRenderAttachment(srcCamColor, 0);
    builder.SetRenderAttachmentDepth(srcCamDepth);
    
    builder.SetRenderFunc((BlitPassData data, RasterGraphContext context) => ExecuteBlitPass(data, context));
}

All the details are in MaterialToLayerRenderPass.cs take it a look, there are a lot of comments!

What is left is to create a new Layer, and assign our balls to it.
Then, we will unselect that Layer from the rendering process (since we are going to take care of the drawing itself)

Unity Scene

In our scene, we will start with a plane to which we will apply the material that we made from our GradientCircle shader.

Then, we will place these elements on a new Layer called Metaballs.

On the other hand, we will have our UniversalRendererData (We can start with PC_Renderer, but remember that if we’re going to use this on mobile, then we’ll have to modify MobileRenderer) here we will first deselect the Metaballs layer, with this, the camera will not draw our balls on the screen!

After that, we add a new render feature; ours!

So we start looking for it:

And we add it, and configure it!

We configure it by telling it that:

• Render Queue = All
• LayerMask = Metaballs
• Material = MetaEffect
• RenderPassEvent = After Rendering SkyBox.

Results

So, with one Metaball we should see this;

But, more of them;

And if we go to the Frame Debugger we will see what is happening behind the curtains:

And since we are here, let’s take a look at the Render Graph Viewer

Conclusion: Blobs and Beyond

And there you have it. We’ve ventured from abstract equations to smooth, organic shapes dancing across your screen. Metaballs may seem like simple blobs at first, but behind them lies a world of blending functions, threshold values, and a bit of computational magic.

The next time you see those mesmerizing, fluid visuals in games or animations, remember that it’s more than just pixels — it’s a mix of math and code working together.

I hope you enjoyed reading this tutorial and, as usual, comments, doubts, questions, suggestions, etc… are all welcome!

Until next time!

TLDR

[Source code] — [Unwalled article]

Introduction

Hey there! Today, we’re going to dive into something that might make your head spin faster than a dusty CPU fan on a hot summer day: 3D projection… in Rust.

Yep, I know that sounds crazy, but bear with me.
This rudimentary version is just a naive approximation of what game engines do nowadays.

I was heavily inspired by this video [ASMR Programming — Spinning Cube — No Talking], so thanks to its author!

The General Idea

First, let’s set the plan.
The idea is to render a 3D cube on a 2D screen.
And to do that, we are going to make use of that thing called Math, and, to make everything a little bit more complicated, we are going to write a program in Rust to achieve our mission.

So, it’s not your one-line “Hello, World!” program, but it’s not rocket science either. Well, actually, it kind of is — but the fun kind, where no one gets blown up if we mess up a calculation.

What are we going to do is:

i. Take each face of the cube
ii. Apply math to calculate the position in the 3D space, taking into consideration rotations on the X, Y, or Z axis
iii. Draw the points that are closer to the camera, applying some projection to make things bigger or smaller accordingly

The Math Behind the Madness

Now, before you run away screaming “Math! My old nemesis!”, let me tell you something, we are not trying to publish a post-doc thesis.

All the formulas that we are going to use look scarier than they really are, so even if math is not your thing, let’s give them a chance!

In this Wikipedia article on the Rotation matrix, they explain to us that:

…a rotation matrix is a transformation matrix that is used to perform a rotation in Euclidean space.

This could be performed in two dimensions, but that is not enough for us, so; fast your seat belt; and prepare to travel to the next level; The 3D world!

So if we jump to this part of the article we have:

This is one of many ways to use math to apply the required transformations (yep, another and arguably better are Quaternions!).

If we multiply this little monster by a [i, j k] vector, we have;

which gives us;

And that’s all, that is the farther that we are going to go, in terms of esoteric math.
That is our rotation matrix!

This, in a code-friendly style, translates each row to the calculate_x, calculate_y, and calculate_z functions in our rendering.rs file.

fn calculate_x(
 i: f32, 
 j: f32, 
 k: f32, 
 alpha: f32,
 beta: f32,
 gamma: f32,
) -> f32 {
 let sin_a = alpha.to_radians().sin();
 let cos_a = alpha.to_radians().cos();
 let sin_b = beta.to_radians().sin();
 let cos_b = beta.to_radians().cos();
 let sin_c = gamma.to_radians().sin();
 let cos_c = gamma.to_radians().cos();
i * cos_a * cos_b // < - - I re arranged the terms a little bit, first `i` components, then `j`, then `k`…
 + j * cos_a * sin_b * sin_c
 - j * sin_a * cos_c
 + k * cos_a * sin_b * cos_c 
 + k * sin_a * sin_c
}
…

Why So Trigonometric?

You might be wondering, “Why all the sines and cosines? Did the developer have a particular fondness for waves?” Well, not exactly. Sines and cosines are like The Good, the Bad and the Ugly of rotation in mathematics (in other words, a classic). They allow us to describe circular motion, which is exactly what we need for rotation.

Let’s break it down:

i. alpha, beta, and gamma are our rotation angles around the z, y, and x axes respectively (why this inverse order? Well, remember that Wikipedia article? Different transformations can be used, and this one uses alpha as yaw, beta as pitch, and gamma as roll angles)
ii. We convert these angles to radians (because computers prefer radians to degrees, they’re quirky like that)
iii. We calculate the sine and cosine of each angle
iv. We then combine these values in a specific way to rotate our point

From 3D to 2D: The Projection

Now that we’ve rotated our point, we need to project it onto our 2D screen. This is where the calculate_for_surface function comes in:

fn calculate_for_surface(
  cube_x: f32,
  cube_y: f32,
  cube_z: f32,
  ch: char,
  z_buffer: &mut [f32; CANVAS_WIDTH * CANVAS_HEIGHT],
  buffer: &mut [char; CANVAS_WIDTH * CANVAS_HEIGHT],
  alpha: f32,
  beta: f32,
  gamma: f32,
  distance_from_camera: f32,
  projection_scale: f32,
) {
  // Apply rotation transformations to the cube coordinates
  let x = calculate_x(cube_x, cube_y, cube_z, alpha, beta, gamma);
  let y = calculate_y(cube_x, cube_y, cube_z, alpha, beta, gamma);
  let z = calculate_z(cube_x, cube_y, cube_z, alpha, beta, gamma);
  // Adjust z-coordinate based on camera distance
  let z = z - distance_from_camera;
  
  // Calculate the inverse of z (ooz: one over z)
  //
  // In our coordinate system:
  // i.- z = 0 is at the origin
  // ii.- Positive z values are closer to the camera
  // iii.- Negative z values are further from the camera
  //
  // After subtracting distance_from_camera, all z values become negative
  // Smaller negative z values are closer to the camera
  let ooz = - 1.0 / z;  
  // So now, regarding ooz:
  // i.- All values are positive
  // ii.- Larger values indicate closer proximity to the camera (useful for z-buffer comparisons)
  
  // Convert 3D coordinates to 2D screen space
  // Note how we use here ooz to 'shrink' or 'expand' the projection
  // xp = ... + x * ...: Positive x values move the point to the right, so we add to the center of the screen.
  let xp = (CANVAS_WIDTH as f32 /2.0 + x * ooz * projection_scale * ASPECT_RATIO) as isize;
  // yp = ... - y * ...: Positive y values (in 3D space) move the point up, 
  // but because screen space has the y-axis increasing downward, we subtract to move the point upward on the screen.
  let yp = (CANVAS_HEIGHT as f32 / 2.0 - y * ooz * projection_scale) as isize;


  // Check if (xp,yp) point is within the canvas boundaries
  if xp < 0 || (xp as usize) >= CANVAS_WIDTH {
      return;
  }
  if yp < 0 || (yp as usize) >= CANVAS_HEIGHT {
      return;
  }
  
  // Calculate the buffer index for the current point
  let idx = xp + yp * CANVAS_WIDTH as isize;
  let idx = idx as usize;
  
  // Update the z-buffer and character buffer only if this point is closer to the camera
  if ooz > z_buffer[idx] {
      z_buffer[idx] = ooz;
      buffer[idx] = ch;
  }
}

This function is where the magic happens. It takes our 3D point, rotates it, and then projects it onto our 2D canvas.

The Secret Sauce: Perspective Division

The key to this projection is the line let ooz = — 1.0 / z;. This is called perspective division, and it’s what gives our projection that 3D feel.

Here’s why it works:

i. Objects further away appear smaller.
ii. In our 3D space representation, smaller z values mean the point is further away (and since our origin is 0.0, the more negative a value is, is further away). And since we push away our points even more with - distance_from_camera all our points should be expected to be negative.
iii. By multiplying by -1.0 and dividing by z, we have a variable ooz that has positive values. The larger the closer to the camera, the bigger that should be on screen.
iv. So, points with smaller ooz values (further away) have a smaller effect on the final x and y coordinates.

It’s like squishing the entire 3D world onto your flat screen, with the added bonus of making far-away things look… well, far away.

Putting It All Together

Now that we understand the individual pieces, let’s look at how they all fit together in the draw_cube function:

Here we will ‘draw’ each face of the cube, point by point.

ub fn draw_cube(
  z_buffer: &mut [f32; CANVAS_WIDTH * CANVAS_HEIGHT],
  buffer: &mut [char; CANVAS_WIDTH * CANVAS_HEIGHT],
  params: &CubeParameters,
) {
  let mut cube_x = -1.0 * HALF_CUBE_WIDTH as f32;
  while cube_x < HALF_CUBE_WIDTH as f32 {
    let mut cube_y = -1.0 * HALF_CUBE_WIDTH as f32;
    while cube_y < HALF_CUBE_WIDTH as f32 {
      // Axis following Rigth-Hand rule
      //      y
      //      |
      //      |
      //      |_ _ _ _ x
      //     /
      //    /
      //   z
      //

      //  Plane K; Back side, We update X and Y, and keep Z constant 
      //        ______
      //      /|  K   |
      //     / |      |
      //    |  |______|
      //    | /      /
      //    |/______/
      //
      let x_value = cube_x;
      let y_value = cube_y;
      let z_value = -1.0 * (HALF_CUBE_WIDTH as f32);
      calculate_for_surface(
        x_value, y_value, z_value,
        'K',
        z_buffer, buffer,
        params.alpha, params.beta, params.gamma,
        params.distance_from_camera, params.projection_scale);

      // Plane F; Front side, We update X and Y, and keep Z constant 
      //       ______
      //     /      /|
      //    /______/ |
      //    |      | |
      //    |  F   | /
      //    |______|/
      //
      let x_value = cube_x;
      let y_value = cube_y;
      let z_value = 1.0 * (HALF_CUBE_WIDTH as f32);
      calculate_for_surface(
        x_value, y_value, z_value,
        'F',
        z_buffer, buffer,
        params.alpha, params.beta, params.gamma,
        params.distance_from_camera, params.projection_scale);


      // ... draw other faces ...

      cube_y += params.resolution_step;
    }
    cube_x += params.resolution_step;
  }
}

This function is like a meticulous artist, painting our cube one point at a time. It loops over the x and y coordinates of each face, calling calculate_for_surface for each point.
The result? A beautiful 3D cube rendered on our 2D screen.

Check how we are using letters to represent each face of the cube:

F = Front, K = Back,
L = Left, R = Right,
T = Top, B = Bottom.

The Z-Buffer: Our Unsung Hero

You might have noticed we’re passing around something called a z_buffer. This isn’t just some fancy term we made up to sound smart (although it does sound pretty cool). The z-buffer is crucial for determining which parts of our cube are visible.

In the calculate_for_surface function, we have this bit of code:

// Update the z-buffer and character buffer only if this point is closer to the camera
if ooz > z_buffer[idx] {
 z_buffer[idx] = ooz;
 buffer[idx] = ch;
}

This is checking if the current point is closer to the camera than what we’ve drawn so far. If it is, we update the z-buffer and draw the point.

This ensures that closer parts of the cube properly obscure parts that are further away. It’s like giving our rendering engine depth perception!

And yes, if you were wondering, the concept of Z-fighting had to come from somewhere!

Conclusion: From Math to Magic

So there you have it, folks. We’ve taken a journey from dry mathematical formulas to a living, breathing 3D cube rendered in ASCII characters. We’ve rotated points in 3D space, projected them onto a 2D plane, and even given our rendering engine the ability to understand depth.

The next time you’re playing your favorite 3D game, spare a thought for the math happening behind the scenes. It’s not just pushing pixels — it’s a beautiful dance of trigonometry, linear algebra, and clever programming.

Remember, every masterpiece of 3D graphics you see, from the latest AAA game to that cool 3D chart in your PowerPoint presentation, is built on these fundamental principles. It’s just that most of the time, the math is hidden away in optimized graphics libraries and powerful GPUs.

But now you know the secret. You’ve peeked behind the curtain and seen the wizard at work.

Until next time, keep your code clean, your algorithms efficient, and your cubes rotating!

—

Finally, the code is in this repo, all ideas, suggestions, and comments are welcome!

TLDR
[Unwalled article]

This article discusses the integration of Rust code into Unity, divided into two main sections: Setting up the Rust plugin and integrating it into a Unity project.

First: Rust Plugin Setup

Create Rust Library Project: Use Cargo, Rust’s package manager, to create a new Rust library project. Here we will create a library called uni_plugin.
Open your terminal and execute the following command:

cargo new uni_plugin --lib

Specify Dynamic System Library Type: Update the cargo.toml file to specify the crate type as cdylib for dynamic system library linkage.
Our Cargo.toml will look like this:

[package]
name = "uni_plugi"
version = "0.1.0"
edition = "2021"

[lib]
crate-type = ["cdylib"]

[dependencies]

Write Rust Code: Define the desired functionality in lib.rs. Since the purpose of this guide is to test the library integration, we will use the code provided by default.

Peeeero, pero, pero, to properly “export” the functions with the name that we defined (and avoid Rust doing some magic that usually does to avoid naming clashes when linking) we have to use the attribute#[no_mangle]. Failing to do this will result in an EntryPointNotFoundException since the method will not be found.

So our almost by default code will be something like this:

#[no_mangle]
pub fn add(left: usize, right: usize) -> usize {
    left + right
}

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn it_works() {
        let result = add(2, 2);
        assert_eq!(result, 4);
    }
}

Build the Library: Build the Rust library by executing this on the terminal:

cargo build

This will generate the library file in the target/debug directory. On Windows, the file will have a .dll extension, on macOS .dylib, and on Linux .so.

Second: Unity Integration

Create Plugins Folder: On your Unity project, create a new folder named Plugins inside the Assets directory, then;

Move/Copy Library File: Move or copy the generated library file (libuni_plugin.so/dylib/dll) into the Plugins folder.

Unity Invocation: To utilize the functionality provided by the Rust library in Unity, create a C# script. For example, create a script named LibraryInvocation.cs in the Assets/Scripts folder. Attach this script to a GameObject in the scene. For instance, create an object named Scripts and add the script as a new component.

Here’s the important part; How do we execute the code of our Rust library in our C# code?

Declare External Function: In the C# script, declare the external function using the DllImport attribute. Adjust the library name and method signature accordingly. Our code looks like this:

using System.Runtime.InteropServices;
using UnityEngine;

public class LibraryInvocation : MonoBehaviour
{
#if UNITY_IPHONE
    [DllImport ("__Internal")]
#else
    [DllImport ("Libuni_plugin")]   
#endif
    private static extern uint add(uint left, uint right);
    
    private void Start()
    {
        Debug.Log("Starting the app");
        
        var results = add(1, 2);
        Debug.Log($"Results from library: {results}");
    }
}

Now, press play to see the results displayed in the console.

Notes and Links:

Exporting the library to different platforms would depend on the platform itself, here is an excellent resource to go deeper on this.

Source of inspiration: Creating A DLL With Rust

Unity Documentation about plugins: Native Plugins

TLDR
[Source code] — [Unwalled article]

In another episode of Learn in Public, I am here to share a little experiment that uses MoveNet to estimate a human pose of a video or the input from a webcam. Then we’ll send that information through our old and loved User Datagram Protocol (yes, these are the words behind UDP).

Basic Structure

We are going to use Python and the program can be logically split into three parts:

i.- Inference Hub: MoveNet is a model offered by TensorFlow. Following the instructions/idea from this tutorial, in the Inference Hub, we initialize the desired flavor of the model and set up the details to invoke the inference method.

ii.- Frame Analysis: Since we are analyzing a video we will extract frame by frame using OpenCV to calculate the inference in a loop.

iii.- Pose Keypoints Transmission: The last part is all about transmitting the serialized results of the inference over the network using UDP.

Inference Hub

This part of the is pretty straightforward, following the tutorial from the TensorFlow site, we extract the functionality to download the desired model (we can choose from lightning or thunder, and its FP16 and INT8 quantized versions) with its inference invocation, the logic to crop a relevant section of the frame (this will help us to make our inference more efficient), and the logic to convert the inference results to key points and edges.

The code is inside pose_transmitter/inference_hub with hub.py as the main entry point.

Frame Analysis

The frame extraction is made in the VideoPoseProcessor object where after the initialization we are ready to run a loop processing our video.

The code is in pose_transmitter/video_pose_processor.py

The parameters are the inference_hub, the source, and a debug flag.

We use OpenCV to work with the video file or with the camera input.
The processing loop is as follows:

Note the line 23 of the snapshot. Since OpenCV represents the colors of the image in the Blue-Green-Red (BGR) format we need to convert it to the Red-Green-Blue (RGB) format, which is the format that TensorFlow uses.

After running the inference we invoke the callback (in case of being defined) and display_results if specified.

The entry point of this logic is the method called start_processing with the only parameter the callback to process the results.

Note how if the debug flag is present, we cannot use threads.
The logic behind this is that all the operations that display UI elements have to run in the main thread. And since the debug flag implies showing the results on the screen (display_results=True) we are limited in this regard.

Pose Keypoints Transmission

The data to be transmitted is serialized and then placed in a Queue. A worker thread is responsible for retrieving and dispatching these messages.

Our selection in this case is a simple UDP transmission, serializing the data using MsgPack which is a simpler approach compared to other serialization methods (like Procolol Buffers or Thrift), with support in different languages, and also more efficient than simple JSON.

The code is in pose_transmitter/qudp_transmitter.py
We can see the constructor for QUDPTransmitter here:

The code that receives the data (and what is used as the callback in the VideoPoseProcessor is this put_message method:

We activate the worker thread with _activate_transmission and while is active, we will be sending the data through _send:

The data consists of a list of pairs of floating-point values (List<float[]>):

[
    [989.79443359375, 635.7391967773438], 
    [1091.6107177734375, 512.1315307617188], 
    [889.1296997070312, 520.48828125], 
    [1204.3812255859375, 533.4179077148438], 
    ...
]

📝 Note:
It is worth noting that not all points may be inferred and included in the list.
For example, if the camera captures half of a person’s body, it will send the information of the points found on that particular half.

Finally, all together...

The last part (ok, this time is the actual ‘last part’) is putting it all together.

The main.py file in our project is in charge of that.

We use argparse to being able to specify parameters from the terminal:

And from there, we wrap up the rest:

So now we are ready to use the program.
On the Terminal, we can execute:

python ./pose_transmitter/main.py --video_source 0 --host_ip 127.0.0.1 --host_port 4900

or

python ./pose_transmitter/main.py --video_source some_video.mp4

or just:

python ./pose_transmitter/main.py --debug

The results are something like:

Notes

Performance: It is not really good. I am getting around 30FPS using my webcam (less than that as you can see in the example) which for this experiment is enough.
In the TensorFlow JS Demo, I am getting 3x times that. Same machine, on the browser, Javascript… You get the idea...

Even in Python, I am pretty sure that there is space for improvement. Maybe the fact that I am extracting the frame, then doing the inference, and then extracting the next frame.
Couldn’t be parallel tasks? Not sure how the Global Interpreter Lock will limit this, but is an interesting exercise that is pending.

—

Finally, the code is in this repo, all ideas, suggestions, and comments are welcome!

>>>[post on mayonesso.com]<<<

So, inspired by learn in public (yep, from that comes the [LiP] tag…) I am here sharing a little adventure that I came up with last week exploring UpWork trying to find something that my zero-rating-zero-stars status could be enough for.

So, looking into Unity stuff I saw a problem in the signing process that someone has faced when building an OSX build.

But the good thing (and a smart move) from the job description was that all the repro steps were there. “Is this Unity version”, “Is this specific plugin”, “These are the repro steps”, “And this is the error that we are getting”.

Digging into it…

While my gf was fighting with her landlord over emails before we went dining out. I downloaded the plugin, set the specific Unity version, and got the same error with the delightful repro steps.

Then I started to look at what could be the error, by the stack trace I saw that some files from the specific plugin that was described (Agora) weren’t signed properly, and the build would be completed if these files from the plugin were removed. So, yes, it was the plugin.

Further investigation, I saw some people having the same issue on the plugin repo, and some people having the same issue on other plugins or files in Stackoverflow it was clear that had something to do with the signature and a re-signature had to be made.

But then, exploring the plugin files I saw two scripts called prep_codesign.sh and signcode.sh ! In a careless move ( 🤞 no rm -rfpls!), I ran the scripts fixing the parameters / ENV variables that were asking me, but no luck.
The error on Xcode was still there and my girlfriend was almost done with the current round. I jumped back to the console and started to read the scripts a little bit more carefully, and then I recognized part of a path that they were using. But the pattern was slightly different. What if… swapped the line to match the structure that I had, ran the scripts, and voila! A green light on XCode!

What next?

This wasn’t a theoretical exercise, but a weird attempt to try to get my first job on UpWork, so I went to the platform, took some screenshots of before/after (I had to create a new project & fix it again), and wrote a “presentation” letter completely different from my other attempts where I try to, somehow with words, convince someone that I actually can do the job at hand.
This time I just said that I was able to reproduce the error, that I found a fix, and that I was attaching screenshots of it.

“Show, don’t tell” in its maximum expression.

The next day I receive a message from the client and we start working on the solution. A few simple steps that in the end I put together in a PostBuild script + a ScriptableObject to set up the required parameters (repo here).

Some lessons learned…

Maybe the most important one is that if you don’t have a reputation in places like UpWork, you will have to work for reduced fees or even for free.
But the point is to do it smartly. Pick a j̵o̵b̵ task that you can finish in a reasonable amount of time and just do it. Before even applying.
Two things could happen a.- once you are done you offer the solution (with proof!), or b.- the job was taken and at least you learned something (hopefully).

The second one, try to achieve a balance in the karmic receive/give thing… Make a post like this, put the solution to Github, share your experience…
I don’t have good arguments on this one, but I think it does more good than harm.

…

If everything goes well I will share more experiences of fixing or doing stuff here, hopefully not only a recipe to fix something but some not-that-well-curated reflexions too!

>>>[post on mayonesso.com]<<<

Firestore is a NoSQL database that is part of the services that Google offers.

In this article, we will create a connection to this database, taking advantage of the fact that (like other Google products) it has a free tier.

The first step is to create a FireStore database in a Google project.

To do this, we visit the Google Cloud Console, and on the Databases category select Firestore.

Here we will create a new database.
We select the native mode:

Assign it a name*, and a zone:

*If we want to be in the free tier, we must leave the name of our database to the default value, that is, (default).

Then we select the “Production Rules” mode.

And that’s it, we now have our db, and we can start creating our collections and documents:

Buuuut, what we really want to do is be able to do these types of operations from a Python script located in an external server.

And for that, we need the following!
1.- Install the dependencies in our project
2.- Obtain a service account to be able to authenticate correctly
3.- Authenticate to Firestore

1.- Install the dependencies in our project

Here we’ll use poetry to install the package firebase-admin:

poetry add firebase-admin

And that is!

2.- Obtain a service account to be able to authenticate correctly

Again, we go to the Google Cloud Console, to the IAM & Admin > Service accounts section and we create a new Service Account:

We fill in the details:

Click in CREATE AND CONTINUE , and after the Service Account is created, we assign the Cloud Datastore User role (It seems to be a legacy name, DataStore instead of Firestore).

Other alternatives that also work are: Firebase Admin SDK Administrator Service Agent or Firebase Develop Admin.
Note that roles like: Firestore Service Agent, or Firestore Editor don’t seem to work.

We left the last section as it is;

Once we click DONE our Service Account is ready to be used!

Now, we have to download the file with the credentials that we will use!

As we already can see our new service account is listed, so we click it to see the details.

Then navigate to Keys, and add a new key in JSON format!

Select JSON type and CREATE.
This will create the key and download it to our computer.

3.- Authenticate to Firestore

We already have our service account and the corresponding key in a JSON file.
The next thing is to use this file to authenticate ourselves and be able to interact with our BDD, here is a sample code.

Important; this JSON file is a password (a key to be more specific) that can present a security risk in the event of a leak or compromise.
It is very important to be careful where we store it, and keep in mind that, since it is a file, it can be mistakenly uploaded to a repository, or to an application that is on the client’s side!

import firebase_admin
from firebase_admin import credentials
from firebase_admin import firestore

cred = credentials.Certificate('./path/to/your_service_account.json')
firebase_admin.initialize_app(cred)
db = firestore.client()


# Write and Append examples:

# Write data with `set`!
data = {"name": "Los Angeles", "state": "CA", "country": "USA"}
db.collection("cities").document("LA").set(data)

# Append data with `merge=True`
db.collection("cities").document("BJ").set({"capital": True}, merge=True)


# Read data with get():
doc_ref = db.collection("cities").document("BJ")
doc = doc_ref.get()
print(doc.to_dict())

And listo!

Useful links:

Google Docs: Authenticate to Compute Engine
Google Docs: Setting up a Service Account
Google Docs: Firestore quickstart

If you’ve ever found yourself repeating a task quite frequently, this article may help you!

Google has a product called Google Run, where you can, in their own words `Run applications fast and securely in a fully managed environment`.

Now, a feature is the possibility of executing scheduled jobs. These are the so-called `Jobs`. If your task is a small script, you could benefit from the free tier.

In this column we will see how to create a `Job` following one of the recommended methods, that is, packaging our code in a Docker image.

First, to work with Google Run Jobs, you need to have one of the following roles:

- Cloud Run Admin: This gives you full permissions to create, update, delete, and run jobs.
- Cloud Run Invoker: This role allows you to run jobs.
- Cloud Run Viewer: It provides the ability to view and list jobs.

Why, you ask? Well, because the necessary permissions include exciting things like:

- run.jobs.{create/update/run/delete/get/list/getIamPolicy/setIamPolicy}
- run.executions.{get/list/delete}
- run.tasks.{get/list}

(Also remember this list if you are creating a custom role)

Next, you’ll need a container image. You can choose one of these types of container images:

- Container images that are stored in the same project as the one you’re creating the job or service in.
- Container images from other Google Cloud projects (assuming you have the right IAM permissions).
- Public container images from Artifact Registry, Container Registry (Deprecated), or the Docker Hub.

But, hold onto your hat, because Google recommends the use of Artifact Registry. So that is what are we going to do!

And then you will be able to select your image and run your code as a scheduled Job.

>> Repository link here.

Creating a container image

So, according to Google, we’re going to use Artifact Registry because it’s the cool kid on the block. As Google puts it, “Artifact Registry provides a single location for storing and managing your packages and Docker container images.”

There are three types of repositories:

- Standard: These are regular repositories for your private artifacts. You upload and download artifacts directly with these repositories and use Artifact Analysis to scan for vulnerabilities and other metadata.
- Remote: This is like a repository that goes on vacation. It acts as a proxy for an external source, like Docker Hub, Maven Central, or the Python Package Index (PyPI). It fetches stuff from an external source and keeps a copy. Lazy but effective!
- Virtual: This is the master of ceremonies. It acts as a single access point to download, install, or deploy artifacts from one or more upstream repositories. Virtual repositories simplify client configuration for consumers of your artifacts.

(Want more details? [Google: Repository Overview])

Although there are different ‘artifact’ options we will focus on creating a Docker artifact in a Standard repository.

For this, we will do the following steps:

0.- Create a new Google Project
1.- Create a private Docker repository in Artifact Registry
2.- Set up authentication (this will allow Docker to push the image to our Artifact Registry in Google)
3.- Push an image to the repository

0.- Create a new Google Project

We will create a new project: cumplo-jobs, make sure Billing is activated, and then activate the `Artifact Registry API`.

Then we will install Docker and Google Cloud CLI.

We initialize the Google Cloud CLI and when prompted, we select the project we just created.

gcloud init

An alternative is to simply specify the project-id we want to work with:

gcloud config set project cumplo-jobs

1.- Create a private Docker repository in Artifact Registry

Run the following command to create a new Docker repository named cumplo-jobs-docker-repo in the location us-central1 with the description "Docker repository for Cumplo Jobs".

gcloud artifacts repositories create cumplo-jobs-docker-repo \
 --repository-format=docker \
 --location=us-central1 \
 --description="Docker repository for Cumplo Jobs"

Which will give us an output similar to this:

Create request issued for: [cumplo-jobs-docker-repo]
Waiting for operation [projects/cumplo-jobs/locations/us-central1/operations/12abc123-f0ab-12d1-a0ff-0c75bf1ef26a] to complete…done.
Created repository [cumplo-jobs-docker-repo].

We can check that the repository has been created with:

gcloud artifacts repositories list

2.- Set up authentication

The first step to being able to push or pull images is to configure Docker so that you can use the Google Cloud CLI to make requests to the Artifact Library.

For that, we execute this command

gcloud auth configure-docker us-central1-docker.pkg.dev

Which will give us an output similar to this:

Adding credentials for: us-central1-docker.pkg.dev
After update, the following will be written to your Docker config file located at [/Users/mayo_nesso/.docker/config.json]:
 {
  "credHelpers": {
    "us-central1-docker.pkg.dev": "gcloud"
  }
}

Do you want to continue (Y/n)?  y

Docker configuration file updated.

This updates our Docker configuration, and we should now be able to push an image!

3.- Push an image to the repository

Google’s tutorial, guides us to download an image from the Docker repository, but in our case, we will use a locally built image to illustrate a more practical example.

3.1.- Our docker image

We have a folder called our_docker_image where we have a Dockerfile and an app.py file, this is enough to illustrate the creation of a Docker image that will be uploaded to the registry.

We go to our_docker_image folder (where our Dockerfile is located), then we will build an image.

In this step, we have to carefully specify the name of our container since it has to have some important details.

The important details are Repository location, Project id, and the Artifact Repository idthat we have been using.

In our example, the name of the image will be simple_image, and the tag just tag1.

So recapitulating we have;

- Repository location: central1
- Project id: cumplo-jobs
- Artifact repository id: cumplo-jobs-docker-repo
- Image Name: simple_image
- tag: tag1

With all that in mind, the build command will be:

cd our_docker_image
docker build -t us-central1-docker.pkg.dev/cumplo-jobs/cumplo-jobs-docker-repo/simple_image:tag1 .

Important : Since Google Cloud Platform requires an AMD image to run it, if you are on Apple silicon (M1, M2,…), then the generated build will not work
(more info in this SO thread)!

To fix that, you will have to follow this steps to create the image!
First, check if you have a builder that can create a linux/amd64 image:

docker buildx ls

Output:

NAME/NODE DRIVER/ENDPOINT STATUS BUILDKIT PLATFORMS
default docker
 default default running v0.11.6+616c3f613b54 linux/arm64, linux/amd64, linux/amd64/v2, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6
desktop-linux * docker
 desktop-linux desktop-linux running v0.11.6+616c3f613b54 linux/arm64, linux/amd64, linux/amd64/v2, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6

If so, we proceed with our _modified_ build command:

cd our_docker_image
docker buildx build --platform linux/amd64 -t us-central1-docker.pkg.dev/cumplo-jobs/cumplo-jobs-docker-repo/simple_image:tag1 .

If we don’t find a builder with the `linux/amd64` capability, we will have to create one (see here that and other options).

3.2.- Pushing our image

Finally, to push the image we use this command:

docker push us-central1-docker.pkg.dev/cumplo-jobs/cumplo-jobs-docker-repo/simple_image:tag1

That will give us something like:

The push refers to repository [us-central1-docker.pkg.dev/cumplo-jobs/cumplo-jobs-docker-repo/simple_image]
13317470e910: Pushed
1fd51346f743: Pushed
a130a3c46754: Pushed
762ff2f0c641: Pushed
a1bf4fc22850: Pushed
f94bda6a158c: Pushed
3ce829cc4970: Pushed
tag1: digest: sha256:c2918dc3efda6123c338669c5a2934f607c40052a8994d3d7b26e23e4b3cf66a size: 1783

Then, we can check on the Google Cloud Console, that the repo, and image are there 😃!

Repository details:

Image details:

Great, we now have our image in the Artifact Repository!

Now, to run the Job, we go to our Google Console, select our project, and go to Cloud Run:

We select `Jobs`:

And there we will create a new Job (`Create Job`):

We select our image previously uploaded to the `Artifact Registry`:

We name our job, and create it (We can select “Execute job immediately” to see what results from the execution)!

After a moment, our Job will be created!

If everything goes well, we should see the green light after our test run:

Job details:

And voila!

Useful links:

• Google Doc: Store Docker container images in Artifact Registry
• Google Doc: Create Jobs Docs
• Docker Doc: Multi-platform images
• Repo to google-job-artifact

Intro: (you can skip this…)

Python Virtual Environments provides an isolated workspace for your projects, allowing you to work with specific Python versions and libraries.

In this small (minimalist) guide, we'll walk you through creating a virtual Python environment using pyenv without the need for additional tools like virtualenv or conda.

What You’ll Need:

Before we dive in, ensure you have the following prerequisites:

• venv: This comes built-in with Python⁰ and allows you to create virtual environments. You can learn more about it in the Python documentation.
• pyenv: Installation instructions on its repository.

Step-by-Step Instructions:

i.- Install your Desired Python Version
First, we will use pyenv to install the Python version that we want to install. For example, if we want to go with 3.8.8:

pyenv install 3.8.8

ii.- Activate the Specific Python Version
Activate the Python version you just installed. This ensures that any subsequent commands will use the desired Python version¹:

pyenv shell 3.8.8

iii.- Create a New Virtual Environment
Now, let’s create a new virtual environment. This environment will be located in a hidden folder² called .virtual_env.
Use the following command³:

python -m venv .virtual_env

iv.- Activate Your New Environment
To work within your new virtual environment, activate it with the following command:

source .virtual_env/bin/activate

v.- (optional) Check the current version
If all worked properly, then we’ll see the desired version when we execute:

python --version

vi.- Deactivate the Virtual Environment
When you’re done working in your virtual environment, deactivate it with:

deactivate

And that’s it! You’ve successfully set up a Python virtual environment using pyenv.
This method gives you precise control over your Python versions without the need for additional tools and with a little better understanding of what is under the hood!

Any comments, improvements, ideas, suggestions, or criticism? leave me a comment.

⁰ We assume that we have at least one version of Python installed.

¹pyenv shell 3.8.8 specifies that 3.8.8 version will be used in the current shell session. Other options are pyenv local 3.8.8, and pyenv global 3.8.8

² The name can be any of our preference, and the same for whether it is a hidden folder or not.

³ If the python command is not found means that the alias is not set, after pyenv shell 3.8.8 this should not be the case, but always you can use python3 instead.