Apple’s Metal API tutorial (part 3— Textures)
Hello everyone! Last time, we learnt how to render a triangle to the screen. In this tutorial, we will cover creating textures and also how to organize our vertices better.
Vertex attributes
Right now, we are reading the vertex data in the shader directly from the buffer. This is okay, but if we would, for instance, like to split positions and colors into separate buffers, we would have to change our shader as well. This is not ideal, and Metal fortunately gives us a tool to improve this. It's called vertex descriptor.
Vertex descriptor
This descriptor tells Metal how to interpret the data contained within the vertex buffer(s). We can start by removing the buffer in our vertex shader and using attributes instead:
...
struct Vertex {
float2 position [[attribute(0)]];
float3 color [[attribute(1)]];
};
...
vertex VertexOut vertexFunction(Vertex in [[stage_in]]) {
VertexOut out;
out.position = float4(in.position, 0.0, 1.0);
out.color = in.color;
return out;
}
...Metal will take care of supplying individual vertex data to the vertex shader, so we don't need the vertex id input for indexing anymore.
But things have also changes on the CPU side. We start by creating a vertex descriptor:
class Renderer: NSObject, MTKViewDelegate {
...
var vertexDescriptor: MTLVertexDescriptor
...
init?(metalKitView: MTKView) {
//Device and command queue
self.device = metalKitView.device!
self.commandQueue = self.device.makeCommandQueue()!
//Vertex descriptor
vertexDescriptor = MTLVertexDescriptor()
vertexDescriptor.layouts[30].stride = MemoryLayout<Vertex>.stride
vertexDescriptor.layouts[30].stepRate = 1
vertexDescriptor.layouts[30].stepFunction = MTLVertexStepFunction.perVertex
vertexDescriptor.attributes[0].format = MTLVertexFormat.float2
vertexDescriptor.attributes[0].offset = MemoryLayout.offset(of: \Vertex.position)!
vertexDescriptor.attributes[0].bufferIndex = 30
vertexDescriptor.attributes[1].format = MTLVertexFormat.float3
vertexDescriptor.attributes[1].offset = MemoryLayout.offset(of: \Vertex.color)!
vertexDescriptor.attributes[1].bufferIndex = 30
}
}First, we access the layout at index 30. Index 30 is the highest buffer index, and I usually use it for vertex buffers so that I can have the 0th slot free for other buffers. We have to set its stride, which is the size of a single vertex in bytes. The step rate of 1 means we want to use every single vertex (we go 1 by 1). We also want to progress on per vertex basis, hence the MTLVertexStepFunction.perVertex.
Now to the attributes. We use the same index as in shader (0 for position, 1 for color). The offset is a bit more interesting. It's the offset of the field inside of the structure, which can be obtained by the MemoryLayout.offset function. The buffer index is once again the slot to which the vertex buffer will be bound. Let's tell the render pipeline state to use this vertex descriptor:
init?(metalKitView: MTKView) {
...
//Render pipeline descriptor
var renderPipelineStateDescriptor = MTLRenderPipelineDescriptor()
renderPipelineStateDescriptor.vertexFunction = vertexFunction
renderPipelineStateDescriptor.fragmentFunction = fragmentFunction
renderPipelineStateDescriptor.vertexDescriptor = vertexDescriptor
...
}And the final step is to change the binding slot of the vertex buffer:
...
//Bind vertex buffer
renderEncoder.setVertexBuffer(vertexBuffer, offset: 0, index: 30)
...You should still see the same good old triangle.
Index buffers
Until now, we have been rendering a triangle. Let's try to render a quad instead:
...
let vertices: [Vertex] = [
Vertex(position: simd_float2(-0.5, -0.5), color: simd_float3(1.0, 0.0, 0.0)), //vertex 0
Vertex(position: simd_float2( 0.5, -0.5), color: simd_float3(0.0, 1.0, 0.0)), //vertex 1
Vertex(position: simd_float2( 0.5, 0.5), color: simd_float3(0.0, 0.0, 1.0)), //vertex 2
Vertex(position: simd_float2(-0.5, -0.5), color: simd_float3(1.0, 0.0, 0.0)), //vertex 0
Vertex(position: simd_float2( 0.5, 0.5), color: simd_float3(0.0, 0.0, 1.0)), //vertex 2
Vertex(position: simd_float2(-0.5, 0.5), color: simd_float3(1.0, 0.0, 1.0)) //vertex 3
]
...And change the vertex count to 6:
...
//Render
renderEncoder.drawPrimitives(type: MTLPrimitiveType.triangle, vertexStart: 0, vertexCount: 6)
...The code works fine, and we get a quad on screen, but we have unnecessary duplicate data in the vertex buffer, namely the 0th and 2nd vertices. Metal has a feature called index buffer which enables us to only use 4 vertices for the quad. How does it work? We will create a second buffer (index buffer) which will contain six indices that Metal will use to index into the vertex buffer. This not only reduces the amount of data we need to store, but the vertex shader only needs to run 4 times instead of 6.
Let's take a look at how we can create such an index buffer:
class Renderer: NSObject, MTKViewDelegate {
...
var vertexBuffer: MTLBuffer
var indexBuffer: MTLBuffer
...
init?(metalKitView: MTKView) {
...
//Create index buffer
let indices: [ushort] = [
0, 1, 2,
0, 2, 3
]
self.indexBuffer = self.device.makeBuffer(bytes: indices, length: indices.count * MemoryLayout.stride(ofValue: indices[0]), options: MTLResourceOptions.storageModeShared)!
...
}
}The creation of the index buffer is very similar to the vertex buffer. We use 16-bit unsigned integer for the indices, but you would most likely want to use 32-bits when rendering more complex geometry. We can also get rid of the duplicate vertices in our vertex buffer:
...
let vertices: [Vertex] = [
Vertex(position: simd_float2(-0.5, -0.5), color: simd_float3(1.0, 0.0, 0.0)), //vertex 0
Vertex(position: simd_float2( 0.5, -0.5), color: simd_float3(0.0, 1.0, 0.0)), //vertex 1
Vertex(position: simd_float2( 0.5, 0.5), color: simd_float3(0.0, 0.0, 1.0)), //vertex 2
Vertex(position: simd_float2(-0.5, 0.5), color: simd_float3(1.0, 0.0, 1.0)) //vertex 3
]
...All that is left now is to tell the render command encoder to use this index buffer when rendering:
func draw(in view: MTKView) {
...
//Render
renderEncoder.drawIndexedPrimitives(type: MTLPrimitiveType.triangle, indexCount: 6, indexType: MTLIndexType.uint16, indexBuffer: indexBuffer, indexBufferOffset: 0)
...
}We use a different method (drawIndexedPrimitives) when drawing with index buffer. We also have to tell Metal what data type do the indices have. If you run the project, you should see a quad instead of a triangle:
Textures
Textures in Metal aren't that different from buffers. They are objects backed by memory, they can be read from and written to. We have actually encountered textures when rendering to the current drawable of the view, since that's a texture as well. But we are going to look at different usage of textures: loading an image from disk and displaying it in the fragment shader.
I will use the Metal logo as an image:
Adding the image to workspace
You need to add the image to your project by dragging it to the workspace:
To make sure everything went successfully, open project settings, click on Build Phases -> Copy Bundle Resources and see if you can find the image.
Loading the image
MetalKit, a utility library for Metal, can load this image from disk and create a texture from it. I know it might be better to create the texture from scratch as well as load the image for educational purposes, but MetalKit provides easy way to do this on both macOS and iOS. The code for creating the texture is relatively simple:
class Renderer: NSObject, MTKViewDelegate {
...
var colorTexture: MTLTexture?
init?(metalKitView: MTKView) {
...
let loader = MTKTextureLoader(device: self.device)
do {
let url = Bundle.main.url(forResource: "metal_logo", withExtension: "png")
self.colorTexture = try loader.newTexture(URL: url!, options: nil)
} catch {
print("Failed to load image 'metal_logo.png'")
}
...
}
}Metal frame capture
If you run the application, you should see no change, but there is a way to check if the texture has been created successfully, and that is Metal frame capture. We can enable it by holding Option and right-clicking on the run button. Go to Run -> Options and under GPU Frame Capture select Metal.
Press Run. You should see a small Metal icon in the lower part of the editor.
Click on it and press Capture. Wait for a few seconds until it loads and click on Show memory. You should see the texture among the other resources.
Metal frame capture is a powerful tool which gives you a lot of insights on where the error happened or where are performance bottlenecks.
Texture coordinates
In order to be able to read the texture in the fragment shader, we need to know at what coordinates we want to read it, and these coordinates are called texture coordinates. They are in normalized range from 0 to 1.
We are going to pass this information for each vertex and it will then get interpolated. Let's replace the color attribute with texture coordinates:
...
struct Vertex {
var position: simd_float2
var texCoord: simd_float2
}
class Renderer: NSObject, MTKViewDelegate {
...
init?(metalKitView: MTKView) {
...
//Create vertex buffer
let vertices: [Vertex] = [
Vertex(position: simd_float2(-0.5, -0.5), texCoord: simd_float2(0.0, 1.0)), //vertex 0
Vertex(position: simd_float2( 0.5, -0.5), texCoord: simd_float2(1.0, 1.0)), //vertex 1
Vertex(position: simd_float2( 0.5, 0.5), texCoord: simd_float2(1.0, 0.0)), //vertex 2
Vertex(position: simd_float2(-0.5, 0.5), texCoord: simd_float2(0.0, 0.0)) //vertex 3
]
...
}
}We should also update our shader:
...
struct Vertex {
...
float2 texCoord [[attribute(1)]];
};
struct VertexOut {
...
float2 texCoord;
};
vertex VertexOut vertexFunction(Vertex in [[stage_in]]) {
...
out.texCoord = in.texCoord;
...
}
fragment float4 fragmentFunction(VertexOut in [[stage_in]]) {
return float4(in.texCoord, 0.0, 1.0);
}The only thing that is left is to bind the texture to our fragment shader and read it. Just like with buffers, textures in shaders are defined using an attribute:
...
fragment float4 fragmentFunction(VertexOut in [[stage_in]], texture2d<float> colorTexture [[texture(0)]]) {
...
}The type texture2d expects a template argument which indicates what data type does the texture data have when read. This is float in most cases, but there is the option to specify a different data type.
There are 2 ways to read a texture:
- Read the data directly
- Read it using a sampler
We will use the second way, but let's first discuss what a sampler is. Sampler is an object which can sample the texture at a coordinate with some customizable behavior. For instance, we can specify what happens when the coordinates are out of range. In this case, we have specified that we want to use linear filter. There are 2 main types of filters:
- Linear — The value is interpolated linearly between the closest 4 pixels
- Nearest — The value is equal to the value of the nearest pixel
You can see the difference between the 2 filters in the image:
Nearest is best suited for pixel art games or if you want to improve performance, while linear tends to give more realistic results. So let's create a sampler with linear filtering and sample the texture:
...
fragment float4 fragmentFunction(VertexOut in [[stage_in]], texture2d<float> colorTexture [[texture(0)]]) {
constexpr sampler colorSampler(mip_filter::linear, mag_filter::linear, min_filter::linear);
float4 color = colorTexture.sample(colorSampler, in.texCoord);
return float4(color.rgb, 1.0);
}And bind the texture before rendering:
func draw(in view: MTKView) {
...
//Bind color texture
renderEncoder.setFragmentTexture(colorTexture, index: 0)
...
}If you run the code, you should see the texture in all it's beauty:
The reason for the white border is that the alpha in those places is 0, so it was never meant to be seen.
Summary
Anyway, that's it for this tutorial. Hope you found it useful. In the next one, we are finally going to extend our quad into a 3D space using matrices. Cheers!
Source code: https://github.com/SamoZ256/MetalTutorial
