Creating a Swift Runtime Library

I recently released a new Swift library Runtime, you can check it out on Github, which adds the ability to view type info at runtime, get and set values via reflection, and creating instances of objects from it’s type. There was some interest on how it works so I thought I would write about it’s implementation details.

All of the information about memory layout that will be talked about in the article was either acquired from the Swift docs, or found out by simply printing out the bytes and figuring it out manually. For all examples 64 bit is assumed.

Memory Layout

To fully understand how Runtime works a general understanding how memory is laid out in Swift is needed. Swift structs are laid out similar to C. The properties are laid out one after another. Some times it can be a little more complicated. Types, just like in C, like to have an address divisible by their size. So for example an Int likes to be on an address divisible by 8. Due to this there can be padding between the values. The reason behind is performance.

Edit: Qaanol on Reddit pointed out that I incorrectly stated that the properties are laid out in the same order they’re declared which is not always true. The compiler can lay them out in any order to reduce the amount of padding needed.

Take this struct for example:

struct Example {
    var a: Int = 1
    var b: Int = 2
    var c: Int = 3
}
// in memory
0x0100000000000000 // a
0x0200000000000000 // b
0x0300000000000000 // c

Classes are very similar to structs, the main difference being they have a header directly before the the properties. The header contains the type’s isa pointer, and the strong and weak reference counts.

Example:

class Example {
    var a: Int = 1
    var b: Int = 2
    var c: Int = 3
}
// in memory
0x8347230987523408 // isa pointer
0x0200000001000000 // strong and weak reference counts
0x0100000000000000 // a
0x0200000000000000 // b
0x0300000000000000 // c

Acquiring Type Metadata

Swift stores a metadata record for every type. This can include things like its kind (struct, class, enum, etc), field names, field offsets and so on. Getting the record is tricky since there is no API provided. We will start out by attempting to get a pointer to the metadata record. Before we can get the metadata address we first need to have an understanding of how a protocol type is laid out (e.g. MyProtocol.Type.self ).

Protocols types are two pointer sized words (16 bytes). The first word is the address to the underlying type’s metadata record, and the second is the address to the witness table for all required protocol conformances. An Any is a special protocol. Every type implicitly conforms to it. Since it has no required properties or functions it does not contain a witness table. Laid out as a Swift struct:

struct ProtocolType {
    let metadataAddress: Int
    let witnessTable: Int
}
struct AnyProtocol {
    let metadataAddress: Int
}

We can obtain the address by first getting the desired type casted as an Any.Type . Since an Any.Type is just an 8 byte address to the metadata record we can do an unsafeBitCast to an Int. Then use that address to obtain a pointer.

let type: Any.Type = Int.self
let address = unsafeBitCast(type, to: Int.self)
let pointer = UnsafeRawPointer(bitPattern: address)!

At this point we have officially obtained a pointer to a type’s metadata record! However it’s still unreadable because our pointer doesn’t have a type to bind the memory too. This is when the Swift documentation really comes into play. From the documentation we can build the types and start reading the values. For the rest of the article we will be paying attention to struct metadata only.

Struct Metadata Layout

All metadata records have a few common fields which include its kind, and the address to its value witness table. Structs also have a pointer to the type’s nominal type descriptor, a pointer to its parent metadata record (always null), field offset vector, and a generic parameter descriptor.

From the documentation we can build the struct metadata type just as they describe. The whole goal of this is to directly mimic the shape of the metadata record, so we have a type to bind the metadata pointer too. For the sake of example I am going to omit a few fields to keep things light. All we are going to look at is the kind and the nominal type descriptor.

From the docs we know that at offset 0 is the kind and at offset 1 the nominal type descriptor is referenced. The reference is not an address to it, but an offset to the record. In Runtime we have the type RelativePointer. It works by reading the offset then advancing from that value by the specified amount and grabs the value bound to the specified type.

struct StructMetadata {
    var kind: Int
    var ntd: RelativePointer<Int, NominalTypeDescriptor>
}

We can build the nominal type descriptor as well. You will also notice a new type RelativeVectorPointer. It works the same as the RelativePointer but it knows its pointing at a vector, which is similar to an array where it has elements stored contiguously.

struct NominalTypeDescriptor {
    var mangledName: RelativePointer<Int32, CChar>
    var numberOfFields: Int32
    var offsetToTheFieldOffsets: RelativeVectorPointer<Int32, Int>
    var fieldNames: RelativePointer<Int32, CChar>
    var fieldTypeAccessor: RelativePointer<Int32, Int>
}

Now that the metadata type has be created, just like before we can grab a pointer to the type’s metadata record and bind that memory to it.

let type: Any.Type = Int.self
let address = unsafeBitCast(type, to: UInt.self)
let pointer = UnsafePointer<StructMetadata>(bitPattern: address)!
print(pointer.pointee.kind) // prints 1 for struct

Now that the pointer knows what type it is bound too we can start reading the values. Some values like the kind are as simple as just reading an Int, and some like the field types can just much more complicated but all use the same technique. I am going to go over getting the field offsets, and field types to show a varying range of complexity.

Field Offsets

From the docs:

The offset to the field offset vector is stored at offset 3. This is the offset in pointer-sized words of the field offset vector for the type in the metadata record. If no field offset vector is stored in the metadata record, this is zero.

To put a little more clearly, the field offsets are a vector in the metadata record, not the nominal type descriptor. The value in the nominal type descriptor is the offset, in pointer sized words, from the base of the metadata record to the beginning of the field offset vector.

From Runtime NominalMetadataType.swift :

func fieldOffsets() -> [Int] {
    return nominalTypeDescriptor
        .pointee // 1
        .offsetToTheFieldOffsetVector // 2
        .vector(metadata: base, n: numberOfFields()) // 3
}

1. Grab the nominal type descriptor from the StructMetadata .
2. Get the RelativeVectorPointer for the field offsets.
3. From the base of the metadata record, advance by the value specified in the nominal type descriptor to the beginning of the field offset vector and read the value.

Field Types

From the docs:

The field type accessor is a function pointer at offset 5. If non-null, the function takes a pointer to an instance of type metadata for the nominal type, and returns a pointer to an array of type metadata references for the types of the fields of that instance. The order matches that of the field offset vector and field name list.

The documentation is a bit misleading here. The field type accessor is referenced at offset 5. The reference is the offset from the value to the field type accessor function pointer.

Firstly we have to create the C function pointer.

typealias FieldTypeAccessor = 
     @convention(c) (UnsafePointer<Int>) -> UnsafePointer<Int>

Next grab the function from the struct metadata. From Runtime NominalMetadataType.swift :

func fieldTypeAccessor() -> FieldTypeAccessor {
    let function = nominalTypeDescriptor
             .pointee // 1
             .fieldTypeAccessor.advanced() // 2
    return unsafeBitCast(function, to: FieldTypeAccessor.self) // 3
}

1. Get the nominal type descriptor.
2. Get the field type accessor reference, then advance the amount specified and grab a pointer to that value.
3. Since the value is actually a function pointer we need to bit cast the pointer to FieldTypeAccessor.

We have the field type accessor function now. We can use this function to grab the field types. From Runtime NominalMetadataType.swift :

func fieldTypes() -> [Any.Type] {
    let start = fieldTypeAccessor()(base) // 1
    let types = start.vector(n: numberOfFields()) // 2
    return types.map{ unsafeBitCast($0, to: Any.Type.self) } // 3
}

1. Get the field type accessor function, and run it with the metadata record pointer as the parameter. This returns a pointer to the start of the field type vector.
2. Read the vector for the number of fields. This returns an [Int] , where each value is the metadata address for each type.
3. Since an Any.Type is just an 8 byte value containing the address to the type’s metadata record we can iterate through the Int array and bit cast each to an Any.Type.

Reflection

Runtime also has a reflection API allowing you to get and set values dynamically. To do this we need two pieces of information. The type, and offset for the property. The offset is the distance in number of bytes to the property from the base of the object. So at a high level, all that needs to happen is to obtain a pointer to our object, advance to the property, rebind the memory to the property’s type and get or set it.

For an example we will have a struct that has three Int properties a, b, and c. We will set b to 5 using the method described above. Since b is the second Int property in the struct we know that it has an offset of 8.

struct Example {
    var a: Int = 1
    var b: Int = 2
    var c: Int = 3
}

var value = Example(a: 1, b: 2, c: 3)

// 1
withUnsafePointer(to: &value) { pointer in
    // 2
    let base = UnsafeRawPointer(pointer)
    // 3
    let rawBPointer = base.advanced(by: 8)
    // 4
    let b = rawBPointer.bindMemory(to: Int.self, capacity: 1)
    // 5
    let mutable = UnsafeMutablePointer<Int>(mutating: b)
    mutable.pointee = 5
}

print(value.b) // prints 5

1. Grab a pointer to the object.
2. The pointer is pointing to an Example object, so if we try to advance by 8 it will advance 192 bytes since the Example struct has a stride of 24 bytes. Converting it to a raw pointer will allow us to advance one byte at a time.
3. Advance by 8 bytes. Now the pointer is pointing at b, but the pointer is still an UnsafeRawPointer and does not know the type of the value its pointing at.
4. Bind the memory to an Int since b is an Int
5. UnsafePointer is not mutable by default, so we convert it to an UnsafeMutablePointer and set the value to 5.

Conclusion

I hope everyone was able to learn something! I had a lot of fun building the library. If you want to contribute to the project don’t hesitate to send a pull request, open an issue, or ask for help. Thanks for reading!

• Twitter: Wes Wickwire (I’m new, but come say hello!)
• Project Link: https://github.com/wickwirew/Runtime