Go(lang) adventure: flexibly storing data of all sorts as byte arrays

Code image above is just to attract attention

Hearing the praise Go(lang) is getting and seeing the demand for it yours truly decided to learn it. And what better way to learn a programming language than to write a project in it? So this is what my project named DAM (Data Action Model) will be written in.

The project is — at least ideally — intended to provide an infinitely flexible data storage and access solution. Yes, I know, I have yet to document that, so we will skip some details — but DAM will need to operate with data blobs with an unpredictable structure. And so it was a bit of an disappointment to discover that Go is not exactly a language for pointer arithmetic. Pointers do exist and can be used but Go is a strictly typed language where the type of data a pointer is associated with it vigorously tracked and no frivolous modification of pointer’s value or its type assciation is permitted.

Let us use the Go Playground (a very convenient resource indeed) to see what options are available to a developer who needs to be able to handle data flexibly.

First let’s try to see if we can just assign pointer values to each other to access data directly. Let’s try to run the code below in the Playground.

package main
import "fmt"
func main(){
    var byte_arr []byte = []byte{'a', 'b', 'c', 'd'}
    var b_ptr *[]byte = &byte_arr
    // Trying to transfer a 4-byte arr into a 32-bit unsigned integer.
    var i0 uint32
    var i0_ptr *uint32
    i0_ptr = b_ptr
    i0 = *i0_ptr
    fmt.Printf("i0 = %x\n", i0)
}

Predictably, we got the following result:

./prog.go:13:16: cannot use b_ptr (type *[]byte) as type *uint32 in assignment
Go build failed.

Having encountered this issue yours truly decided to look for other options. It turned out the concept of union (as in C/C++) does not exist in Go. Then I thought of using an equivalent of a void pointer, i.e., universal pointer. That too seems not to be an option in Go.

Go offers a concept of interface which is a lot like interface in other languages such as Java — a description of data and methods and functions to be implemented as appropriate for various data types. This in a way implements overloading, too. Go also provides a concept of empty interface:

interface{}

A variable defined as an empty interface is something you can assign any value too. But does it work in reverse — even when the types are size-wise compatible?

package main
import (
    "fmt"
)
func main() {
    var ei0 interface{}
    var ei1 interface{}
    var ui32_0 uint32 = 165
    var fl32_0 float32 = 0.134
    var fl32_1 float32
    ei0 = ui32_0
    ei1 = fl32_0
    fl32_1 = ei0
    fmt.Printf("fl32_1 = %f\n", fl32_1)
}

Running the above yields the following:

./prog.go:16:9: cannot use ei0 (type interface {}) as type float32 in assignment: need type assertion
Go build failed.

So an attempt to map a 4 byte integer into a 4 byte float by way of an empty interface failed. Let’s see if a type conversion is going to work.

package main
import (
    "fmt"
)
func main() {
    var ei0 interface{}
    var ei1 interface{}
    var ui32_0 uint32 = 165
    var fl32_0 float32 = 0.134
    var fl32_1 float32
    ei0 = ui32_0
    ei1 = fl32_0
    fl32_1 = float32(ei0)
    fmt.Printf("fl32_1 = %f\n", fl32_1)
}

The result:

./prog.go:16:18: cannot convert ei0 (type interface {}) to type float32: need type assertion
Go build failed.

So this doesn’t work either. Yes, Go tracks types carefully! So pointer-based mapping cross types is not an option.

So what one ends up having to use is arrays as a way of reference and math functions for mapping float data types.

Let’s now take a look at how arrays can be manipulated by reference.

package main
import (
    "fmt"
)
func byte_arr_modify(a *[4]byte, position int, new_value byte){
    a[position] = new_value
}
func main() {
    var b_arr [4]byte
    b_arr[0] = 'a'
    b_arr[1] = 'b'
    b_arr[2] = 'c'
    b_arr[3] = 'd'
    fmt.Printf("prior to modification: b_arr = %s\n", b_arr)
    byte_arr_modify(&b_arr, 2, 'S')
    fmt.Printf("after modification: b_arr = %s\n", b_arr)
}

Note the array b_arr is passed by reference. Now let us review the output:

prior to modification: b_arr = abcd
after modification: b_arr = abSd

So here we have a way to modify any byte in a byte array any way we see fit. You can also use bitwise operators to modify any integer type.

For example, here we can use same 4 byte array to represent a 32 bit integer and manipulate it.

package main
import (
   "fmt"
)
func byte_arr_modify(a *[4]byte, incrementby uint32){
    var int_value uint32 = 0
    int_value += uint32((*a)[0])
    int_value += uint32((*a)[1]) << 8
    int_value += uint32((*a)[2]) << 16
    int_value += uint32((*a)[3]) << 24
    int_value += incrementby
    (*a)[0] = byte(int_value & 0x000000ff)
    (*a)[1] = byte((int_value >> 8) & 0x000000ff)
    (*a)[2] = byte((int_value >> 16) & 0x000000ff)
    (*a)[3] = byte((int_value >> 24) & 0x000000ff)
}
func main() {
    var b_arr [4]byte
    b_arr[0] = 'a'
    b_arr[1] = 'b'
    b_arr[2] = 'c'
    b_arr[3] = 'd'
    fmt.Printf("prior to modification: b_arr = %s\n", b_arr)
    byte_arr_modify(&b_arr, 256)
    fmt.Printf("after modification: b_arr = %s\n", b_arr)
}

Running the code above yields us this:

prior to modification: b_arr = abcd
after modification: b_arr = accd

As we can see, the code acted as expected treating the 4 byte array as a representation of a 32 bit unsigned integer in the little-endian addressing model — we added 256 to it and it advanced the second byte position by 1.

Now let us have some fun with floats stored as bytes.

package main
import  "fmt"
import "math"
func main(){
    var b0 byte = 0x08
    var b1 byte = 0xAA
    var b2 byte = 0x11
    var b3 byte = 0x22
    var i1 uint32
    var i2, i3 uint32
    var f1 float32
    var f2 float32
    var f3 float32
    i1 = (uint32(b1) << 8 | uint32(b0)) | (uint32(b2) << 16) | (uint32(b3) << 24)
    f1 = (float32)(i1)
    fmt.Printf("i1 = %x\n", i1) 
    fmt.Printf("f1 = %e\n", f1)
    f2 = math.Float32frombits(i1)
    f3 = math.Float32frombits(i1 ^ 0x80000000)
    i2 = math.Float32bits(f2)
    i3 = math.Float32bits(f3)
    fmt.Printf("f2 = %e\nf3 = %e\ni2 = %x\ni3 = %x\n", f2, f3, i2, i3)
}

Output:

i1 = 2211aa08
f1 = 5.715830e+08
f2 = 1.974118e-18
f3 = -1.974118e-18
i2 = 2211aa08
i3 = a211aa08

So as you can see by using functions math.Float32frombits and math.Float32bits one can convert a 32 bit integer to a float and vice versa. The conversion works back and forth properly. I threw in a binary bit flip to switch the sign for the value of a float encoded as an integer and that worked too.

Methods presented above allow one to store and access all basic data types in a byte array in Go.

References

Go

Go: fmt package

Go Math package

The Go Playground

Why should you leverage Go programming language
Rutva Safi, Softweb Solutions Inc., 13 October 2020

DAM (GitHub)

Endianess
freeCodeCamp

Social media links

Locals

Gab

Minds

Gettr

Facebook

Website

borisepstein.info

Support

Subscribestar

Patreon

Originally published here on 4 October 2021.