Go store some strings

For quite some time now, I’ve been hearing from inexperienced Go programmers (as well as some experienced ones) how byte-slices are the right way to deal with string-like data in Go, especially if you’re performing a lot of append/slice operations.
Most Go programmers, like me, have at least some experience with C and may find themselves, consciously or subconsciously comparing concepts from Go to their analogs in C. The assertion that byte slices are better than strings in Go probably maps onto the comparison between C strings, which are just null-terminated char arrays, with the std::string type in C++ STL, which still represents strings as char arrays in memory (how else would you do it?) but also wraps that “bare-bones” logic in a C++ class, abstracting allocation, concatenation etc. Because the total size of the dataset is stored in the std::string object, it doesn’t need to be null-terminated (but still is). You could also convert between std::string and C strings easily. For the former, calling c_str() method on a std::string object returns a char*. The std::string constructor handles the inverse of that.
For me, it all started at the university. I took CS201 — Introduction to Computer Programming, which was essentially a weeks of C basics and then a very confusing, extremely frustrating jump to C++ and the object-oriented paradigm. In the following years, we had to learn embedded programming in C and that only caused further confusion. I ended up knowing-but-not-really the difference between C and C++. Especially when it came to strings, the professors and the TAs warned us: “You’d do well to never use std::string because C works differently, and C is what you’ll be working with. Also, C strings are better anyway.” That last part was never really defended, at least not by anyone at my university. It was just accepted.
It did make some intuitive sense. The “overhead” from all the logic embedded in the std::string class seems entirely unnecessary, especially when you’re writing code for tiny CPUs with 20 MHz clock and 256 kB RAM, and you don’t know what that “overhead” is. Also, who in their right minds would use C++ for embedded applications? (A lot of people, apparently, but that’s another discussion).
So how does all that compare with Go. In some ways, Go’s implementation of string is pretty similar to C++’s std::string. When you instantiate a string in Go, if the string isn’t empty, an array is allocated for it somewhere on the heap. But you won’t be dealing with that array directly. Instead, what you get is a “string-header”. It’s the string-header that you’ll be passing around by value, instead of the underlying array. The mechanism is similar to std::string in that when you allocate a std::string on the stack, the underlying char array is internally allocated on the heap, and the std::string can be passed around safely. Like std::string the Go string also carries the length property around with it.
C strings, on the other hand, need to be allocated by you. You can allocate it on the stack, as a fixed-size array. Or you can make a call to something like malloc() that will allocate memory on the heap and return a pointer to that memory to you. It’s your responsibility to either keep track of the size of your string and/or ensure that it’s terminated with a null character.
There are some handy functions in the standard library that will allow you to perform some operations on the string. But there’s nothing fancy to the way they perform operations. It’s almost always just loops and comparisons. Even strlen() would just loop over the char array until it finds a null byte, and that there is your string’s length. If you don’t null-terminate your string, strlen() doesn’t care. It’ll continue accessing memory even outside your allocated range, until either a null char is found, or a segmentation fault occurs. Also, you gotta free the memory if you allocated it dynamically. There’s a lot of responsibility. Little mistakes can cause huge problems!
Regardless, there’s something majestic about the simplicity of C strings. There’s almost zero overhead. All you need to have, to access the underlying data, is a single word pointer on the stack. From thereon, it’s just arrays and loops! The efficiency and room for optimization makes this scheme ideal for everything that needs to run fast with low memory. (Which, in my opinion, is everything. Better or more hardware doesn’t justify tolerance of bad programming. But that’s another discussion.)
“So, string in Go works like std::string in C++, yeah? And C strings are better than std::string, you say? And byte-slices are analogous to char arrays. So []byte is like a C string? Better than string, right? No overhead…”
…Let me stop you there. You couldn’t be more wrong.
In Go, all slices work in the same way as strings. When you instantiate a byte slice, you don’t get the pointer to allocated memory on heap. Instead, you get a “slice-header”. The slice-header is what you pass around! In fact, it’s not the slice-header that’s modeled after string storage in Go, strings are modeled after slices!
So essentially, it’s Go slices that work like std::string in that, the values you work with are not the pointers to allocated memory, but structs (objects in case of C++) wrapping those pointers.
The only difference that a string has from a[]byte is that in Go, strings are immutable. Which means you can’t do this:
package mainfunc main() {
str := "Hello, world!" str = str + " I'm Zia" // this is ok (concatenation)
_ = str[:5] // this too (substring)
_ = str[0] == 'H' // and this (access by index) str[0] = 'G' // but this is NOT ok
}// Run in Go Playground
The above code will generate a compile-time error on the last statement. Because str is a string it’s immutable. Once it has been stored, you cannot modify the contents of the underlying array. You can access its contents by indexing as you would for a normal array. You can also perform operations on it, like take a substring out of it, or concatenate it with other strings. But you cannot change its contents directly.
That is what differentiates a string from a byte-slice in Go. So let’s check out slices.
package mainfunc main() {
b := []byte("Hello, world!")
// you can't do this;
// + operator works only with strings
b = b + []byte(" I'm Zia")
// this is how you concatenate byte-slices
b = append(b, []byte(" I'm Zia")...)
_ = b[:5] // this is ok
_ = b[0] == 'H' // this too
b[0] = 'J' // and this is ok too
}// Run in Go Playground
So the downside of using byte slices is you cannot use the ‘+’ operator to concatenate them. The upside is that you can directly modify the contents of a slice (not just a byte-slice, any type of slice). But that’s how both strings and byte-slices work in Go. But none of that makes byte-slices better, or worse. (And I’ll argue that they’re exactly that, not better — but not worse either.)
The common assertion is that string operations involve an “overhead”. Don’t repeat my mistakes and take that statement without knowing exactly what that “overhead” is…
So one operation that does have overhead is converting a string to a byte-slice. Because a string is immutable, when you “cast” it to a byte slice, the go runtime will copy the underlying array of the string to a new location on the heap. It’ll then wrap a pointer to that new location in a slice-header, and voila, you’ve got yourself a byte-slice representation of the same string. The “overhead” is the time it takes to copy the string data to a new location, and that now there exist 2 copies of the same data in memory. Let’s see:
package mainimport (
"fmt"
"reflect"
"unsafe"
)func main() {
s := "Hello, world!" // string header
b := []byte(s) // byte header // to actually see what these headers contain, we cast
// the pointer to string or slice to unsafe.Pointer,
// then cast the Pointer to a StringHeader pointer, then
// copy off its value
shead := *(*reflect.StringHeader)(unsafe.Pointer(&s))
bhead := *(*reflect.SliceHeader)(unsafe.Pointer(&b)) fmt.Printf("shead = %+v\n", shead)
fmt.Printf("bhead = %+v\n", bhead)
}// Run it in Go Playground
You might observe in the output, that the Data fields of both the StringHeader as well as the SliceHeader point to different addresses in the memory.
So when the statements := "Hello, world!" executes, it puts a byte array containing {'H', 'e', 'l', 'l', 'o', ' ', 'w', 'o', 'r', 'l', 'd'} onto some memory address, then constructs a StringHeader, the .Data property of which contains the address of the memory location where our array exists, and the .Len property contains the total size of our dataset. The string s is actually that very StringHeader (not a byte* to the location in the memory where the string is stored.
When the statement b := []byte(s) runs, the array containing the characters is copied onto a new memory address. A SliceHeader is then constructed, which contains the address of the location where the data was just copied to as SliceHeader.Data, and the length of the data as SliceHeader.Len. But the SliceHeader also has another field called .Cap. That’s the actual size that’s allocated for this slice. This will be higher than, or equal to the .Len. Thing is, Go slices are designed to be extended frequently. To avoid having to reallocate every time a slice is stretched, the slice has a “capacity”. So let’s say, the slice length is 13, but the capacity is 32. That means that the actual number of bytes allocated for the slice are 32, only 13 of which are currently being used. If I decide to append another byte-slice of length 8, the new bytes will be just copied in front of the slice, and .Len property of the slice header will be updated to 21. I can append up to (32–13=) 19 bytes in front of the current slice. Now this capacity isn’t some hard limit. If I append() more than 19 bytes, the slice will be reallocated to a higher capacity, and SliceHeader.Cap will be updated. The builtin cap() and len() functions return the capacity and length of a slice, respectively. This “capacity” feature of slices, as we’ll see later, also makes byte-slices more desirable than strings, in some cases. In other cases, that’s what makes byte-slices undesirable. (The case I’m trying to make here is that byte-slices are neither better nor any worse than strings. They’re just different, suited to different applications.)
So the “overhead” we were talking about, lies in the time it took to copy the string data from *StringHeader.Data over to *SliceHeader.Data (or vice-versa) and the fact that after the cast, these duplicate each other in memory until one of them is garbage collected.
But why would someone cast a string to a slice-header and vice versa? Well, because sometimes a function (even a standard library one) will return a string value, but you need to modify the contents of that string to make it workable for your purposes. There will be times when you’ll want to convert between byte-slices and strings, but it would serve you well to limit these instances, especially as the datasets become larger, because converting between byte-slices and strings does carry a significant overhead (precisely, O(N) both temporally and spatially).
But what about string operations that don’t cast from strings to byte-slices or vice-versa? Let’s see.
package mainimport (
"fmt"
"reflect"
"unsafe"
)func main() {
str0 := "Hello, world!"
str1 := str0[7:] // no bytes were copied in str1 := str0[7:]
// str1 is just a new StringHeader that points to
// the location (StrHeader(&str0).Data + 7)
fmt.Printf("str0 header: %+v\n", StrHeader(&str0))
fmt.Printf("str1 header: %+v\n", StrHeader(&str1)) // concat needs reallocation and copying len(str0)
// bytes from StrHeader(&str0).Data
str0 += " I'm Zia"
fmt.Printf("str0 header after concat: %+v\n",
StrHeader(&str0)) bts0 := []byte(str0)
bts1 := bts0[7:] // no bytes copied in bts1 := bts0[7:]
// SlcHeader(&bts1).Data == (SlcHeader(&bts0).Data + 7)
fmt.Printf("bts0 header: %+v\n",
SlcHeader(&bts0))
fmt.Printf("bts1 header: %+v\n",
SlcHeader(&bts1)) // appended bytes fall within cap
// no realloc necessary
bts0 = append(bts0, []byte(" I'm Zia")...)
fmt.Printf("bts0 header after concat: %+v\n",
SlcHeader(&bts0)) // appended bytes overflow cap(bts0)
// reallocation may be needed
bts0 = append(bts0, []byte(`
Lorem ipsum dolor sit amet, consectetur adipiscing elit.
Curabitur quis porta elit.
Praesent eget ante ac tortor consequat cursus.
Cras vitae fermentum libero.
`)...) // bts0 header may point to a different location now
fmt.Printf("bts0 header after cap overflow: %+v\n",
SlcHeader(&bts0))
}// StrHeader reads the given string pointer as StringHeader
// and returns the value
func StrHeader(s *string) reflect.StringHeader {
return *(*reflect.StringHeader)(unsafe.Pointer(s))
}// SlcHeader reads the given []byte pointer as SliceHeader
// and returns the value
func SlcHeader(b *[]byte) reflect.SliceHeader {
return *(*reflect.SliceHeader)(unsafe.Pointer(b))
}// Run in Go Playground
When you’re extracting substrings/sub-slices, e.g. str1 := str0[:7] or bts1 := bts0[:7] there’s no overhead. This is true for both strings and byte-slices (or any slices). What you get is just a new string- or slice-header that points to an address within the originally allocated array of the previous string- or slice-header.
When concatenating (or appending, in case of byte-slices) strings will be reallocated and copied before new data is appended. What you get is a new string-header, which points to the new address, with a new length (length of previous string + length of appended string). Byte-slices may need reallocation, depending on the cap() of the original byte-slice (you can go up and read about slice capacity if you missed it).
So, we know this:
- Strings are immutable, byte-slices are not. To modify a string, you may have to cast it to a byte-slice, which does have some overhead.
- When extracting a portion of a byte-slice or string, there’s no overhead. The new header will just point to an address corresponding to the index inside the original array. The new header will have different length (duh!) and may also have different capacity (in case of byte-slices).
- When data is appended to a string, the original string is reallocated, so there is overhead. When data is appended to a byte-slice, the byte-slice will be reallocated only if its capacity needs to be increased. So there may be overhead.
- When appending data to a string, overhead is unavoidable. When appending data to byte-slices, overhead may be avoided by planning ahead, and instantiating the byte slice with
make([]byte, len, cap)to a capacity large enough so as to minimize reallocation.
For Go, a good rule of thumb is, when in doubt, take cue from the standard library. The source is easily available, highly readable and can be browsed here. The standard library seems to use strings where the data may not need to be, or must not be modified. For example, http.Request.Header.Get() accepts a string argument (header name) and returns a string value (header value). Most functions that deal with filenames or paths (e.g. os.Open()) accept strings. Byte-slices are used when dealing with data that needs to be read, written or is otherwise rapidly changing (e.g. http.ResponseWriter.Write() or for that matter, all derivatives of the Reader or Writer interfaces use byte-slices).
Also, the standard library has a couple of great packages for working with both strings and byte-slices, namely strings and bytes.
There is a marvelous post titled “Go Slices: usage and internals” over at the Go official blog. It explains the internal mechanics of slices way better than I ever could. There’s also another equally awesome post titled: “Arrays, slices (and strings): The mechanics of ‘append’”.
One last thing that I want to cover: you might be tempted to do something like this:
package mainimport (
"fmt"
"reflect"
"unsafe"
)func main() {
str := "Hello, world!"
strheader := *(*reflect.StringHeader)(unsafe.Pointer(&str)) // create a new zero-value slice header
bts := make([]byte, 0, 0) // set btsheader.Data to strheader.Data
(*(*reflect.SliceHeader)(unsafe.Pointer(&bts))).Data = strheader.Data // set btsheader.Len and .Cap to strheader.Len
(*(*reflect.SliceHeader)(unsafe.Pointer(&bts))).Len = strheader.Len
(*(*reflect.SliceHeader)(unsafe.Pointer(&bts))).Cap = strheader.Len btsheader := *(*reflect.SliceHeader)(unsafe.Pointer(&bts))
fmt.Printf("strheader: %+v\t\tstr: %s\n", strheader, str)
fmt.Printf("btsheader: %+v\tbts: %s\n", btsheader, bts) bts[0] = 'J' // RUNTIME ERROR
fmt.Printf("%s\n", bts)
}// Run in Go Playground
But this won’t work. Go has a separate sort of read-only memory areas for string allocation. Even if you can get a slice-header to point at the memory where string data exists, if you try to change the data there, the runtime won’t let you.
But let’s imagine it weren’t this way. That the immutability of strings was just a compile-time feature, and you were able to do some pointer magic to cast a string-header to a slice-header without having to copy data or reallocate memory. Would that be so wrong?
Yes, yes it would. The thing is, Go is a garbage collected language. Which means, unlike in C where you have to free() your dynamically allocated memory, Go runtime will periodically automatically free memory that’s no longer in use. To accomplish that feat, the runtime has to keep a table of how much data was allocated, and where.
Say you create a string inside a function. So the memory is allocated for that string on the heap, and the string-header exists on the stack of your current function. You then go ahead and cast the string-header to a slice-header, do something else, and at the end, return the byte-slice (or the slice-header by value) that you think points to the same memory location on the heap where your string data was placed. Once the stack of that function unravels, the Go garbage collector notices that a string-header that existed on that stack won’t be used anymore. It goes and frees the memory allocated on the heap for that string-header. Some new strings are created by code in the current function. Go reuses the memory that was previously being used for your now non-existent string. And in a few cycles, you go and try to access the same memory through that evil, evil little slice-header of yours. That is the definition of a memory leak!
unsafe package is named so for a very good reason. It’s unsafe. Use only when absolutely necessary, and use sparingly. Do not make it a habit to cast pointers willy-nilly. Let the type safeguards and the garbage collector work for you, not against you. And if you need extra control at the expense of added responsibility, you could always go back to programming in C.