Generating Prefixed Base64 ID’s In Golang

Currently I’m working on a personal project and I’ve learnt that using incrementing ID’s for my entities poses a security risk. I’m not going to go into detail in this post about why it’s a security risk but if you’re interested you should read about the the Moonpig Bug and hopefully you’ll be able to see the flaws caused by incrementing ID’s. If you don’t want to read the article I highly recommend this video by Tom Scott which talks about the same bug.

Instead of using prefixed incrementing ID’s for my entities e.g USR_001, MET_001 I’ve decided to use randomly generated base64 encoded strings and add a prefix to each ID. In the rest of this post I will demonstrate how you can do this using Golang.

If you don’t know what base64 is you can read about it here and it’s what YouTube use for their video ID’s.

Step One— Generating The ID

The first thing we need to do is write a function that will generate a unique ID for us.

// Generate a random base64 url encoded string
func generateBase64ID(size int) (string, error) {
    // First create a slice of bytes
b := make([]byte, size)
    // Read size number of bytes into b 
_, err := rand.Read(b)
if err != nil {
return "", err
}
    // Encode our bytes as a base64 encoded string using URLEncoding
encoded := base64.URLEncoding.EncodeToString(b)
    return encoded, nil
}

The most important part of this function is that we use URL Encoding for our base64 string and not Standard Encoding. This is because standard encoding uses / and + which don’t work well in URLs whereas URL Encoding replaces these with — (minus) and _ (underscore). If you use the standard encoding and use ID’s in your URLs then its going to cause problems when you try to navigate to a page using an ID that has a / in it.

Step Two — Adding A Prefix

This part is optional but I like to add a prefix to my ID’s to make it clear what they are an ID for. For example, all my users ID’s would be prefixed with USR_, Meeting ID’s with MET_ and so on. To do this all we need are some constants and then we can prepend the prefix to the ID like so.

const prefixUser    string = "USR_"
// Call our generateBase64ID function to generate a user ID
id, err := generateBase64ID(10)
if err != nil {
log.Println(err)
}
// Concatenate the prefix and the ID to create a user ID
prefixedID := prefixUser + id

The above code could be repeated to generate ID’s for all entities and all we would need to change each time would be the prefix that we prepend to the ID. However, rather than calling generateBase64ID() and then adding our prefix to the ID. It would be more efficient to refactor generateBase64ID() so that we can pass the prefix in as a parameter which would save us from copying and pasting lots of code.

// Generate a random base64 url encoded string
func generateBase64ID(size int, prefix string) (string, error) {
    // First create a slice of bytes
b := make([]byte, size)
    // Read size number of bytes into b 
_, err := rand.Read(b)
if err != nil {
return "", err
}
    // Encode our bytes as a base64 encoded string using URLEncoding
encodedID := base64.URLEncoding.EncodeToString(b)
    // Add our prefix to the ID
prefixedID := prefix + encodedID
    return prefixedID, nil
}

Step Three — Check Our ID Is Unique

Since our ID isn’t auto incrementing we need to manually check if it has already been taken. To do this we need to open a connection to the database and query the user table to check that our ID doesn’t match an ID that already exists in the user table.

We can create a function that opens a connection to our database like so.

const ( 
dbType = "mysql"
dbUser = "test"
dbPassword = "test"
dbAddr = "localhost:3306"
dbName = "example-app"

connString = dbUser + ":" + dbPassword +
"@(" + dbAddr + ")/" + dbName + "?parseTime=true"
)
func connectToDB() *sql.DB { 
conn, err := sql.Open(dbType, connString)
if err != nil {
log.Println("Error connecting to DB: ", err)
}
return conn
}

We can then create the following function that checks whether or not the user ID has been taken.

func userIdUnqiue(id string) bool {
    // Open a connection to the DB
conn := connectToDB()
    // Store query result in this variable
var idTaken bool
    // Call a stored procedure that checks if the ID already exists
// in the database and store the result in idTaken
err := conn.QueryRow("call spUserIdTaken(?)",
id,
).Scan(&idTaken))
    // Check for errors    
if err != nil {
log.Println(err)
return false
}
    return isTaken
}

In this function we open a connection to the database and call a stored procedure that accepts our ID as a parameter and checks that it doesn’t already belong to a user. We then store and return the result of the stored procedure as a boolean value.

For reference your stored procedure should look similar to this

PROCEDURE `spUserIdTaken`(IN p_id varchar(150))
BEGIN
SELECT count(user_id)
FROM User
WHERE user_id = p_id
END

Using this stored procedure with the function above means that if the ID doesn’t already belong to a user then the stored procedure will return zero and therefore isTaken will be false . Otherwise if the ID belongs to a user the result of the stored procedure will be 1 and therefore isTaken will be true .

Step Four — Refactor Again

// Generate a random base64 url encoded string
func generateBase64ID(size int, prefix string) (string, error) {
    // First create a slice of bytes
b := make([]byte, size)
    // Read size number of bytes into b 
_, err := rand.Read(b)
if err != nil {
return "", err
}
    // Encode our bytes as a base64 encoded string using URLEncoding
encodedID := base64.URLEncoding.EncodeToString(b)
    var fullID string
var idTaken bool
    // Switch to create and check for ID based on the prefix
// passed in
switch prefix {
case prefixUser:
fullID = prefixUser + encodedID
idTaken = userIdTaken(prefixedId)
break
default:
log.Fatal("Prefix Not Implemented")
}
    // If the ID has been taken recursively call generateBase64ID
// until we generate a unique ID
if idTaken {
return generateBase64ID(size, prefix)
}
    return fullID, nil
}

You’ll notice that two significant changes have been made to this function. The first is we’ve added a switch statement so we can deal with generating ID’s for multiple entities. For example, if we need to generate an ID for a Meeting all we need to do now is add a case in and create a meetingIdTaken function that works the same way as the userIdTaken function. The second change is that we’re now using recursion which means if the ID has already been taken the function will call itself until it generates an ID that hasn’t been taken. Ensuring that the ID returned from this function will always be unique.

Review

At this stage our code should look like the following. Note, these functions are all in the one package for demonstration purposes only.

package main
const (

// Database connection properties
dbType = "mysql"
dbUser = "test"
dbPassword = "test"
dbAddr = "localhost:3306"
dbName = "example-app"

connString = dbUser + ":" + dbPassword +
"@(" + dbAddr + ")/" + dbName + "?parseTime=true"
    // User ID Prefix
prefixUser = "USR_"
)

// Open a connection to our database
func connectToDB() *sql.DB {
conn, err := sql.Open(dbType, connString)
if err != nil {
log.Println("Error connecting to DB: ", err)
}
return conn
}

// Generate a random base64 url encoded string
func generateBase64ID(size int, prefix string) (string, error) {
    // First create a slice of bytes
b := make([]byte, size)
    // Read size number of bytes into b 
_, err := rand.Read(b)
if err != nil {
return "", err
}
    // Encode our bytes as a base64 encoded string using URLEncoding
encodedID := base64.URLEncoding.EncodeToString(b)
    var fullID string
var idTaken bool
    // Switch to create and check for ID based on the prefix
// passed in
switch prefix {
case prefixUser:
fullID = prefixUser + encodedID
idTaken = userIdTaken(prefixedId)
break
default:
log.Fatal("Prefix Not Implemented")
}
    // If the ID has been taken recursively call generateBase64ID
// until we generate a unique ID
if idTaken {
return generateBase64ID(size, prefix)
}
    return fullID, nil
}

// Check that the userId generated is unique
func userIdUnqiue(id string) bool {
    // Open a connection to the DB
conn := connectToDB()
    // Store query result in this variable
var idTaken bool
    // Call a stored procedure that checks if the ID already exists
// in the database and store the result in idTaken
err := conn.QueryRow("call spUserIdTaken(?)",
id,
).Scan(&idTaken))
    // Check for errors    
if err != nil {
log.Println(err)
return false
}
    return isTaken
}

And that’s pretty much it. All you have left to do is to take the value returned by generateBase64ID() and assign it as the ID for whatever entity you’re adding to your database.

I hope you’ve learned something by reading this and if there’s anything I could improve upon please let me know. All feedback is appreciated.