Handling User Datatypes in Golang with JSON and SQL database

Handling JSON Unmarshal, JSON Marshal, SQL Scan and value methods for custom struct

Handling JSON Unmarshal, JSON Marshal, SQL Scan and value methods for custom struct

In Golang, structs are powerful data structures used to define user types. In back-end development, we might need to send the data as JSON response or store it in a database. JSON and database have a small number of built-in types. It should not limit our code to just built-in types and we should be able to handle our data struct in these scenarios with minimum code duplication. Let’s implement a simple solution for this

What we need for our struct:

1. JSON Marshalling and Unmarshalling should be possible
2. Should be able to insert and scan the data from the database
3. Should be able to handle null values in database and Json since GO uses zero values or default values like 0 and empty string instead of None or NULL

Enough theory, let’s start coding

We will create a new struct type Email to store the email address and domain separately

Email Type, It is a struct containing built-in types string and bool

We use the Valid flag to check whether the struct contains any data or not. This is required because golang sets initials values or zero values to variables when they are created. These zero values might be an issue while converting into NULL values. Valid flag is very useful when we insert it into the database or convert into JSON

And create a Person using our Email type

Person Struct will store email address as an Email object instead of a string

We should be able to Marshal the Person struct into the following JSON and Unmarshal it back directly

{
     "name": "Steve Mcqueen",
     "email": "steve.mcqueen@gmail.com"
}

For handling this task let’s use the built-in JSON package “encoding/json”

Here the email is of type string in the above JSON. We need to convert it into an Email object during the json.Unmarshal/Decoding. Similarly, we should convert the Email object into string during json.Marshal/Encoding

So how does json.Unmarshal and json.Marshal works ???

Let’s look into the documentation

To unmarshal JSON into a value implementing the Unmarshaler interface, Unmarshal calls that value's UnmarshalJSON method, including when the input is a JSON null.
type Unmarshaler interface {
UnmarshalJSON([]byte) error
}
________________________________________________________________
If an encountered value implements the Marshaler interface and is not a nil pointer, Marshal calls its MarshalJSON method to produce JSON
type Marshaler interface {
MarshalJSON() ([]byte, error)
}

If you can’t understand this right away don’t worry, it is pretty simple and we will explore it in depth.

For every struct type, we need to define the method UnmarshalJSON([]byte). Struct type should be a receiver for these methods. So when the JSON tries to unmarshal the data it will call the UnmarshalJSON for that specific type.

Similarly for Marshaling we need to define the method MarshalJSON() with the struct type as its receiver.

You will get a better idea about this as we proceed with our code

Now let’s define UnmarshalJSON and MarshalJSON methods for the type Email. Furthermore, note that the functions should contain the same signature specified in the interface. That includes the type and number of parameters passed and the values that are returned.

Let’s see UnmarshalJSON in details

(The code might look slightly longer and not optimized for some readers. That is because the main focus of the article is to explain the concept in a very simple manner. You can implement your code in more optimal ways)

Let’s understand the code

• line 2: Define UnmarshalJSON on the struct Email
• lines 5–8: If the value is “null” we need to set the Valid flag false. GO will set default values to all fields in the struct, So later in our program, it will be easy for us to check if the Email address is valid. Also, the Valid flag will be very useful to us while inserting into the database
• lines 11–14: Converts the bytes from JSON into a string, because that is how originally the email address is represented in the JSON. Here it can be directly converted into a string, but we prefer to use json.Unmarshal to make use of the error detection and other behaviours
• lines 18–20: Split the string at the character ‘@’ to find the Username and Domain, then we assign it to separate variables.
• line 23: Create a new Email object using the values we got and assign it to the pointer which we received in the function

Let’s test our code now

output:

>> &{Steve Mcqueen {steve.mcqueen gmail.com true}}

Now let’s write MarshalJSON. To make our job easier, will also write a String method on Email

• lines 1–7: String() method will convert the Email object to a string of proper email format. It is useful for conversion and packages that use Stringer interface
• line 10: MarshalJSON function for Email
• lines 13–15: If the Email obj is invalid, return a null value to the JSON
• line 16: Get the string from the Email obj and do json.Marshal on the string

Now let’s see the full code in action, you can try it yourself in the playground

Well now since the JSON is out of our way, we will see how to insert and read the Email from the database.

We want to store our email address as a string (varchar) in our table. So when we insert and read from the database, our code should automatically convert between Email object and string

We will try to insert and scan an Email address from a SQL database

So how does the ‘database/sql’ library insert and scan data from the database? Let’s see the documentation

database/sql makes use of Scanner interface and driver Valuer interface fetching and inserting data
Scanner interface :
type Scanner interface {
// Scan assigns a value from a database driver.
Scan(src interface{}) error
}
driver Valuer interface :
type Valuer interface {
// Value returns a driver Value.
// Value must not panic.
Value() (Value, error)
}

Just like how we defined the JSON methods, let’s implement two new methods to do the job for us

Let’s first define the Scan method

If you have worked with SQL and golang you might be familiar with “NULL” value issues. While reading rows containing NULL values we will end up with the following error

Scan error on column index 0, name "data": converting NULL to string is unsupported

Here converting NULL to string is the issue. GO does not allow us to store a nil or null value in basic built-in types. Instead of nil, GO stores default values or zero values during initialization. This might be a problem for us because some of those zero values might be valid in some cases.

Let’s see an example. Say we have a foreign key column with an integer type. So when a reference is not established we might want to insert a NULL in that column, which is fine from a DBMS perspective. But GO assigns a default value 0 (zero) and in our database table that might mean a reference to something or we could get a foreign key constrain error, both of these cases are undesirable. So we need a way to identify whether the value in a variable is NULL or not

The efficient solution for this problem is to use the Null types defined in the SQL package. You can read about using these types in this excellent blog post.

Let’s see how NullInt64 is implemented in the SQL package

type NullInt64 struct {
     Int64 int64
     Valid bool // Valid is true if Int64 is not NULL
}

In NullInt64 implementation, Int64 is wrapped in another struct and uses a boolean variable Valid to check if it is valid. While assigning values we will set it as true and when not in use it will be false

The same technique can be applied in our struct Email. We will set the Valid variable in our struct true when it contains valid data and false when we want to show it is empty.

So let’s write the Scanner function for Email and see the above concepts in action. Also, note that Scan is called during a database read operation

line 2: When we do a Scan on database result, it calls the Scan method for the type by passing the value from the database as a value. In our case value is going to contain an email id in string, eg: ‘steve.mcqueen@gmail.com’

lines 5–8: If the value is nil (NULL in the database) then we set the Valid flag for email as false

line 10–12: Split the string in the database into username and domain based on the location of ‘@’ symbol

line 13: Create a new Email obj and copy its content to the Email pointer (receiver to the method)

Let’s see how the Valuer interface is implemented. We use the Value method to insert it into the database. Value function receives the Email object as a receiver. We will convert this object into string and pass to SQL.

line 2: Value() returns the data in whichever type we want it to store it in the database. In our example, we need to store an email id as a string in the database. The return type of the function is driver.Value, Which is an empty interface. So as long as our database accepts, we can return whatever we want

lines 3–5: if the email id is not Valid, we will return a nil value. Which will be stored in the database as NULL

line 6: Converts our Email object into a string using the String() method that was defined earlier while JSON Marshalling. Return the string with error as nil

We have our Scanner and Value driver implemented. Let’s see how we make use of this in our complete code. Before that let’s create a database table email_address for testing our code.

CREATE TABLE email_address (
    id serial NOT NULL,
    email varchar NULL,
    CONSTRAINT email_pk PRIMARY KEY (id)
);

After inserting some dummy data our table will look like this

Now let’s see the full code

HURRAY !!!. now we have a custom struct which can be used with JSON and SQL database just like any built-in types

You can find the full code with both JSON and SQL database implementation in this GitHub repo

A few things to keep in mind:

• If you want to insert or retrieve values from the database for built-in types, You can use the types defined inside SQL they include NullBool, NullFloat64, NullInt32, NullInt64, NullString, and NullTime. But you might need to define marshal and unmarshal methods for JSON. You can find a blog post about that here.
• I have used Postgresql in the code. Depending on whatever database you use you can customize it. But the basic concepts will be the same

I hope you enjoyed the article.

If you can make any improvement or correction to the above article please leave a comment or feel free to contact me