Differentiate between empty and not-set fields with JSON in Golang

3 min readSep 1, 2020

A few weeks ago, I was working on a Golang microservice where I needed to add support for CRUD operations with JSON data. Normally, I would make a structure for the entity with all the fields defined in it along with ‘omitempty’ attribute as shown

type Article struct {
   Id   string      `json:"id"`
   Name string      `json:"name,omitempty"`
   Desc string      `json:"desc,omitempty"`
}

Problem

But this kind of representation poses a serious problem, especially for Update or Edit operations.

For example, let’s say that the update request JSON looks something like this

{"id":"1234","name":"xyz","desc":""}

Notice the empty desc field. Now let's see how it is unmarshalled in Go

func Test_JSON1(t *testing.T) {         
   jsonData:=`{"id":"1234","name":"xyz","desc":""}`
   req:=Article{}
   _=json.Unmarshal([]byte(jsonData),&req)
   fmt.Printf("%+v",req)
}Output:
=== RUN   Test_JSON1
{Id:1234 Name:xyz Desc:}

Here description comes as an empty string, It’s clearly visible that the client wants to set desc as an empty string and that is inferred by our program.

But what if the client does not want the change the existing value for Description, in that case, sending a big description string again is not the right thing to do, hence the JSON would look like this

{"id":"1234","name":"xyz"}

Let’s unmarshal it into our structure

func Test_JSON2(t *testing.T) {         
   jsonData:=`{"id":"1234","name":"xyz"}`
   req:=Article{}
   _=json.Unmarshal([]byte(jsonData),&req)
   fmt.Printf("%+v",req)
}Output:
=== RUN   Test_JSON2
{Id:1234 Name:xyz Desc:}

Well, we still get Desc as an empty string , so how do we differentiate between not-set field and empty field

Short answer? Pointers

Solution

This is inspired by some existing Golang libraries like go-github. We can change our struct fields to pointer types, which would look like this

type Article struct {
   Id    string      `json:"id"`
   Name *string      `json:"name,omitempty"`
   Desc *string      `json:"desc,omitempty"`
}

By doing this we add an extra state to our fields. If the field does not exist in the raw JSON then the struct field will be null (nil).

On the other hand, if the field does exist and its value is empty, then the pointer is not null and the field contains the empty value.

Note- I did not change ‘Id’ to a pointer type because it cannot have a null state, id needs to be present at all times, its similar to a database id.

Let’s try it out.

func Test_JSON_Empty(t *testing.T) {
   jsonData := `{"id":"1234","name":"xyz","desc":""}`
   req := Article{}
   _ = json.Unmarshal([]byte(jsonData), &req)
   fmt.Printf("%+v\n", req)
   fmt.Printf("%s\n", *req.Name)
   fmt.Printf("%s\n", *req.Desc)
}
func Test_JSON_Nil(t *testing.T) {
   jsonData := `{"id":"1234","name":"xyz"}`
   req := Article{}
   _ = json.Unmarshal([]byte(jsonData), &req)
   fmt.Printf("%+v\n", req)
   fmt.Printf("%s\n", *req.Name)
}

Output

=== RUN   Test_JSON_Empty
{Id:1234 Name:0xc000088540 Desc:0xc000088550}
Name: xyz
Desc: 
--- PASS: Test_JSON_Empty (0.00s)=== RUN   Test_JSON_Nil
{Id:1234 Name:0xc00005c590 Desc:<nil>}
Name: xyz
--- PASS: Test_JSON_Nil (0.00s)

In the first case, as the description is set to an empty string, we get a non-null pointer in Desc with an empty string value. In the second case , where the field is not-set we get a null string pointer.

Hence we are able to differentiate between the two kinds of updates. This way works not just for strings but all the other data types including integers, nested structs, etc.

But this approach also comes with some problems.

Null Safety: Non-pointer data types have inherent null safety. Meaning, a string or int can never be null in Golang. They always have a default value. But if pointers are defined then those data types are null by default if not set manually. Hence trying to access the data of those pointers without verifying the nullability can lead to crashes in your application.

#The following code will crash because desc is null
func Test_JSON_Nil(t *testing.T) {
   jsonData := `{"id":"1234","name":"xyz"}`
   req := Article{}
   _ = json.Unmarshal([]byte(jsonData), &req)
   fmt.Printf("%+v\n", req)
   fmt.Printf("%s\n", *req.Desc)
}

This can be easily fixed by always checking for null pointers, but may make your code look dirty.

Printability: As you might have noticed in the pointer-based solution output, the value of the pointers is not printed. Instead, the hex pointer is printed which is not very useful in applications. This can also be overcome by implementing the stringer interface.

func (a *Article) String() string {
   output:=fmt.Sprintf("Id: %s ",a.Id)
   if a.Name!=nil{
   output+=fmt.Sprintf("Name: '%s' ",*a.Name)
   }
   if u.Desc!=nil{
   output+=fmt.Sprintf("Desc: '%s' ",u.Desc)
   }
   return output
}

Appendix:

Another solution to the above problem can be using an external library to have nullable types which provide you methods to check if they are null or not without caring about pointers.
https://github.com/guregu/null
https://github.com/google/go-github

Differentiate between empty and not-set fields with JSON in Golang

Written by Arpit Khurana