Better way to read and write JSON file in Golang
In EACIIT we like to create our own library a lot. In this case we create a library to ease doing CRUD and Querying in JSON file.
The unmarshal way
When we first write the code for reading and filtering data from JSON file we do it like this.
This is not efficient since the we need to unmarshal and hold the whole data in memory then doing some iteration to filter the data. It might be ok with small JSON file. But will be a problem if we have big JSON file.
The JSON Decoder way
Better way to read JSON file is using json.Decoder. Because instead of unmarshal the whole content of a file the decoder will decode one line/record at a time while we doing filtering in data. This is much more efficient and less burden in memory.
To make sure that our code is correct that the json.Decoder way is much more efficient we benchmark it with 2 json file.
The first file contains list of American Movie scrapped from wikipedia, the file size is only 3.4MB. The second file contains 200,000+ random jeopardy question, i got this file from reddit post. The second file size is around 16x bigger, about 55MB. I put both files link in the end of this story.
And…. the results are as expected.
As you can see there are no difference when querying small file. But when the benchmarking on the bigger file there is a big difference in the memory efficiency. Memory of code that using json.Decoder is 182561704 B, that is almost half memory used rather than using unmarshal. And it’s faster too, I believe that the gap difference will increase along with the file size.
Writing JSON file
The writing part is similar with the reading one, in the first release we do it like this.
jsonString, _ := json.Marshal(makeData(1000000)) ioutil.WriteFile("big_marhsall.json", jsonString, os.ModePerm)Because we use Marshal, it seems fast enough when handling small file (because we never compare it with other alternative before.. oopsss…), But when the file is bigger file it slows down.. obviously.
After light discussion before going home, we learn that json.Marshal return string. So there must be a direct way to write into io.Writer instead of returning the value. And… yes… there is a direct way, because golang contributors are smart, so we changed it to this.
file, _ := os.OpenFile("big_encode.json", os.O_CREATE, os.ModePerm) defer file.Close() encoder := json.NewEncoder(file)
encoder.Encode(makeData(1000000))
And after benchmarking it, the one that using encode is faster (obviously). But what is strange is, the time/operation difference is bigger when using smaller data than the bigger one. Maybe my benchmarking code is not correct or have some flaws, if you have any idea why this happens please tell me in the comment section 😀
I hope that my story here could help you guys to resolve any problem in life. Hopefully. All code and JSON files source used can be found in this gist
