Reducing Memory Allocation

Powerful Command-Line Applications in Go — by Ricardo Gerardi (53 / 127)

👈 Profiling Your Tool | TOC | Tracing Your Tool 👉

You now know that the tool spends a lot of time with the garbage collector because it’s allocating too much memory. You also know that the bulk of the memory allocation comes from the ReadAll function of the encoding/csv package, which you’re calling from the csv2float function.

If you follow the program’s logic, it’s reading all the records from each CSV file into memory and then processing them one at a time with a loop, storing the result in another slice. It’s not doing anything that requires the entire content of the file to be in memory, so you can replace the calls to ReadAll with the function Read from the encoding/csv package. This function reads one record at a time, so you can execute it directly inside the loop, preventing the whole file from being read at once.

Start by removing the call to ReadAll and the associated error check. Remove these lines from the csv2float function in the csv.go file:

​ ​// Read in all CSV data​
​ allData, err := cr.ReadAll()
​ ​if​ err != nil {
​   ​return​ nil, fmt.Errorf(​"Cannot read data from file: %w"​, err)
​ }

Next, replace the loop header. Instead of using the range operator on allData, use an infinite loop since you don’t know beforehand how many records it needs to read: