git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Go] High memory usage on CSV read into table


Wasn't sure where the best place to discuss this, but I've noticed that when running the following piece of code


On a CSV files that contains roughly 1 million records (about 100mb of data), the memory usage of the process leaps to about 9.1GB

The records look something like this

"2018-08-27T20:00:00Z","cdnA","dash","audio","http","programme-1","3577","2018","08","27","2018-08-27","live"
"2018-08-27T20:00:01Z","cdnB","hls","video","https","programme-2","14","2018","08","27","2018-08-27","ondemand"


I've attached a pprof output of the process.

From the looks of it the heavy use of _strings_ might be where most of the memory is going.

Is this expected? I'm new to the code, happy to help where possible!


Attachment: pprof001.svg
Description: image/svg