git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Go] High memory usage on CSV read into table


Sorry just realised SVG doesn't work.

PNG of the pprof can be found here: https://i.imgur.com/BVXv1Jm.png


Daniel Harper
http://djhworld.github.io


On Sun, 18 Nov 2018 at 21:07, Daniel Harper <djharperuk@xxxxxxxxx> wrote:

> Wasn't sure where the best place to discuss this, but I've noticed that
> when running the following piece of code
>
> https://play.golang.org/p/SKkqPWoHPPS
>
> On a CSV files that contains roughly 1 million records (about 100mb of
> data), the memory usage of the process leaps to about 9.1GB
>
> The records look something like this
>
>
> "2018-08-27T20:00:00Z","cdnA","dash","audio","http","programme-1","3577","2018","08","27","2018-08-27","live"
>
> "2018-08-27T20:00:01Z","cdnB","hls","video","https","programme-2","14","2018","08","27","2018-08-27","ondemand"
>
> I've attached a pprof output of the process.
>
> From the looks of it the heavy use of _strings_ might be where most of the
> memory is going.
>
> Is this expected? I'm new to the code, happy to help where possible!
>
>
> Daniel Harper
> http://djhworld.github.io
>