git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Go SDK: Teardown() not being called with dataflow runner


Essentially I have the following code:

type Writer struct {
  Pool WriterPool
}

func (w *Writer) Setup() {
 w.Pool = Init()
}

func (w* Writer) ProcessElement(ctx, elem Elem) {
  w.Pool.Add(elem)
}

func (w* Writer) Teardown() {
  w.Pool.Write()
  w.Pool.Close()
}

beam.ParDo0(scope, &Writer{}, elemCollection)

The above code runs fine with the direct runner but not with dataflow.

 I added log lines to the above methods, and the ones in Teardown() never appear in the logs. 
If I change my code as follows:

func (w* Writer) ProcessElement(ctx, elem Elem) {
  w.Pool.Add(elem)
  w.Pool.Write()
}

Then I see the data being written, but I lose the ability to pool, plus I am leaking connections.

Is this a known issue, or I am going something wrong?

Thanks again for the help.