Thanks for the reply. We are currently looking into the batch pipeline. Your suggested approaches are also the direction that we are investigating, next to using Spark or Flink directly.
Why would you say that it is better to materialize the intermediate state? Is this more/less efficient? Suppose I output my "finished" processing to one collection and I output my "unfinished" data to another PCollection. Will that still be less efficient?
Looking at the data, currently, I would need 6 iterations with an average of 4. However, that of course depends on the data which can evolve over time. So the current, custom, implementation has a hard limit of 15 that I would take over. How does this affect the performance if I have 9 times a NOOP?