Hello beam team,
I'm reading from a Kafka topic and writing to Pubsub. The code constructs PubsubMessage objects from the Kafka messages
I'm setting some attributes for the PubsubMessage objects, namely message_id and message_timestamp2
I wish to use these attribute values downstream, so that subscribing dataflow jobs use the message_id for exactly once processing and the message_timestamp2 for windowing. Therefore, when I write to Pubsub, I do this
However, instead of setting the values downstream with these attribute values, dataflow is updating them attributes with the default message_id and the process time.
What am I doing wrong?
love your data
Terry Dhariwal | Solutions Architect
t +44 (0) 7460 877 412w www.pythian.com