I am running python pipeline with google.cloud.bigquery library.
on the local runner, everything runs great
bigquery.__version__ is 0.28.0
on the dataflow runner, the version is 0.23.0 bigquery.__version__ is 0.23.0
and there are many API changes between these versions.
What will be the best way to change the installed version on the workers? I was assuming the the worker has all the master machine libraries installed when the execution is done from datalab - is that true?
I am not generating any requirements.txt, the execution is done through the run button on the datalab UI.
please help me solve that issue.