[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Providing default values for ValueProvider

It sounds like it may simply be an oversight that @Default does not work with ValueProvider. Copying dev@ to see if there's some pitfall associated with adding this.


On Thu, Dec 20, 2018 at 5:17 PM Jeff Klukas <jklukas@xxxxxxxxxxx> wrote:
I am also now realizing that runtime params show up in --help with type of ValueProvider, so I can likely meet my documentation needs via a README for the project that explains how runtime parameters behave and that ValueProvider type is the flag that lets you know a parameter can be overridden at runtime.

On Thu, Dec 20, 2018 at 4:47 PM Jeff Klukas <jklukas@xxxxxxxxxxx> wrote:
I did some more testing this afternoon, and it looks like I'm incorrect on some details here.

Parameters implemented as ValueProviders can be provided at compile time and they do end up in the template. A value specified at runtime overrides. If not specified at runtime, the compile-time value is in effect.

It still looks, though, like @Default has no effect for ValueProviders. If not value is specified at compile time or runtime, the ValueProvider is not available, even if it has a @Default configured.

Does anybody have tips for documenting which params are overrideable at runtime? Or experience handling default values for ValueProviders?

On Thu, Dec 20, 2018 at 3:49 PM Jeff Klukas <jklukas@xxxxxxxxxxx> wrote:
I am deploying Beam pipelines with the DataflowRunner and would like to move more of the pipeline options to use the ValueProvider interface so they can be specified at runtime rather than at template compile time, but running into various issues.

First, it's been unclear to the operations engineers deploying pipelines which options are runtime and which are compile-time. The engineers are typically using the gcloud command-line interface to deploy rather than the console, so I don't see much benefit from implementing a metadata json file.

AFAICT, Dataflow happily accepts values for runtime parameters when compiling the template, but those values are completely ignored and need to be defined again at runtime, furthering the confusion. Should I just go ahead and add "(runtime parameter)" to each of the @Description annotations to at least document the distinction in --help output?

Finally, it's not clear whether the @Default annotation supports runtime parameters. The dataflow docs show an example where @Default is used on a ValueProvider [0], but this doesn't appear to actually have any effect. If I don't pass in a value for a runtime parameter when executing a template, the pipeline throws a "Value only available at runtime" exception on calls to .get(), rather than returning the default value. Have others encountered this? Is there a pattern for providing defaults for runtime parameters?