git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Dealing with data latency


Hi. What is the recommended way to deal with data latency? For example, I
have a feed that is not considered final until 72 hours have passed after
the end of the daily period.

For example, Monday's data would be ready by Thursday at 23:59.

Should I pull data based on the execution date minus a 72 hour offset or
use the execution date and somehow delay the data pull for 72 hours?

The latter would be more intuitive (data pull date = execution date) but I
am not sure if it's a good pattern.

Thanks,

Pedro