git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

S3keysonsor


Hi ,

I’m trying to evaluate airflow to see if it suits my needs.

Basically i can have below steps in a DAG



1)Look for a file arrival on given s3 location (this path is not completely
static) (i can use S3Keysensor in this step)

  i should be able to specify to look either for latest folder or 24hrs or
n number of days older folder which has _SUCCESS file as mentioned below

  sample file location(s):

  s3a://mybucket/20180425_111447_data1/_SUCCESS

  s3a://mybucket/20180424_111241_data1/_SUCCESS

  s3a://mybucket/20180424_111035_data1/_SUCCESS



2)invoke a simple restapi using HttpSimpleOperator once the above
dependency is met ,i can set upstream for step2 as step1



Does S3keysensor supports step1 out of the box?

Also in some cases i may to have a DAG without start date & end date it
just needs to be triggered once file is available in a given s3 location



 *Please suggest !*


( ! ) Warning: include(msgfooter.php): failed to open stream: No such file or directory in /var/www/git/apache-airflow-development/msg03410.html on line 105
Call Stack
#TimeMemoryFunctionLocation
10.0029362976{main}( ).../msg03410.html:0

( ! ) Warning: include(): Failed opening 'msgfooter.php' for inclusion (include_path='.:/var/www/git') in /var/www/git/apache-airflow-development/msg03410.html on line 105
Call Stack
#TimeMemoryFunctionLocation
10.0029362976{main}( ).../msg03410.html:0