git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Fwd: Decrease initial source read speed


Hi,


Right now it is a Kafka source, but I had the same issue when reading data from local FS.

It looks like a common problem for many (all?) sources.
When incoming data is very small (paths to large archives) but each entry requires a significant time to process (unpack, parse, etc.) Flink detects the back pressure with delay and too much data becomes part of the first transaction.



--
Best regards,
Andrei Shumanski



On Fri, May 18, 2018 at 11:44 AM, makeyang <riverbuilding@xxxxxxxxxxx> wrote:
Andrei Shumanski:
    which source are u using?



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/



( ! ) Warning: include(msgfooter.php): failed to open stream: No such file or directory in /var/www/git/apache-flink-users/msg09354.html on line 77
Call Stack
#TimeMemoryFunctionLocation
10.0043364568{main}( ).../msg09354.html:0

( ! ) Warning: include(): Failed opening 'msgfooter.php' for inclusion (include_path='.:/var/www/git') in /var/www/git/apache-flink-users/msg09354.html on line 77
Call Stack
#TimeMemoryFunctionLocation
10.0043364568{main}( ).../msg09354.html:0