git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Multiple concurrent SFTP file downloads


Hi

There is a JIRA ticket about concurrent downloads. Use the vote in
JIRA to tell us that you like that.



On Tue, Jun 19, 2018 at 7:30 PM, Alexander Barrow <barrowaw@xxxxxxxxx> wrote:
> I am developing an application that downloads large files from a
> remote SFTP server over a bandwidth limited network. Download speeds
> are bottle-necked, making it highly desirable to be able to download
> multiple files concurrently. From what I've gathered, this is not a
> feature supported out of the box by the standard SFTP component.
>
> A solution I engineered to work around this was to use a route defined
> something like as follows:
>
> from( "sftp:user@host?download=false..." )
> .threads( 5 ).maxQueueSize( 0 )
> .pollEnrich().simple(
> "sftp:user@host?localWorkDirectory=.inbound&include=$simple{file:name}&disconnect=true..."
> )
> .to( "file:downloads" );
>
> This uses the route's consumer to supply a "list" of files to
> download, but delegates the actual downloading to the poll-enrich EIP
> across multiple threads. The issue with this approach is that it uses
> multiple connections and repeatedly connects and disconnects from the
> remote server, which the SFTP server owners dislike.
>
> Is there another existing component or feature that would support the
> multiple concurrent downloads use case, or is this a gap in Camel's
> feature list?



-- 
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2



( ! ) Warning: include(msgfooter.php): failed to open stream: No such file or directory in /var/www/git/apache-camel-users/msg03448.html on line 110
Call Stack
#TimeMemoryFunctionLocation
10.0008358376{main}( ).../msg03448.html:0

( ! ) Warning: include(): Failed opening 'msgfooter.php' for inclusion (include_path='.:/var/www/git') in /var/www/git/apache-camel-users/msg03448.html on line 110
Call Stack
#TimeMemoryFunctionLocation
10.0008358376{main}( ).../msg03448.html:0