git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Multiple concurrent SFTP file downloads


Claus, is the ticket you're referring to CAMEL-6058: camel-ftp - Allow
concurrent download of files? The verbiage in that ticket appears to
refer only to the FTP consumer, whereas my use case requires the SFTP
consumer.

Additionally, I've learned more about the network I'm working with and
found that download speeds are throttled per connection, not per
download request. My current line of thought is that an SFTP consumer
connection pool may be the solution I'm looking for. Though perhaps a
connection pool would be implicit in any realization of concurrent
downloads, as the JIRA ticket description suggests for the FTP
consumer.

Is CAMEL-6058 still the appropriate JIRA ticket for my use case
requiring the SFTP consumer?

On Tue, Jun 26, 2018 at 6:05 AM, Claus Ibsen <claus.ibsen@xxxxxxxxx> wrote:
> Hi
>
> There is a JIRA ticket about concurrent downloads. Use the vote in
> JIRA to tell us that you like that.
>
>
>
> On Tue, Jun 19, 2018 at 7:30 PM, Alexander Barrow <barrowaw@xxxxxxxxx> wrote:
>> I am developing an application that downloads large files from a
>> remote SFTP server over a bandwidth limited network. Download speeds
>> are bottle-necked, making it highly desirable to be able to download
>> multiple files concurrently. From what I've gathered, this is not a
>> feature supported out of the box by the standard SFTP component.
>>
>> A solution I engineered to work around this was to use a route defined
>> something like as follows:
>>
>> from( "sftp:user@host?download=false..." )
>> .threads( 5 ).maxQueueSize( 0 )
>> .pollEnrich().simple(
>> "sftp:user@host?localWorkDirectory=.inbound&include=$simple{file:name}&disconnect=true..."
>> )
>> .to( "file:downloads" );
>>
>> This uses the route's consumer to supply a "list" of files to
>> download, but delegates the actual downloading to the poll-enrich EIP
>> across multiple threads. The issue with this approach is that it uses
>> multiple connections and repeatedly connects and disconnects from the
>> remote server, which the SFTP server owners dislike.
>>
>> Is there another existing component or feature that would support the
>> multiple concurrent downloads use case, or is this a gap in Camel's
>> feature list?
>
>
>
> --
> Claus Ibsen
> -----------------
> http://davsclaus.com @davsclaus
> Camel in Action 2: https://www.manning.com/ibsen2