git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Usage of readLockMinAge


Hi

Have you tried with a custom filter where you can check file size and
skip small files.
On Thu, Sep 27, 2018 at 2:16 PM Sunil O <sunilomk2002@xxxxxxxxx> wrote:
>
> We have a implementation scenario with File consumer polling large number
> of folders (3000+) and 200K files per day with 1 mts SLA per file
> processing.
>
> We also need to use readLock to avoid picking files which are being
> written.
>
> In this scenario - we went for readLock=changed option. However this option
> results in thread sleeping for minium time as specified by
> readLockCheckInterval period option. While looking for workarounds - we
> found the readLockMinAge option - which allows to pick up files which are
> old enough without getting into the sleep mode. This has reduced the time
> for picking files and thereby reducing the overall processing time.
>
> However whenever a file is encountered with age below minage - the sleep
> occurs as per the logic in FileChangedExclusiveReadLockStrategy.  If this
> 'sleep' step can be avoided when readLockMin age is specified - then
> insteading of sleeping - the consumer can go on to pick other files. This
> modified behavior would be useful in scenario where overall throughput and
> processing performance is important than sequential processing etc.
>
>
> While browsing similar issues - found JIRA issue 9324 which also discusses
> the issue regarding the Sleep step -
>
> So it would be good if one of the below is available
>
> a)  there is a separate ExclusiveReadLockStrategy similar to
> FileChangedExclusiveReadLockStrategy which deals only with readLockMin age
> and skips file if age is not met instead of sleeping.
>
> Or
>
> b) an option skip/sleep should be added for
> FileChangedExclusiveReadLockStrategy when readLockMinAge is used.
>
>
> Please give your suggestions.



-- 
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2