git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Scanning the list of entries present on a bookie


Any other suggestion? JV, Sijie?
My current idea for bp14 is to have a client driven scan.



Enrico

Il ven 13 apr 2018, 10:22 Enrico Olivelli <eolivelli@xxxxxxxxx> ha scritto:

>
>
> Il gio 12 apr 2018, 19:59 Sam Just <sjust@xxxxxxxxxxxxxx> ha scritto:
>
>> IIRC, InterleavedLedgerStorage has for each ledger an index file
>> mapping the entries to entry logger offsets, you could probably scan
>> that directly (particularly if you included a lower ?bound -- probably
>> the client's current idea of the LAC).
>> -Sam
>>
>
> Thank you Sam,
> I think that the major point is about introducing a standard interface to
> be implemented by LedgerStorage, now we already have 3 implementations.
>
> Options are:
> 1) let the client drive the scan
> 2) add a scan() method to LedgerStorage
> 3) add a getMaxEntry() to LedgerStorage
>
> For bp14 I just need 1) or 3) but 1) is also a protocol issue and so it is
> more expensive for long term maintenance
>
> Enrico
>
>>
>> On Thu, Apr 12, 2018 at 12:31 AM, Enrico Olivelli <eolivelli@xxxxxxxxx>
>> wrote:
>> > Hi BookKeepers,
>> > during implementation of BP-14 I am facing a problem so I am asking for
>> > suggestions.
>> >
>> > My need is to be able to know the list of all entries stored on a
>> > LedgerStorage given a ledgerId.
>> >
>> > Scanning from 0 to LedgerStorage#getLastAddConfirmed() does not seem to
>> > work because we have to deal with WriteAdvHandle, so there can be
>> temporary
>> > "gaps" in the sequence of entries.
>> >
>> > I can have a writer which writes entries 0,1,5,6,7. Its LAC will be at
>> most
>> > 1 as entries 2,3,4 are not written yet.
>> > I need on the bookie to able to know that entries 0, 1, 5, 6, 7 are
>> stored
>> > on LedgerStorage.
>> >
>> > I cannot issue a scan from 0 to Long.MAX_VALUE, my current 'solution'
>> it to
>> > make the client (writer) send the 'maximum entry id' and perform a scan
>> > from 0 to maxEntryId.
>> > In the example the writer will send a forceLedger RPC with maxEntryId =
>> 7.
>> >
>> > This is need only for recoveries are bookie restart because I have to
>> > reconstruct the knowledge about which entries have been persisted
>> durably
>> > on the Bookie.
>> >
>> > I am not very expert about LedgerStorage implementations, and I don't
>> know
>> > if it would be feasible to have such 'scan all entries' method.
>> >
>> > This is the code I am talking about
>> >
>> https://github.com/apache/bookkeeper/pull/1317/files#diff-3b81b1c90d1f51017627b3c032676168R1210
>> >
>> > Any help is really appreciated
>> > Enrico
>>
> --
>
>
> -- Enrico Olivelli
>
-- 


-- Enrico Olivelli