git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Scanning the list of entries present on a bookie


Il gio 12 apr 2018, 19:59 Sam Just <sjust@xxxxxxxxxxxxxx> ha scritto:

> IIRC, InterleavedLedgerStorage has for each ledger an index file
> mapping the entries to entry logger offsets, you could probably scan
> that directly (particularly if you included a lower bound -- probably
> the client's current idea of the LAC).
> -Sam
>

Thank you Sam,
I think that the major point is about introducing a standard interface to
be implemented by LedgerStorage, now we already have 3 implementations.

Options are:
1) let the client drive the scan
2) add a scan() method to LedgerStorage
3) add a getMaxEntry() to LedgerStorage

For bp14 I just need 1) or 3) but 1) is also a protocol issue and so it is
more expensive for long term maintenance

Enrico

>
> On Thu, Apr 12, 2018 at 12:31 AM, Enrico Olivelli <eolivelli@xxxxxxxxx>
> wrote:
> > Hi BookKeepers,
> > during implementation of BP-14 I am facing a problem so I am asking for
> > suggestions.
> >
> > My need is to be able to know the list of all entries stored on a
> > LedgerStorage given a ledgerId.
> >
> > Scanning from 0 to LedgerStorage#getLastAddConfirmed() does not seem to
> > work because we have to deal with WriteAdvHandle, so there can be
> temporary
> > "gaps" in the sequence of entries.
> >
> > I can have a writer which writes entries 0,1,5,6,7. Its LAC will be at
> most
> > 1 as entries 2,3,4 are not written yet.
> > I need on the bookie to able to know that entries 0, 1, 5, 6, 7 are
> stored
> > on LedgerStorage.
> >
> > I cannot issue a scan from 0 to Long.MAX_VALUE, my current 'solution' it
> to
> > make the client (writer) send the 'maximum entry id' and perform a scan
> > from 0 to maxEntryId.
> > In the example the writer will send a forceLedger RPC with maxEntryId =
> 7.
> >
> > This is need only for recoveries are bookie restart because I have to
> > reconstruct the knowledge about which entries have been persisted durably
> > on the Bookie.
> >
> > I am not very expert about LedgerStorage implementations, and I don't
> know
> > if it would be feasible to have such 'scan all entries' method.
> >
> > This is the code I am talking about
> >
> https://github.com/apache/bookkeeper/pull/1317/files#diff-3b81b1c90d1f51017627b3c032676168R1210
> >
> > Any help is really appreciated
> > Enrico
>
-- 


-- Enrico Olivelli