git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Scanning the list of entries present on a bookie


@Enrico I thought I had a pretty simple view on this.
1. No ensemble changes allowed on a ledger in this mode.
2. A sync is sent to all bookies on the ensemble and it is required that
the sync must be successful to proceed forward.
3. Sync and Write are serialized. or sync can return maxEntryId.
4. If we get a successful sync from all bookies, then the ' smallest
maxEntryId ' returned will be the LAC.
5. An implicit sync is sent on close.

IMO,  no ensemble change will make the whole thing simple and avoids scans
from client side.


On Wed, Apr 18, 2018 at 2:02 AM, Enrico Olivelli <eolivelli@xxxxxxxxx>
wrote:

> Any other suggestion? JV, Sijie?
> My current idea for bp14 is to have a client driven scan.
>
>
>
> Enrico
>
> Il ven 13 apr 2018, 10:22 Enrico Olivelli <eolivelli@xxxxxxxxx> ha
> scritto:
>
> >
> >
> > Il gio 12 apr 2018, 19:59 Sam Just <sjust@xxxxxxxxxxxxxx> ha scritto:
> >
> >> IIRC, InterleavedLedgerStorage has for each ledger an index file
> >> mapping the entries to entry logger offsets, you could probably scan
> >> that directly (particularly if you included a lower ?bound -- probably
> >> the client's current idea of the LAC).
> >> -Sam
> >>
> >
> > Thank you Sam,
> > I think that the major point is about introducing a standard interface to
> > be implemented by LedgerStorage, now we already have 3 implementations.
> >
> > Options are:
> > 1) let the client drive the scan
> > 2) add a scan() method to LedgerStorage
> > 3) add a getMaxEntry() to LedgerStorage
> >
> > For bp14 I just need 1) or 3) but 1) is also a protocol issue and so it
> is
> > more expensive for long term maintenance
> >
> > Enrico
> >
> >>
> >> On Thu, Apr 12, 2018 at 12:31 AM, Enrico Olivelli <eolivelli@xxxxxxxxx>
> >> wrote:
> >> > Hi BookKeepers,
> >> > during implementation of BP-14 I am facing a problem so I am asking
> for
> >> > suggestions.
> >> >
> >> > My need is to be able to know the list of all entries stored on a
> >> > LedgerStorage given a ledgerId.
> >> >
> >> > Scanning from 0 to LedgerStorage#getLastAddConfirmed() does not seem
> to
> >> > work because we have to deal with WriteAdvHandle, so there can be
> >> temporary
> >> > "gaps" in the sequence of entries.
> >> >
> >> > I can have a writer which writes entries 0,1,5,6,7. Its LAC will be at
> >> most
> >> > 1 as entries 2,3,4 are not written yet.
> >> > I need on the bookie to able to know that entries 0, 1, 5, 6, 7 are
> >> stored
> >> > on LedgerStorage.
> >> >
> >> > I cannot issue a scan from 0 to Long.MAX_VALUE, my current 'solution'
> >> it to
> >> > make the client (writer) send the 'maximum entry id' and perform a
> scan
> >> > from 0 to maxEntryId.
> >> > In the example the writer will send a forceLedger RPC with maxEntryId
> =
> >> 7.
> >> >
> >> > This is need only for recoveries are bookie restart because I have to
> >> > reconstruct the knowledge about which entries have been persisted
> >> durably
> >> > on the Bookie.
> >> >
> >> > I am not very expert about LedgerStorage implementations, and I don't
> >> know
> >> > if it would be feasible to have such 'scan all entries' method.
> >> >
> >> > This is the code I am talking about
> >> >
> >> https://github.com/apache/bookkeeper/pull/1317/files#diff-
> 3b81b1c90d1f51017627b3c032676168R1210
> >> >
> >> > Any help is really appreciated
> >> > Enrico
> >>
> > --
> >
> >
> > -- Enrico Olivelli
> >
> --
>
>
> -- Enrico Olivelli
>



-- 
Jvrao
---
First they ignore you, then they laugh at you, then they fight you, then
you win. - Mahatma Gandhi