git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Scanning the list of entries present on a bookie


Sorry, maxEntryId can't be used for LAC.  Let me review the patch and
respond tomorrow. :)
I was mostly thinking of sync-on-close which is my use case. Intermittent
syncs will need a more robust way to define the LAC.

On Sat, Apr 21, 2018 at 12:00 AM, Venkateswara Rao Jujjuri <
jujjuri@xxxxxxxxx> wrote:

> @Enrico I thought I had a pretty simple view on this.
> 1. No ensemble changes allowed on a ledger in this mode.
> 2. A sync is sent to all bookies on the ensemble and it is required that
> the sync must be successful to proceed forward.
> 3. Sync and Write are serialized. or sync can return maxEntryId.
> 4. If we get a successful sync from all bookies, then the ' smallest
> maxEntryId ' returned will be the LAC.
> 5. An implicit sync is sent on close.
>
> IMO,  no ensemble change will make the whole thing simple and avoids scans
> from client side.
>
>
> On Wed, Apr 18, 2018 at 2:02 AM, Enrico Olivelli <eolivelli@xxxxxxxxx>
> wrote:
>
>> Any other suggestion? JV, Sijie?
>> My current idea for bp14 is to have a client driven scan.
>>
>>
>>
>> Enrico
>>
>> Il ven 13 apr 2018, 10:22 Enrico Olivelli <eolivelli@xxxxxxxxx> ha
>> scritto:
>>
>> >
>> >
>> > Il gio 12 apr 2018, 19:59 Sam Just <sjust@xxxxxxxxxxxxxx> ha scritto:
>> >
>> >> IIRC, InterleavedLedgerStorage has for each ledger an index file
>> >> mapping the entries to entry logger offsets, you could probably scan
>> >> that directly (particularly if you included a lower ?bound -- probably
>> >> the client's current idea of the LAC).
>> >> -Sam
>> >>
>> >
>> > Thank you Sam,
>> > I think that the major point is about introducing a standard interface
>> to
>> > be implemented by LedgerStorage, now we already have 3 implementations.
>> >
>> > Options are:
>> > 1) let the client drive the scan
>> > 2) add a scan() method to LedgerStorage
>> > 3) add a getMaxEntry() to LedgerStorage
>> >
>> > For bp14 I just need 1) or 3) but 1) is also a protocol issue and so it
>> is
>> > more expensive for long term maintenance
>> >
>> > Enrico
>> >
>> >>
>> >> On Thu, Apr 12, 2018 at 12:31 AM, Enrico Olivelli <eolivelli@xxxxxxxxx
>> >
>> >> wrote:
>> >> > Hi BookKeepers,
>> >> > during implementation of BP-14 I am facing a problem so I am asking
>> for
>> >> > suggestions.
>> >> >
>> >> > My need is to be able to know the list of all entries stored on a
>> >> > LedgerStorage given a ledgerId.
>> >> >
>> >> > Scanning from 0 to LedgerStorage#getLastAddConfirmed() does not
>> seem to
>> >> > work because we have to deal with WriteAdvHandle, so there can be
>> >> temporary
>> >> > "gaps" in the sequence of entries.
>> >> >
>> >> > I can have a writer which writes entries 0,1,5,6,7. Its LAC will be
>> at
>> >> most
>> >> > 1 as entries 2,3,4 are not written yet.
>> >> > I need on the bookie to able to know that entries 0, 1, 5, 6, 7 are
>> >> stored
>> >> > on LedgerStorage.
>> >> >
>> >> > I cannot issue a scan from 0 to Long.MAX_VALUE, my current 'solution'
>> >> it to
>> >> > make the client (writer) send the 'maximum entry id' and perform a
>> scan
>> >> > from 0 to maxEntryId.
>> >> > In the example the writer will send a forceLedger RPC with
>> maxEntryId =
>> >> 7.
>> >> >
>> >> > This is need only for recoveries are bookie restart because I have to
>> >> > reconstruct the knowledge about which entries have been persisted
>> >> durably
>> >> > on the Bookie.
>> >> >
>> >> > I am not very expert about LedgerStorage implementations, and I don't
>> >> know
>> >> > if it would be feasible to have such 'scan all entries' method.
>> >> >
>> >> > This is the code I am talking about
>> >> >
>> >> https://github.com/apache/bookkeeper/pull/1317/files#diff-3b
>> 81b1c90d1f51017627b3c032676168R1210
>> >> >
>> >> > Any help is really appreciated
>> >> > Enrico
>> >>
>> > --
>> >
>> >
>> > -- Enrico Olivelli
>> >
>> --
>>
>>
>> -- Enrico Olivelli
>>
>
>
>
> --
> Jvrao
> ---
> First they ignore you, then they laugh at you, then they fight you, then
> you win. - Mahatma Gandhi
>
>
>


-- 
Jvrao
---
First they ignore you, then they laugh at you, then they fight you, then
you win. - Mahatma Gandhi