git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: HBase Short-Circuit Read Questions


On Fri, Jun 1, 2018 at 9:36 AM, Mike Drob <madrob@xxxxxxxxxxxx> wrote:

> Hi folks, I was going through our docs looking at SCR set up and had some
> confusion. Asking here before filing JIRA issues. After writing this, I'm
> realizing the length got a bit out of hand. I don't want to split this into
> several threads because I think the information is all related, but may
> have to do that if a single one becomes difficult to follow.
>
>
Thanks for taking a look. See in below.



>
> The main docs link: http://hbase.apache.org/book.html#shortcircuit.reads
>
> 1)
>
> Docs claim: dfs.client.read.shortcircuit.skip.checksum = true so we don’t
> double checksum (HBase does its own checksumming to save on i/os. See
> hbase.regionserver.checksum.verify for more on this.
>
> Code claims:
>
> https://github.com/apache/hbase/blob/master/hbase-
> common/src/main/java/org/apache/hadoop/hbase/util/
> CommonFSUtils.java#L784-L788
>
> That if this property is set, then we log a warning?
>
> Unrelated, this is duplicated between CommonFSUtils and FSUtils, will need
> a jira to clean that up later.
>
> Also, there's a comment in
> https://github.com/apache/hbase/blob/master/hbase-
> server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.
> java#L689-L690
> that claims we automatically disable it, which we do in the HFileSystem
> constructor by setting the same dfs property in our conf to true.
>
> So I'm confused if we should be setting the property like the docs claim,
> not setting it like FSUtils warns, or ignoring it and letting RS auto-set
> it.
>
>

This is a classic font of confusion with layers of attempts over time at
simplification, auto-config'ing, code migration, and poor doc (I believe
I'm author of good bit of this mess here). Would be cool if it got a revamp
informed by tinkering with configs and an edit by a better writer than I.



> 2)
>
> Docs claim: dfs.client.read.shortcircuit.buffer.size = 131072 Important to
> avoid OOME — hbase has a default it uses if unset, see
> hbase.dfs.client.read.shortcircuit.buffer.size; its default is 131072.
>
> This is very confusing, we should set the property to some value, because
> if it's unset then we will use... the same value? This reads like needless
> operator burden.
>
> Looking at the code, the default we really use is 64 * 1024 * 2 = 126976,
> which is actually close, but off by enough to give me pause.
>
> The default HDFS value is 1024 * 1024, which suggests that they're
> expecting a value in the MB range and we're giving one in the KB range?
> See:
> https://github.com/apache/hadoop/blob/master/hadoop-
> hdfs-project/hadoop-hdfs-client/src/main/java/org/
> apache/hadoop/hdfs/client/HdfsClientConfigKeys.java#L146-L147
>
> Just now, I'm realizing that the initial comment in the docs might mean to
> tune it way down to avoid OOME, my initial reading was that we need to
> increase the ceiling from whatever default setting comes in via HDFS. Would
> be good to clarify this, and also figure out what units the value is in.
>
>
Agree.

IIRC, intent was to set it way-down from usual default because hbase runs
w/ many more open files than your typical HDFS client does.




> 3)
>
> Docs suggest: Ensure data locality. In hbase-site.xml, set
> hbase.hstore.min.locality.to.skip.major.compact = 0.7 (Meaning that 0.7 <=
> n <= 1)
>
> I can't find anything else about this property in the docs. Digging through
> the code, I find an oblique reference to HBASE-11195, but there's no RN
> there or docs from there, and reading the issue doesn't help me understand
> how this operates either. It looks like there was follow on work done, but
> it would be useful to know how we arrived at 0.7 (seems arbitrary) and how
> an operator could figure out if that setting is good for them or needs to
> slide higher/lower.
>
>
I don't know anything of the above.

Thanks,
S



>
> Thanks,
> Mike
>