Subject: in hdfs, part of the last contact value increases



Recently, I have increased two disks on a single datanode . I encountered a
problem, in hdfs, part of the last contact value increases, resulting in hbase
the handler is occupied.




The datanode prints the log as follows:
2016-12-21 10:13:50,816 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
RemoteException in offerService
org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException):
java.lang.NullPointerException


at org.apache.hadoop.ipc.Client.call(Client.java:1411)
at org.apache.hadoop.ipc.Client.call(Client.java:1364)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at com.sun.proxy.$Proxy14.blockReport(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.blockReport(DatanodeProtocolClientSideTranslatorPB.java:175)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.blockReport(BPServiceActor.java:488)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:711)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:846)
at java.lang.Thread.run(Thread.java:744)
2016-12-21 10:13:51,256 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src:
/10.90.18.60:50010, dest: /10.90.18.60:39095, bytes: 24894792, op: HDFS_READ,
cliID:
DFSClient_hb_rs_l-hbase40.dba.cn2.qunar.com,60020,1473253709668_1716681090_37,
offset: 0, srvID: b7cb495e-de56-4f4e-b089-abe20a3e0e53, blockid:
BP-329468547-10.90.18.19-1444480193809:blk_1238808018_165078752, duration:
1214526037


2016-12-21 10:13:51,987 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
RemoteException in offerService
org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException):
java.lang.NullPointerException


at org.apache.hadoop.ipc.Client.call(Client.java:1411)
at org.apache.hadoop.ipc.Client.call(Client.java:1364)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at com.sun.proxy.$Proxy14.blockReport(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.blockReport(DatanodeProtocolClientSideTranslatorPB.java:175)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.blockReport(BPServiceActor.java:488)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:711)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:846)
at java.lang.Thread.run(Thread.java:744)




Use jstack print out the contents of regionserver as follows:
"B.defaultRpcServer.handler=62,queue=2,port=60020" daemon prio=10
tid=0x0000000000e91800 nid=0x63b8 in Object.wait() [0x00007f9b83f43000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:503)
at
org.apache.hadoop.hbase.regionserver.wal.FSHLog.syncer(FSHLog.java:1371)
- locked <0x00007f9cf1334468> (a java.util.concurrent.atomic.AtomicLong)
at
org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1485)
at
org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:5968)
at
org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2580)
at
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2261)
at
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2213)
at
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2217)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.doBatchOp(HRegionServer.java:4257)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3459)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3348)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29593)
at
qunar.tc.hbase.interceptor.support.CustomCompositeInterceptor.callMethod(CustomCompositeInterceptor.java:76)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2008)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
at java.lang.Thread.run(Thread.java:744)



"B.defaultRpcServer.handler=61,queue=1,port=60020" daemon prio=10
tid=0x0000000000e8f800 nid=0x63b7 in Object.wait() [0x00007f9b84044000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:503)
at
org.apache.hadoop.hbase.regionserver.wal.FSHLog.syncer(FSHLog.java:1371)
- locked <0x00007f9cf1334468> (a java.util.concurrent.atomic.AtomicLong)
at
org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1485)
at
org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:5968)
at
org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2580)
at
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2261)
at
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2213)
at
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2217)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.doBatchOp(HRegionServer.java:4257)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3459)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3348)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29593)
at
qunar.tc.hbase.interceptor.support.CustomCompositeInterceptor.callMethod(CustomCompositeInterceptor.java:76)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2008)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
at java.lang.Thread.run(Thread.java:744)



I would like to ask what is the specific reason, and how to optimize.



Programming list archiving by: Enterprise Git Hosting