Testing with transaction enabled.
Our system is using HortonWorks and is running HBASE 0.94 across 12 nodes.
Our hbase max store size was 1GB.
We noticed during loading of large tables, that we would get error 97 returned from the TM and that the batch of rows were not added.
Turns out that our table was being split and that the TM is not handling this at the moment.
We also found out that after a split, the hbase balancer would move the new region to another region server. When this happened, we got more error 97.
WORKAROUND:
- We changed the MAX STORE SIZE to 100GB.
- We changed the SPLIT POLICY to org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPolicy which causes split to happen only once the max size if reached. Default for HBASE 0.94 and up is new POWERof2 policy which causes splits more often.
- We turned off HBASE BALANCER via hbase shell.
Revision 39062 Partial fix merged from seatrans_2 to datalake_64 branch: Added code to region close code to suspend region splits until there are no active transactions in the region. This can be configured through an environment variable, default is on. The code already prohibits region splits while transactions are in phase 2 to ensure DB consistency.