Trafodion

Hbase goes down during gate regression tests

Bug #1410355 reported by Suresh Subbiah
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Trafodion
New
Medium
Joanie Cooper
Trafodion r2.0

Bug Description

During gate tests, while on the seabase test suite, HBase went down right at the start of the first test. It seems to have been OK during suite setup when some rows are inserted into the defaults table. However for the first line in the the test (seabase/test010) there are errors reported. The delivery that failed is https://review.trafodion.org/#/c/937/

Here is snippet from the log

~/workspace/core-regress-seabase-cdh5.1/trafodion/core/sql/regress/tools ~/workspace/core-regress-seabase-cdh5.1/trafodion/core/sql/regress
Trafodion Conversational Interface 0.9.1
(c) Copyright 2014 Hewlett-Packard Development Company, LP.
>>-- Used to perform initialization of the SQL database during seabase regr run
>>--
>>-- @@@ START COPYRIGHT @@@
>>--
>>-- (C) Copyright 2013-2014 Hewlett-Packard Development Company, L.P.
>>--
>>-- Licensed under the Apache License, Version 2.0 (the "License");
>>-- you may not use this file except in compliance with the License.
>>-- You may obtain a copy of the License at
>>--
>>-- http://www.apache.org/licenses/LICENSE-2.0
>>--
>>-- Unless required by applicable law or agreed to in writing, software
>>-- distributed under the License is distributed on an "AS IS" BASIS,
>>-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
>>-- See the License for the specific language governing permissions and
>>-- limitations under the License.
>>--
>>-- @@@ END COPYRIGHT @@@
>>
>>#ifdef SEABASE_REGRESS
>>
>>cqd mode_seabase 'ON';

--- SQL operation complete.
>>
>>delete from TRAFODION."_MD_".DEFAULTS
+> where ATTRIBUTE in('MODE_SEABASE', 'MODE_SEAHIVE', 'SCHEMA',
+> 'SEABASE_VOLATILE_TABLES');

--- 0 row(s) deleted.
>>
>>insert into TRAFODION."_MD_".DEFAULTS
+> values
+> ('MODE_SEABASE ', 'ON', 'inserted during seabase regressions run'),
+> ('SCHEMA ', 'TRAFODION.SCH ', 'inserted during seabase regressions run'),
+> ('SEABASE_VOLATILE_TABLES ', 'ON', 'insert during seabase regressions run');

--- 3 row(s) inserted.
>>
>>insert into TRAFODION."_MD_".DEFAULTS
+> values
+> ('MODE_SEAHIVE ', 'ON', 'inserted during seabase regressions run');

--- 1 row(s) inserted.
>>
>>create shared schema trafodion.sch;

--- SQL operation complete.
>>
>>#endif
>>
>>exit;

End of MXCI Session

***********

Regression test env vars are now set - cd to /home/jenkins/workspace/core-regress-seabase-cdh5.1/trafodion/core/sqf/sql/regress and run!

***********

~/workspace/core-regress-seabase-cdh5.1/trafodion/core/sql/regress
Running regressions from seabase
Checking if processes are up.

Checking attempt: 1; user specified max: 2. Execution time in seconds: 0.

The SQ environment is up!

Process Configured Actual Down
------- ---------- ------ ----
DTM 2 2
RMS 4 4
MXOSRVR 0 0

Making /home/jenkins/workspace/core-regress-seabase-cdh5.1/rundir/seabase - continuing with tests
~/workspace/core-regress-seabase-cdh5.1/trafodion/core/sql/regress/seabase ~/workspace/core-regress-seabase-cdh5.1/trafodion/core/sql/regress
Running regressions against MX format objects

~/workspace/core-regress-seabase-cdh5.1/trafodion/core/sqf/sql/regress/tools ~/workspace/core-regress-seabase-cdh5.1/trafodion/core/sql/regress/seabase
Trafodion Conversational Interface 0.9.1
(c) Copyright 2014 Hewlett-Packard Development Company, LP.
>>-- Used to perform initialization of the SQL database during seabase regr run
>>--
>>-- @@@ START COPYRIGHT @@@
>>--
>>-- (C) Copyright 2013-2014 Hewlett-Packard Development Company, L.P.
>>--
>>-- Licensed under the Apache License, Version 2.0 (the "License");
>>-- you may not use this file except in compliance with the License.
>>-- You may obtain a copy of the License at
>>--
>>-- http://www.apache.org/licenses/LICENSE-2.0
>>--
>>-- Unless required by applicable law or agreed to in writing, software
>>-- distributed under the License is distributed on an "AS IS" BASIS,
>>-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
>>-- See the License for the specific language governing permissions and
>>-- limitations under the License.
>>--
>>-- @@@ END COPYRIGHT @@@
>>
>>#ifdef SEABASE_REGRESS
>>
>>cqd mode_seabase 'ON';

--- SQL operation complete.
>>
>>delete from TRAFODION."_MD_".DEFAULTS
+> where ATTRIBUTE in('MODE_SEABASE', 'MODE_SEAHIVE', 'SCHEMA',
+> 'SEABASE_VOLATILE_TABLES');

--- 4 row(s) deleted.
>>
>>insert into TRAFODION."_MD_".DEFAULTS
+> values
+> ('MODE_SEABASE ', 'ON', 'inserted during seabase regressions run'),
+> ('SCHEMA ', 'TRAFODION.SCH ', 'inserted during seabase regressions run'),
+> ('SEABASE_VOLATILE_TABLES ', 'ON', 'insert during seabase regressions run');

--- 3 row(s) inserted.
>>
>>insert into TRAFODION."_MD_".DEFAULTS
+> values
+> ('MODE_SEAHIVE ', 'ON', 'inserted during seabase regressions run');

--- 1 row(s) inserted.
>>
>>create shared schema trafodion.sch;

*** ERROR[1022] Schema TRAFODION.SCH already exists.

--- SQL operation failed with errors.
>>
>>#endif
>>
>>exit;

End of MXCI Session

***********

Regression test env vars are now set - cd to /home/jenkins/workspace/core-regress-seabase-cdh5.1/trafodion/core/sqf/sql/regress and run!

***********

~/workspace/core-regress-seabase-cdh5.1/trafodion/core/sql/regress/seabase
~/workspace/core-regress-seabase-cdh5.1/trafodion/core/sqf/sql/regress/tools/logsort_src ~/workspace/core-regress-seabase-cdh5.1/trafodion/core/sql/regress/seabase
/usr/local/bin/g++ -fPIC -w -g -O0 -g -Wno-unknown-pragmas -fshort-wchar -DNA_LINUX -DNGG -D_M_DG -D_NSKFS_ -D_FULL_BUILD -DARKFS_GENERATOR -D_DP2NT_ -D_LITTLE_ENDIAN_ -DARG_PRESENT_OMIT -DNSK_USE_MSGSYS_SHELLS -D_GLIBCXX_DEPRECATED -D_TNS_R_TARGET -D_X86_ -DWIN32_LEAN_AND_MEAN -D_WIN32_WINNT=0x402 -DNA_ITOA_NOT_SUPPORTED -DMPI_ -D_DEBUG -c line.c -o line.o >> line.is 2>&1
/usr/local/bin/g++ -fPIC -w -g -O0 -g -Wno-unknown-pragmas -fshort-wchar -DNA_LINUX -DNGG -D_M_DG -D_NSKFS_ -D_FULL_BUILD -DARKFS_GENERATOR -D_DP2NT_ -D_LITTLE_ENDIAN_ -DARG_PRESENT_OMIT -DNSK_USE_MSGSYS_SHELLS -D_GLIBCXX_DEPRECATED -D_TNS_R_TARGET -D_X86_ -DWIN32_LEAN_AND_MEAN -D_WIN32_WINNT=0x402 -DNA_ITOA_NOT_SUPPORTED -DMPI_ -D_DEBUG -c logsort.c -o logsort.o >> logsort.is 2>&1
/usr/local/bin/g++ -fPIC -w -g -O0 -g -Wno-unknown-pragmas -fshort-wchar -DNA_LINUX -DNGG -D_M_DG -D_NSKFS_ -D_FULL_BUILD -DARKFS_GENERATOR -D_DP2NT_ -D_LITTLE_ENDIAN_ -DARG_PRESENT_OMIT -DNSK_USE_MSGSYS_SHELLS -D_GLIBCXX_DEPRECATED -D_TNS_R_TARGET -D_X86_ -DWIN32_LEAN_AND_MEAN -D_WIN32_WINNT=0x402 -DNA_ITOA_NOT_SUPPORTED -DMPI_ -D_DEBUG -c row.c -o row.o >> row.is 2>&1
/usr/local/bin/g++ -fPIC -w -g -O0 -g -Wno-unknown-pragmas -fshort-wchar -DNA_LINUX -DNGG -D_M_DG -D_NSKFS_ -D_FULL_BUILD -DARKFS_GENERATOR -D_DP2NT_ -D_LITTLE_ENDIAN_ -DARG_PRESENT_OMIT -DNSK_USE_MSGSYS_SHELLS -D_GLIBCXX_DEPRECATED -D_TNS_R_TARGET -D_X86_ -DWIN32_LEAN_AND_MEAN -D_WIN32_WINNT=0x402 -DNA_ITOA_NOT_SUPPORTED -DMPI_ -D_DEBUG -c rowlst.c -o rowlst.o >> rowlst.is 2>&1
/usr/local/bin/g++ -fPIC -w -g -O0 -g -Wno-unknown-pragmas -fshort-wchar -DNA_LINUX -DNGG -D_M_DG -D_NSKFS_ -D_FULL_BUILD -DARKFS_GENERATOR -D_DP2NT_ -D_LITTLE_ENDIAN_ -DARG_PRESENT_OMIT -DNSK_USE_MSGSYS_SHELLS -D_GLIBCXX_DEPRECATED -D_TNS_R_TARGET -D_X86_ -DWIN32_LEAN_AND_MEAN -D_WIN32_WINNT=0x402 -DNA_ITOA_NOT_SUPPORTED -DMPI_ -D_DEBUG -c symtab.c -o symtab.o >> symtab.is 2>&1
/usr/local/bin/g++ -fPIC -w -g -O0 -g -Wno-unknown-pragmas -fshort-wchar -DNA_LINUX -DNGG -D_M_DG -D_NSKFS_ -D_FULL_BUILD -DARKFS_GENERATOR -D_DP2NT_ -D_LITTLE_ENDIAN_ -DARG_PRESENT_OMIT -DNSK_USE_MSGSYS_SHELLS -D_GLIBCXX_DEPRECATED -D_TNS_R_TARGET -D_X86_ -DWIN32_LEAN_AND_MEAN -D_WIN32_WINNT=0x402 -DNA_ITOA_NOT_SUPPORTED -DMPI_ -D_DEBUG -c tokstr.c -o tokstr.o >> tokstr.is 2>&1
/usr/local/bin/g++ -fPIC -w -g -O0 -g -Wno-unknown-pragmas -fshort-wchar -DNA_LINUX -DNGG -D_M_DG -D_NSKFS_ -D_FULL_BUILD -DARKFS_GENERATOR -D_DP2NT_ -D_LITTLE_ENDIAN_ -DARG_PRESENT_OMIT -DNSK_USE_MSGSYS_SHELLS -D_GLIBCXX_DEPRECATED -D_TNS_R_TARGET -D_X86_ -DWIN32_LEAN_AND_MEAN -D_WIN32_WINNT=0x402 -DNA_ITOA_NOT_SUPPORTED -DMPI_ -D_DEBUG line.o logsort.o row.o rowlst.o symtab.o tokstr.o -o logsort.exe >> .is 2>&1
~/workspace/core-regress-seabase-cdh5.1/trafodion/core/sql/regress/seabase
/home/jenkins/workspace/core-regress-seabase-cdh5.1/trafodion/core/sqf/sql/regress/tools/runregr_seabase.ksh
(RELEASE build)
Creating directory /home/jenkins/workspace/core-regress-seabase-cdh5.1/trafodion/core/sqf/sql/regress/FilteredLogs
Creating directory /home/jenkins/workspace/core-regress-seabase-cdh5.1/trafodion/core/sqf/sql/regress/FilteredLogs/win
Creating directory /home/jenkins/workspace/core-regress-seabase-cdh5.1/trafodion/core/sqf/sql/regress/FilteredLogs/win/seabase
Creating directory /home/jenkins/workspace/core-regress-seabase-cdh5.1/trafodion/core/sqf/sql/regress/FilteredLogs/win/seabase/logfiles
Creating directory /home/jenkins/workspace/core-regress-seabase-cdh5.1/trafodion/core/sqf/sql/regress/FilteredLogs/win/seabase/expfiles
FilteredLogs: /home/jenkins/workspace/core-regress-seabase-cdh5.1/trafodion/core/sqf/sql/regress/FilteredLogs/win/seabase
Running tool with
/home/jenkins/workspace/core-regress-seabase-cdh5.1/trafodion/core/sqf/sql/regress/tools/runregr_seabase.ksh
------------------------------------------------------------
ls: cannot access test???*: No such file or directory
ls: cannot access expected???*: No such file or directory
ls: cannot access filter???*: No such file or directory
ls: cannot access *.known*: No such file or directory
-- Skipped testfiles:
TEST021

-- Testfiles:
TEST010 TEST011 TEST012 TEST013 TEST014
TEST016 TEST018 TEST020 TEST022 TEST024
TEST025
--
-- Executables:
lrwxrwxrwx. 1 jenkins jenkins 100 Jan 13 00:18 /home/jenkins/workspace/core-regress-seabase-cdh5.1/trafodion/core/sqf/export/bin64/sqlci -> /home/jenkins/workspace/core-regress-seabase-cdh5.1/trafodion/core/sql/lib/linux/64bit/release/sqlci
lrwxrwxrwx. 1 jenkins jenkins 105 Jan 13 00:18 /home/jenkins/workspace/core-regress-seabase-cdh5.1/trafodion/core/sqf/export/bin64/tdm_arkcmp -> /home/jenkins/workspace/core-regress-seabase-cdh5.1/trafodion/core/sql/lib/linux/64bit/release/tdm_arkcmp
--
-- Current work directory:
/home/jenkins/workspace/core-regress-seabase-cdh5.1/rundir/seabase
------------------------------------------------------------

------------------------------------------------------------
-- Starting test TEST010:
------------------------------------------------------------

cat: /home/jenkins/workspace/core-regress-seabase-cdh5.1/trafodion/core/sqf/sql/regress/seabase/cidefs: No such file or directory
Trafodion Conversational Interface 0.9.1
(c) Copyright 2014 Hewlett-Packard Development Company, LP.
>>-- @@@ START COPYRIGHT @@@
>>--
>>-- (C) Copyright 2013-2014 Hewlett-Packard Development Company, L.P.
>>--
>>-- Licensed under the Apache License, Version 2.0 (the "License");
>>-- you may not use this file except in compliance with the License.
>>-- You may obtain a copy of the License at
>>--
>>-- http://www.apache.org/licenses/LICENSE-2.0
>>--
>>-- Unless required by applicable law or agreed to in writing, software
>>-- distributed under the License is distributed on an "AS IS" BASIS,
>>-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
>>-- See the License for the specific language governing permissions and
>>-- limitations under the License.
>>--
>>-- @@@ END COPYRIGHT @@@
>>
>>-- defs included during seabase regr run
>>
>>#ifdef SEABASE_REGRESS
>>cqd mode_seabase 'ON';

--- SQL operation complete.
>>cqd seabase_volatile_tables 'ON';

--- SQL operation complete.
>>cqd hbase_async_drop_table 'OFF';

--- SQL operation complete.
>>cqd hbase_serialization 'ON';

--- SQL operation complete.
>>--cqd hbase_coprocessors 'OFF';
>>cqd hbase_filter_preds 'ON';

--- SQL operation complete.
>>--cqd traf_default_aligned_format 'ON';
>>--cqd traf_default_col_charset 'UTF8';
>>set SCHEMA $$TEST_SCHEMA$$;

--- SQL operation complete.
>>#endif
>>
>>-- Tests for SeaBase
>>-- Added June 2013
>>--
>>-- @@@ START COPYRIGHT @@@
>>--
>>-- (C) Copyright 2013-2014 Hewlett-Packard Development Company, L.P.
>>--
>>-- Licensed under the Apache License, Version 2.0 (the "License");
>>-- you may not use this file except in compliance with the License.
>>-- You may obtain a copy of the License at
>>--
>>-- http://www.apache.org/licenses/LICENSE-2.0
>>--
>>-- Unless required by applicable law or agreed to in writing, software
>>-- distributed under the License is distributed on an "AS IS" BASIS,
>>-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
>>-- See the License for the specific language governing permissions and
>>-- limitations under the License.
>>--
>>-- @@@ END COPYRIGHT @@@
>>
>>obey TEST010(clean_up);
>>?section clean_up
>>--------------------------------------------------------------------------
>>drop table if exists t010t1;

*** ERROR[8448] Unable to access Hbase interface. Call to ExpHbaseInterface::getRowOpen returned error HBASE_OPEN_ERROR(-703). Cause:
org.apache.hadoop.hbase.client.NoServerForRegionException: Unable to find region for TRAFODION _MD_ OBJECTS BT after 35 tries.
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1194)
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1110)
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1067)
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:908)
org.apache.hadoop.hbase.client.HTable.getRegionLocation(HTable.java:485)
org.apache.hadoop.hbase.client.transactional.RMInterface.registerTransaction(RMInterface.java:73)
org.apache.hadoop.hbase.client.transactional.RMInterface.get(RMInterface.java:134)
org.trafodion.sql.HBaseAccess.HTableClient.startGet(HTableClient.java:333)
.

*** ERROR[8448] Unable to access Hbase interface. Call to ExpHbaseInterface::getRowOpen returned error HBASE_OPEN_ERROR(-703). Cause:
org.apache.hadoop.hbase.client.NoServerForRegionException: Unable to find region for TRAFODION _MD_ OBJECTS BT after 35 tries.
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1194)
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1110)
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1067)
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:908)
org.apache.hadoop.hbase.client.HTable.getRegionLocation(HTable.java:485)
org.apache.hadoop.hbase.client.transactional.RMInterface.registerTransaction(RMInterface.java:73)
org.apache.hadoop.hbase.client.transactional.RMInterface.get(RMInterface.java:134)
org.trafodion.sql.HBaseAccess.HTableClient.startGet(HTableClient.java:333)
.

*** ERROR[1398] Error -705 occured while accessing the hbase subsystem. Fix that error and make sure hbase is up and running. Error Details:
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=35, exceptions:
Tue Jan 13 00:54:29 GMT 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@702da30a, java.net.ConnectException: Connection refused
Tue Jan 13 00:54:29 GMT 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@702da30a, org.apache.hadoop.hbase.ipc.RpcClient$FailedServerException: This server is in the failed servers list: slave-cdh51-05.trafodion.org/172.16.0.16:45314
Tue Jan 13 00:54:30 GMT 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@702da30a, org.apache.hadoop.hbase.ipc.RpcClient$FailedServerException: This server is in the failed servers list: slave-cdh51-05.trafodion.org/172.16.0.16:45314
Tue Jan 13 00:54:31 GMT 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@702da30a, org.apache.hadoop.hbase.ipc.RpcClient$FailedServerException: This server is in the failed servers list: slave-cdh51-05.trafodion.org/172.16.0.16:45314

Tags: foundation
Revision history for this message
Steve Varnau (steve-varnau) wrote :

initialize trafodion seemed to run fine.

Why do we hang in this state? Is that the desired behavior if hbase goes down?

Atanu Mishra (atanu-mishra)
Changed in trafodion:
milestone: r1.1 → r2.0
assignee: nobody → ron cassou (ron-cassou)
tags: added: foundation
Changed in trafodion:
assignee: ron cassou (ron-cassou) → Joanie Cooper (joanie-cooper)
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.