Trafodion

ERROR: Unable to access Hbase interface.

Bug #1436526 reported by Chirag Bhalgami
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Trafodion
Invalid
Critical
Joanie Cooper
Trafodion r1.1

Bug Description

From trafci, querying table works fine but dropping throws following error message:

-----------------------------------------------------------------------------

SQL>show tables;
TABLE NAMES
----------------------
ICEWALLTEST_MD5 ICEWALLTEST_PLAIN ICEWALLTEST_PLAIN2

SQL>select count(*) from ICEWALLTEST_PLAIN2;
(EXPR)
--------------------
                   3
--- 1 row(s) selected.

SQL>drop table ICEWALLTEST_PLAIN2;
*** ERROR[8448] Unable to access Hbase interface. Call to ExpHbaseInterface::nextRow returned error HBASE_ACCESS_ERROR(-705). Cause:
java.util.concurrent.ExecutionException: java.io.IOException: PerformScan error on coprocessor call, scannerID: 0 java.io.IOException: peformScan encountered Exception txID: 17179870191 Exception: org.apache.hadoop.hbase.UnknownScannerException: ScannerId: 0, was scanner already closed?, transaction id 17179870191, nextCallSeq was 1, for region TRAFODION._MD_.VIEWS_USAGE,,1426027968880.03bee199768c6d4c25c1b757d6f71fd2.
java.util.concurrent.FutureTask.report(FutureTask.java:122)
java.util.concurrent.FutureTask.get(FutureTask.java:188)
org.trafodion.sql.HBaseAccess.HTableClient.fetchRows(HTableClient.java:575)
. [2015-03-25 18:59:17]
*** ERROR[8839] Transaction was aborted. [2015-03-25 18:59:17]

-----------------------------------------------------------------------------

Trafodion Build: Version 1.0.1 Release 1.1.0 (Build release [goyalna], branch a94de6e-master, date 06Mar15)
Hadoop Distro: HDP 2.2

Everything is green at Ambari admin.

See original description
Revision history for this message
Chirag Bhalgami (chirag-bhalgami) wrote :

Additional info:
From sqlci it also throws below error message:

--------------------------------------------------------------------------------
>>drop table ICEWALLTEST_PLAIN2;
 *** ERROR[8604] Transaction subsystem TMF returned error 82 while starting a transaction.
--- SQL operation failed with errors.
--------------------------------------------------------------------------------

tags: added: connectivity-mxosrvr sql-general
Sandhya Sundaresan (sandhya-sundaresan)
Changed in trafodion:
assignee: nobody → Joanie Cooper (joanie-cooper)
Atanu Mishra (atanu-mishra)
Changed in trafodion:
milestone: none → r1.1
status: New → In Progress
Revision history for this message
Joanie Cooper (joanie-cooper) wrote :

Chirag provided a DDL/DML script to reproduce the problem. I tried this script on a workstation with the latest (032615) R1.1.0 build. The script ran without any "performScan" errors.

It was suggested that the Trafodion installed version be upgraded to the latest build and tried again on the test cluster.

We have seen "performScan" errors when the various components are not all up to date and in-sync.

There is also one remaining "performScan" problem being addressed in the TrxRegionEndpoint coprocessor.

This problem can be confirmed to be the same issue by setting the HBase tracing to TRACE.

We'll wait for the Trafodion instance upgrade and see if the problem can be reproduced.

Revision history for this message
Joanie Cooper (joanie-cooper) wrote :

Some additional information provided by Chirag on reproducing the problem:

Executing the provided DDL/DML won't reproduce the issue. It was provided to recreate the schema in your environment and anlayze it. Trafodion went down when execution of the ODB utility (which uses Traf ODBC drivers with 3rd party driver manager unixODBC) was interupted - which somehow brought HBase into weird state. Also, both trafci and sqlci were successfully able to query the table but not succsesfully dropping it. So the bug was opened to analyze what caused HBase to go into this weird state where Trafodion can query (select statements), but the DB but cannot drop the table.

Chirag Bhalgami (chirag-bhalgami)
description: updated
Revision history for this message
Joanie Cooper (joanie-cooper) wrote :

From: Bhalgami, Chirag
Sent: Monday, March 30, 2015 10:44 PM
To: Moran, Amanda; Cooper, Joanie; Bucaojit, Oliver; Anderson, Marvin; Mishra, Atanu
Cc: Messavussu, John; Goyal, Narendra (Trafodion R&D); Pham, Van; Broeder, Sean
Subject: RE: [Bug 1436526] [NEW] ERROR: Unable to access Hbase interface.

Hi Amanda,

Can you please tell us what type of permission issues it had? Do you think changing those permissions were part of the fresh hortonworks installation after wiping everything and reinstalling it from scratch?

Just curious, how everything worked without issues earlier if there were permission issues. Also, it will help with QA of backup/restore scripts and also in debugging https://bugs.launchpad.net/trafodion/+bug/1436526.

Thanks,
- Chirag

From: Moran, Amanda
Sent: Monday, March 30, 2015 10:39 PM
To: Cooper, Joanie; Bucaojit, Oliver; Bhalgami, Chirag; Anderson, Marvin; Mishra, Atanu
Cc: Messavussu, John; Goyal, Narendra (Trafodion R&D); Pham, Van; Broeder, Sean
Subject: RE: [Bug 1436526] [NEW] ERROR: Unable to access Hbase interface.

HI there All-

After many permissions issues with Hortonworks, it is finally up and running.

Installed Trafodion with no issues.

[root@n019 installer]# su trafodion
[trafodion@n019 ~]$ sqcheck
Checking if processes are up.
Checking attempt: 1; user specified max: 2. Execution time in seconds: 0.

The SQ environment is up!

Process Configured Actual Down
------- ---------- ------ ----
DTM 6 6
RMS 12 12
MXOSRVR 12 12

Thanks all!

--Amanda

Revision history for this message
Joanie Cooper (joanie-cooper) wrote :

Thanks Amanda for installing a new HDP 2.2 and Trafodion install.

The current node list is “-w n024 –w n023 –w n022 –w n021 –w n020 –w n019”.

We had some questions if reversing the order of nodes was causing a problem.

A clean install seems to work fine. But, it might still be worth investigating
if reversing the order has any impact on systems undergoing an upgrade.

I have left the LP bug at “In progress” for a bit.

If any changes need to be addressed by the TDI team for permission issues,
then this bug could be reassigned to that team.

I have updated the bug report with the latest status.

Revision history for this message
Joanie Cooper (joanie-cooper) wrote :

We confirmed that a fresh HDP 2.2 and Trafodion installation resolved the HBase connection issues.
Other HDP upgrade installations with Trafodion R1.1.0 have not had these configuration issues.
We are closing out this LP bug as a HDP configuration issue which was resolved by reinstalling HDP 2.2.

Changed in trafodion:
status: In Progress → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.